My November column discussed how to identify the system functions you need in a demand generation system. However, functions are only one piece of the puzzle, and many vendors argue that their real advantage actually lies in superior “ease of use.” How do you measure that?
This is a serious challenge. Functionality is relatively straightforward - either a system has a particular capability or it doesn’t - but ease of use is more subtle. Formal, objective measurement is difficult and expensive, so most people end up relying on subjective assessments instead. And even subjective assessments are hard to organize.
To make any sense at all of the topic, we need to take a couple steps back. First, ease of use is one component of the larger and ultimately more important concept of “usability.” This is defined in different terms but generally includes ease of use, ease of learning and functionality.
Second, both ease of use and usability are highly contextual. They can only be measured for specific tasks for specific users in specific circumstances. This makes perfect sense if you think about it for a moment: a system may be quite easy to use for one task, yet very difficult for another. But it means that a single “usability score” makes much less sense than a single “functionality score.” (Not that functionality scores make much sense to begin with. The point of last month’s column was to measure only the functions that matter to you.)
The good news is that accepting the contextual nature of usability is the first step toward a reasonable way to measure it. Specifically, it hints that you should start by defining the contexts you need to consider. Then you can actually assess usability within those contexts.
As mentioned, the three broad types of contexts are tasks, users and circumstances.
The tasks you need to consider are essentially the same ones identified in the use cases that drive your functional requirements. Usability analysis adds one new dimension to these: measuring the effort to do a task the first time compared with the effort to do it repeatedly. The difference is setup effort, which applies to the first iteration only. This turns out to be a significant differentiator among demand generation systems, because some are optimized for simple, one-off campaigns (not much setup but little reuse) while others are best at creating many variations within a fixed theme.
Users come in many varieties. Some dimensions to consider are: experience with the system, frequency of system use, skill sets (marketing, analysts, technologists, etc.) and system access (end users versus administrators). You will first identify the characteristics of your users, recognizing that you will probably have several sets of users who fall into different groups. Then, determine which types of users will perform which tasks, bearing in mind that some tasks may be shared across different groups. For example, administrators who are technically skilled and frequent users may set up program templates that are then completed by infrequent end users who are primarily marketers.
Circumstances vary as well. Will your users be focused on the demand generation system or will they be in chaotic environments with many distractions? Will they be under extreme time pressure or be able to plan ahead? Is there some leeway for error or must everything be perfect the first time, perhaps for regulatory reasons? The answers bear on system attributes such as display style, alert functions, error checking capabilities, approval workflows, versioning and security. Again, different tasks will likely be performed in different circumstances.
Once you have worked through these issues, your original task list will now be extended to include information on which types of users will perform each task, and in what kinds of circumstances. This allows you to assess the usability of each task against the right criteria.
This still hasn’t answered the question of how to do the assessment itself. In an ideal world, and sometimes in the real world when the stakes are high enough, you would install a test version of the software, train the appropriate users and let them work with it. But most evaluation projects don’t have the time or resources to make such an investment.
Be the first to comment on this post using the section below.