This post originated from a response I wrote to a question on the list that dare not utter its name. Someone asked about whether ratings of usability issues should be ranked with an interval or ordinal scale. I thought the question was somewhat specious, because when you’re dealing with behavioral phenomena, claiming your measurement tool is interval vs. ordinal is a distinction without a difference.
What is important, however, is behaviorally anchoring your rating choices. That is, as much as possible you should base your usability severity ratings on observable – or well-defined inferable – criteria. There’s nothing earth-shatteringly new in this post, mind you. I’m just taking the highly non-controversial position that you should define your usability and user experience issue ratings using observable examplars of behavior.
You may notice that I do move off the reservation a bit when you read my rating definitions. I’ve included information about how a user experience issue could affect an organization’s brand equity and revenue as well. I’m not entirely satisfied with how I’ve lumped these (important but somewhat orthogonal) issues together with “straight” usability; I may break them out into separate ratings that accompany each usability issue. So my rating schema would work like this:
- Usability severity
- Impact to brand equity
- Impact to revenue or (other key performance metric)
Anyway….here is my current set of behaviorally-anchored user experience issue ratings; feel free to borrow, modify, criticize, adapt, ignore, etc.
Critical
A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability.
High
A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability.
Medium
Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability.
Low
Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials.
{ 6 comments }
I do rate findings and think it's important to do so, but I've used the same “standard” ones for so long that I rarely give thought to the scale. I appreciate, therefore, the topic and like how you focus these on business/financial impact as in the end, that should be which usability elements get tackled first.
Agree very strongly. Had a go at scales based on the National Beer Scoring Scale from CAMRA
http://www.processforusability.co.uk/QIUSS/QIUS...
Like the scale above but prefer users doing the rating to me doing it.
As a sometime home-brewer I can get behind metric that was originally intended for rating beer.
Paul, how do you factor how frequency into your overall prioritization of usability issues? For example, you may observe low or medium issue in frequently performed task. That could likely merit more consideration than a medium issue in a very infrequently performed task.
I find frequency to be a tough variable to measure. Since we typically work with such small samples, I don't trust the frequencies that are derived from usability issues identified with n's of 5 or 6 or even 10. So I guess, rightly or wrongly, that I tend to discount frequency. I don't totally disregard it; I do factor it heuristically into my thinking about an issue. But I typically don't measure it as granularly as others may.
Ranking severity is often subjective although influenced by years of experience. Some calculations can be made when determining the affect on revenue or profitability, but branding and the like is always subjective.
I provide such ratings not for my own purposes, but for the benefit of the stakeholders. Those that determine priorities should be given some guidance as to the level of severity each issue has on the company's bottom line.
Great post.
Comments on this entry are closed.
{ 2 trackbacks }