Behaviorally-Anchored User Experience Issue Severity Ratings

This post originated from a response I wrote to a question on the list that dare not utter its name. Someone asked about whether ratings of usability issues should be ranked with an interval or ordinal scale. I thought the question was somewhat specious, because when you’re dealing with behavioral phenomena, claiming your measurement tool is interval vs. ordinal is a distinction without a difference.

What is important, however, is behaviorally anchoring your rating choices. That is, as much as possible you should base your usability severity ratings on observable – or well-defined inferable – criteria. There’s nothing earth-shatteringly new in this post, mind you. I’m just taking the highly non-controversial position that you should define your usability and user experience issue ratings using observable examplars of behavior.

You may notice that I do move off the reservation a bit when you read my rating definitions. I’ve included information about how a user experience issue could affect an organization’s brand equity and revenue as well. I’m not entirely satisfied with how I’ve lumped these (important but somewhat orthogonal) issues together with “straight” usability; I may break them out into separate ratings that accompany each usability issue. So my rating schema would work like this:

• Usability severity
• Impact to brand equity
• Impact to revenue or (other key performance metric)

Anyway….here is my current set of behaviorally-anchored user experience issue ratings; feel free to borrow, modify, criticize, adapt, ignore, etc.

Critical
A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability.

High
A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability.

Medium
Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability.

Low
Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials.