Online survey service. Insufficient guidance on question type selection. Results: see picture.
{ 13 comments }
Research
This post originated from a response I wrote to a question on the list that dare not utter its name. Someone asked about whether ratings of usability issues should be ranked with an interval or ordinal scale. I thought the question was somewhat specious, because when you’re dealing with behavioral phenomena, claiming your measurement tool is interval vs. ordinal is a distinction without a difference.
What is important, however, is behaviorally anchoring your rating choices. That is, as much as possible you should base your usability severity ratings on observable – or well-defined inferable – criteria. There’s nothing earth-shatteringly new in this post, mind you. I’m just taking the highly non-controversial position that you should define your usability and user experience issue ratings using observable examplars of behavior.
You may notice that I do move off the reservation a bit when you read my rating definitions. I’ve included information about how a user experience issue could affect an organization’s brand equity and revenue as well. I’m not entirely satisfied with how I’ve lumped these (important but somewhat orthogonal) issues together with “straight” usability; I may break them out into separate ratings that accompany each usability issue. So my rating schema would work like this:
- Usability severity
- Impact to brand equity
- Impact to revenue or (other key performance metric)
Anyway….here is my current set of behaviorally-anchored user experience issue ratings; feel free to borrow, modify, criticize, adapt, ignore, etc.
Critical
A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability.
High
A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability.
Medium
Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability.
Low
Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials.
{ 9 comments }
Back about two years ago I was working on a product line that took a number of potentially objectionable actions with customers’ systems. I pushed back against the product teams, saying that these actions put our products at risk of being perceived as malware.
They in turn pushed back on me, essentially telling me to prove my allegations.
So I went away for a few days, did some research, and returned with my (fanfare) Malware Perception Risk Assessment Tool. Ta-da!
Uh, sorry, I meant “thud”. It went over like a lead ballon. No takers. So I wrote it up in an article at UXmatters, hoping it’d become adopted. More deafening silence. Dejection.
But here’s the thing: systems are becoming more and more interconnected, and more than ever, applications are utilizing aspects of your personal, semi-public, and public data to derive value (presumably for you as well as themselves). Thus the risk of an application being perceived as malware has only increased.
I strongly believe that our field needs to provide the wider world with a tool that can help assess the risk that a particular product or service might be tagged as malware in the minds of users or the market at large.
So I again submit to the UX, dev, and product management communities the Malware Risk Assessment Checklist.
To measure the probability of people perceiving a product as malware, I created a checklist representing a set of attributes that typically characterize malware. I grouped these attributes into these five categories, each containing two or more representative attributes:
- personal information gathering and usage
- modification of data or system configuration
- stealth and resistance to removal or modification
- resource utilization
- transparency and disclosure of third-party relationships
This time, I’m explicitly calling out the fact that the checklist is light on data propagation via social networking applications. And I’m asking for help in rounding out that aspect of the checklist. So help a guy out and suggest some social media items. I am releasing this checklist under a “Creative Commons non-commercial share alike-derivative works permitted” license, so you can remix this, add to it, etc. When I receive some good item suggestions, I’ll re-roll the list and publish again.
Here’s the checklist as it stood in 2008. Peeps, have at it.
Personal Information Gathering and Usage
The product or Web site…
Gathers and transmits users’ personal data or information about users’ behavior to the organization providing the product
____Yes
____No
Gathers and transmits users’ personal data or information about users’ behavior to a third party.
____Yes
____No
Uses personal data and data the product developer obtained from third parties to assemble profiles of users that are more complete and comprehensive than users expect.
____Yes
____No
Exposes more of users’ personal information to their contacts or a community than users expected or wanted.
____Yes
____No
Does any of the above without user notification and consent.
____Yes
____No
Does any of the above and does not allow users to opt out.
____Yes
____No
Modification of Data or System Configuration
The product or Web site…
Overwrites, modifies, or destroys users’ data without their knowledge or consent.
____Yes
____No
Modifies other applications on users’ computers or their operating system settings or computing environment.
____Yes
____No
Fails to restore modifications to other applications, operating system settings, or the computing environment when the user uninstalls the product.
____Yes
____No
Damages or renders inoperative other software or hardware on users’ computing systems.
____Yes
____No
Stealth and Resistance to Removal or Modification
The product or Web site…
Hides or renders its files and resources inaccessible to the user through normal means—that is, using standard file managers and viewers.
____Yes
____No
Resists attempts at removal.
____Yes
____No
Modifies antivirus, antispyware, and other computing hygiene applications or application settings, to make itself appear harmless or less harmful than it actually is.
____Yes
____No
Resource Utilization
The product or Web site…
Overuses computing resources—CPU, GPU, memory, and so on—to a noticeable extent.
____Yes
____No
Utilizes computing resources for purposes not directly related to the tasks users typically perform with the software.
____Yes
____No
Transparency and Disclosure of Third-Party Relationships
The product or Web site…
Installs third-party applications that demonstrate any of the above behaviors.
____Yes
____No
Installs third-party applications without user notification and consent.
____Yes
____No
C’mon people, let’s make this checklist useful, and maybe even a de facto standard.
{ 6 comments }
Last week I gave a presentation at the Online Marketing Summit 2010 about website usability and conversion. I used the famous Ling’s Cars site as an example of conversion, arguing that “unique” (read: ugly) design coupled with “stealthy” usability can facilitate conversion.
You can check out the preso here, or go to Slideshare to see it in full-screen glory. Bonus points to anyone who reports back with the page number of my “Captain Obvious” reference. I hope you enjoy the presentation. Please feel free to email or comment with questions, refutations, counter-arguments, and the like. -Paul
Web Usability and Conversion (View more presentations from Paul Sherman)
{ 10 comments }
