Testing Your Own Designs: Discussion

by Paul Sherman on September 10, 2009 · 12 comments

I’m putting an article together for UXmatters on the topic of usability testing and validating one’s own designs. My goal is to develop some guidelines for self-testing.

I’d love to get your feedback on some questions I have:

Testing your own designs: all-around bad idea? Or is it possible to do it well?
If so, what should you do / not do when testing designs you’ve created?
What should you look out for?
Got a good story or anecdote to share? Please do. Either success stories or cautionary tales are welcomed.

Thanks in advance. -Paul

Tagged as: Announcements, Methods, Research

http://www.usefulusability.com/ Craig Tomlin

When I recently redesigned my own personal site (wcraigtomlin.com) I on purpose made sure to consult with others about usability reviews – and not test it myself. My belief is it's too easy to be comfortable with your own thoughts & design, meaning becoming blind to potential issues. I guess I believe in the old saying “Doctor, don't even think about healing yourself” (wait, is that a saying? oh well, guess it is now).
DecafPat

Hi!

I agree with Craig, as the designer, you have worked on the “product” so long that you know the potential problems, but unless you ask your Target Consumer to test it, you will never realize how small or big the problem you thought you had is. And you will be blind to how the consumer will use it and the problems they experience.
Big companies spend a lot of money in consumer focus groups, some of us do not have the same $$ to spend, but is still doable. I was a product manager for kids products, I did not have budget for a full blown study. What did I do? ask my friends with kids to bring in their kids one afternoon and use my new product. It was as expensive as supplying drinks and snacks to the moms!
So, yes, I do think is not a good idea to test it yourself.
It will be interesting to read from someone that believes you can do it yourself, I am always open to learn!
Jim

As someone in QA rather than Design, I think that yes, you *can* do (some of) the testing yourself. But you have to be able to take two steps back, one sideways, and then put yourself into a usability testing mindset.

In this mindset, you have to divorce yourself from what you know about the design and implementation and approach it fresh, questioning the decisions up and down the pike. You have to recognize your own biases and be able to put them aside.

This is not an easy thing to do. (Myself, I honed some of these skills playing card and board games against myself as a kid. When you can play chess against yourself, Grasshopper, and win…) And you will still be only one piece of testing data, albeit a knowledgeable one.

Testing your designs yourself is better than *not* testing them, but not as good as having others test them as well.
http://jay-photo.com/blog Jay Zipursky

Hey, Paul. Reading these comments, it seems people might be interpreting your question to mean “reviewing” it themselves. I think you mean the situation where the designer also facilitates usability testing with representative users.

I think this is possible (I've done it myself), but you need to be aware of your own bias and be especially careful to keep your questions neutral and not to throw observations away. As the designer, it's easier to ignore or rationalize away observed difficulties. It also helps to have observers that aren't quite as in-love with the design as you are.

http://www.usabilityblog.com/ pjsherman

I think you're right about how some readers are interpreting this, Jay.

Also, thanks for the f.b. and the suggestion to have observers on-hand.

ThinkLikeaUser

Testing your own designs: all-around bad idea? Or is it possible to do it well?

You can do some preliminary testing, like checking for consistency, credibility, readability, and checking tech stuff like validation, links, alt text, cross-browsers, scrolling, screen size, printing etc. But there comes a point when you need to ask other people, because you “get it and think it's great” because you built it, and you can't spot the flaws.

Even things like proofreading are difficult to do on your own because you're programmed to think “I'm brilliant, I never make mistakes”, even though we all do make typos all the time :)

If so, what should you do / not do when testing designs you’ve created? What should you look out for?

Follow a heuristic check list, l(ike one one produced by Xerox years ago) and check the basics. Also test your site against your commercial objectives, or other goals.

At a push you could use some really well fleshed out personas to guide your testing, for example cehcking terminolgy, relevance of prominent home page links etc.

So in conclusion, formal heuristic evaluation makes a big difference, but at some point you absolutely have to do informal/formal user testing with a representative set of users if you really want to optimise your site.

I'd be delighted to offer more information if you think it's useful.

I had to design a user interface in a “clean room” environment, which meant for IPR reasons I was not allowed to speak to anyone who had used the existing system – they were terrified the existing supplier might sue the new supplier for copying.

I had no choice but to rely on my own skills and knowledge, doing user testing via a proxy group who gave me filtered responses, until the UI principles were firmly established (8 months into the project!!). The project cost is £300m in total, so it was pretty high profile, and quite stressful for me, used to engaging directly with users :)
ecarlos

The Straw Man approach works best for me. You elicit early feedback from key stakeholders before investing additional resources into the project. Most users know or don't know what they want, until they see it. A good way to test your own designs is the Straw Man approach.

It's always good to critique yourself (keeps you from selling out), but do it objectively. Jeffery Zeldman sums it up best,”If your old work doesn’t shame you, you’re not growing.” That's what makes our job creative, right?

Great discussion!
jessicaholliday

I agree with the comments that suggest one can test at the beginning of a project to check for validation and more exploratory in nature, but I think as one gets further down the path, a third party unbiased partner is key. Whenever I do test my own designs, I never disclose this is my own design because I don't ever want people to feel as though they are “hurting my feelings”.

admin

Yeah, I can agree with that as well. The longer and more invested you are with a design, the harder it is to change it.

Pingback: saxuleko
http://chapolito.com/ chapolito

I agree with everyone here that self testing can only go so far, but I do think it is useful. One thing I wanted to add is that, along the lines of taking a step back, as a designer I find myself at times concentrating on testing the functionality that I think matters. Ignoring equally important, but less flashy parts of a design can really hinder your ability to self test.

A recent example of this concerns a social network I helped build. While testing I was concerned with the interactions with the site, but after creating a profile I never took the time to go back and fully test the registration and login portions of the site. When I did it made a big difference.
http://chapolito.com/ chapolito

I agree with everyone here that self testing can only go so far, but I do think it is useful. One thing I wanted to add is that, along the lines of taking a step back, as a designer I find myself at times concentrating on testing the functionality that I think matters. Ignoring equally important, but less flashy parts of a design can really hinder your ability to self test.

A recent example of this concerns a social network I helped build. While testing I was concerned with the interactions with the site, but after creating a profile I never took the time to go back and fully test the registration and login portions of the site. When I did it made a big difference.