How to Avoid Pitfalls in Usability Testing

Meg Davis  

January 16, 2012

Related Topics

Designing and moderating usability tests requires the specialized skills of understanding how people’s brains work. Even more so, interpreting the results from usability testing requires the specialized skill of switching between meticulous details of interaction and broad strokes of experiences across participants.

Here are 5 tips to avoiding pitfalls when interpreting usability test sessions:

  1. Suspend judgment. While moderating usability tests, it’s easy for me to get caught up in the moment. I feel so close and empathetic towards each user that it’s easy to take what one person says and run with it. However, I have to remind myself that I’m not designing the system for one user or even one type of user most of the time. I force myself to suspend judgment until I can go back and review the recordings of the sessions. I recommend both video capture of the participant’s face as well as of the computer screen.
  2. Look across users. As a moderator and human being, I find I have biases in terms of what I find interesting in a usability session or what I take notice of during a usability test. Having a partner in crime watching the usability test is the best insurance that our team will get unbiased results. Another good technique I’ve learned for eliminating my biases is lining each task of the usability test in a column on a spreadsheet. I go through each participant and put details from each participant’s experience in the rows corresponding to the column tasks. This way, all the participant’s details (including completion rate, portions of the task they struggled with, quotes, etc.) are lined up and easy to scan for patterns.
  3. Listen, no really listen, to participants. If it is clear to us that we missed the mark on an interaction, it is very helpful to play/rewind/play/rewind the video tapes to listen to how participants talk about the interaction. It’s so valuable to see what participants do, but it’s equally valuable to understand their mental models and how they see the feature through the language they use to describe it. Recording quotes from each participant can offer some surprising insights into how to course-correct a feature to match people’s mental models.
  4. Know the limitations of testing. I’m an advocate of usability testing, but I know there are limitations for how realistic it can be. Sometimes our prototype doesn’t provide the same flexibility the real system would. Sometimes we are testing something new that might change people’s workflows over time. When looking at the results of the usability test, I have to remind myself what we were testing and what we were NOT testing. This helps remove some of the confusion and noise of the data.
  5. Understand trade-offs. There’s no such thing as a free lunch. As a user experience designer, there’s no such thing as a perfect design. When making decisions about what needs to change from a usability testing session, I have to remember that each feature does not exist in a vacuum. Changing the details of one feature could negatively impact another feature. Before making recommendations, I consider all the implications of changing something in the ecosystem. If there is time, iterative testing is a great way to really understand these trade-offs.