Usability Tests: A Few Comments

Nathaniel's picture

A few comments in response to this week's reading responses covering usability tests:

  • One thing we noticed about several students' sample questions (for surveys) is that many of them consist of only yes/no questions. While surveys may contain a few of these, students need to think about the quality and usability of the answers they will get. "Yes" and "No" answers won't provide feedback on the "why" or "how." With yes/no answers students may discover that their instructions don't work, but they will not have learned why they didn't work or how to improve them.
  • In his response this week, Matt made a good point about how the context of a usability test might affect how users perform. He was right in pointing out that some forms of usability testing can themselves "get in the way" of usability. We would point out that this is an issue with any kind of testing, and that the content of the test, in addition to its context, affects the usability test's results. Nathaniel was listening to "Mike and Mike in the Morning" a few days ago and they were discussing a poll conducted by ESPN and a university. The survey was attempting to gauge sports fans' attitudes about steroids in baseball. The survey was as follows:

    How much do you care about steroids in baseball?

    1. A lot
    2. A little
    3. Not at all

    See the problems? For starters, the poll was self-selecting. That is, it was not sent to a random sampling of baseball fans, but was completed by fans who sought out the survey. Additionally, it offer a limited range of responses. It also doesn't assess how these attitudes manifest themselves. For instance, does caring a lot lead to not watching or watching baseball? This is clearly an issue for usability. We don't just want to know whether something is confusing or not; we want to know what the confusion will do to the user.

    This is why we need to think carefully about our usability tests. As Matt points out, the context of a test can affect the outcome. And, just as importantly, so can the questions themselves. Every question suggests a certain kind of answer.

As students create their tests they should pay special attention to how the test itself asks people to use to the instructions and how it asks them to respond. Questions get the answers they deserve; write questions that ask for concrete and specific answers.