One of the main insights to make about the reading on usability is that it is important to analyze the document and how it can be tested. I do just that:
In order to the document, we must first understand what it means for the document to be “good”. In our case, the document consists of a set of instructions. For this document to be good, a user with the instructions must be able to achieve their goals. Let’s break this down even further. First, we need to identify what the goals of the user are. This is not a trivial task. The Purdue Course Search Application can be used for a variety of things. The program is mainly for allowing users to retrieve course related information, but different users are interested in different information. Here are some of the common tasks that users might attempt:
- Finding specific information for a course.
- Finding which courses a professor teaches.
- Getting a broader sense of the class catalog.
- Performing a custom search.
I suggest running a usability laboratory where users are given a computer and asked to answer questions and perform tasks. The users should be Purdue students because students like them are the users of this software. This is very close to testing how the document will be used in real life. The answers of the questions and the ability and degree of the completion of the tasks will indicate how well users can perform given the instructions. The goal is to revise the document until users achieve reasonably high scores. Of course, the same users cannot be tested more than once because they have already gained experience with the software. This can be costly so the instructions should be as good as possible before being tested.
In order to determine if the users can perform the tasks I previously outlined, I propose asking them the following questions:
- Question: Find the meeting times for ENGL 10600 Lecture Section 294. Answer: MWF 8:30 AM - 9:20 AM
- Question: How many credits is MA 26300? Answer: 4
- Question: Find the course registration number for CS18000 Recitation 002. Answer: 12953
- Question: What courses does Professor Atallah teach? Answer: CS 38100, CS 59000, CS 69800, CS 69900
- Question: How many 20000 level astronomy classes are there? Answer: 3
- Question: How many courses does Purdue offer on wine? Answer: 6
Questions 1 – 3 test how well users can find specific course information. Question 4 tests their ability to search by professor instead of by course. Question 5 tests their ability to get a broader sense of the class catalog. Finally, question 6 asks them to perform a very custom and unusual search for “wine”. Some of the questions may seem a little vague but that is because I wanted to make sure that they don’t themselves include instructions because then they won’t be testing the instructions document.
Finally, the users should be given a task specific to themselves: Create your current semester’s schedule. This is another common task but also the ultimate test to see if they understand how to use the software as a whole.
Are you testing your instructions, or the software?
It seems as though what you outline here is designed to test the function of your subject (The Purdue Course Search Application) rather than your instructions. We are focused on determining how your instructions function for your different audiences, so your usability testing should examine the successes and failures of your documents. You could perhaps use what you outline here to help you produce your instructions by identifying where people encounter difficulty with the system, but this would be a separate task.
Well...
I can see both sides to this...The usability test, to me, is simply a means for gathering data on the test subjects and their abilities in relation to how well they interact with the instruction documents. What is critical is how the information from this test is analyzed. In this case the test is to test the end result: "does the user, given these instructions, get the correct outcome?" This is similar to a summary test, but in a more directed sense. Of course, this sort of test alone doesn't include the other aspects of the document that need to be check as well; such as mechanical issues, readability, being able to locate parts of the document (from the read and find test)-things like this. So, maybe this test is ok, but certainly not enough.