LarryRoth.net

What does the user think is a success?

When conducting a usability test, it is common practice to follow a procedure similar to this:

• Create user scenarios
• Prepare tasks
• Observe various users while executing these tasks
• Mark successes and failures
• Look for patterns and areas of concerns and provide feedback to the application owner

I recently booked a room at a national hotel chain using their on-line reservation system. When I arrived at the hotel, I found out that both the rate and the number of guests differed from my confirmation sheet. It turns out that their reservation system faxes the information to the hotel property and that information is then manually keyed in—leaving obvious room for error. In my case, the error was rectified politely and professionally, but it made me think about what makes a usable experience. It made me wonder about a successful task vs. a successful process and what the perception of the user may be.

In the example of a hotel reservation system, if I had been asked to conduct a usability study I would have focused on the reservation form(s) and made sure that users could complete the task of reserving a room for a specific property on a specific day. If they could, I would have thought of it as a success. And that may be fine for the purposes of the usability test. But stepping back and examining the experience from a users perspective, the task may not be a success until they are actually in their room!

I know what your are thinking. First off, there are a lot of "what ifs..." here, and secondly, it certainly isn't the goal of a usability test to identify such broadly scoped successes or failures. I agree with both of those statements, and I am certainly not proposing larger, more involved testing—unless someone wants to fly me around the country trying out reservations at 5 star hotels!

Seriously, my point is that as usability experts, it's our goal to remind the client that we are testing very granular tasks to a sometimes much larger process. While we may help to remove all the barriers a users encounters, the user will be judging the process as a whole, and there may be some hidden barriers that can't easily be tested. Thoughts?