« Frustration not an issue | Main | KM for lawyers »

Usability testing completed

Apologies for the lack of posts over the last few days. I've been incredibly busy running usability tests (3 days of 5 per day) at the Area Health Service, while sick with the flu. Looking forward to the weekend.

The usability testing went well, I think. We had a huge range of users, everyone from managers, community representatives to the apprentice gardener.

Overall, it didn't reveal any startlingly new information. It did, however, confirm the views developed during the usability evaluation of the intranet and website. For example:

• Most people ignored the options in the header and sidebar. They considered them "outside of the page", and simply filtered them out.
• There was an even mix between people who only searched, and those who only browsed.
• The high-level structure of the intranet is not well understood by users.
• The ad-hoc structure and layout of key pages causes considerable confusion.

Key tasks, such as finding a leave form, or booking for training, were almost impossible to do. Unless they had already had exposure to these sections, users universally struggled for up to 10 minutes trying to work out how to do them.

It was really interesting to see the mismatch between user success and opinions. Almost every user described the intranet as "pretty easy to use", despite struggling to find even the simplest of information. (Some users went around in a loop four times looking for the right page.) This highlights that you can't rely on user feedback to characterise the usability of a site.

Some observations and thoughts that surfaced during the testing:

• Ensure that the history is cleared in the browser before each test, otherwise the followed links will give the users strong clues.
• It's hard to construct activities that will take a consistent amount of time. Some people completed the whole exercise in 20 mins, while others were still struggling at 40 mins.
• It's not clear whether there is a lot of value to be gained in testing a site that is obviously broken. Perhaps waiting until some of the key problems had been resolved would have been better. (The proposal was developed before I had an opportunity to examine the intranet.)
• "Go with the flow" during the tests. For some users, they were so skilled that the tests were very easy. These users, however, had some really useful insight, so we explored that instead.
• If the usability testing goes quickly, use the remaining session time for a stakeholder interview. That way, you can get maximum value out of the time.

The only thing left to do now is work through the 115 pages of raw material, and develop the final report. A task for next week. I'll also report further on the results of the tests over the coming days.

Posted by jamesr on August 14, 2002 10:55 PM
Categories: Area Health Service project, Intranets, Usability & user-centered design

Comments

I agree that there isn't much value in testing a site that is broken - especially if you have enough experience to judge it. The main value is in setting a benchmark to compare changes against (and this is valuable unless you have to put people through significant pain to get there).

Posted by: Donna Maurer on August 14, 2002 11:17 PM

Yes, I agree. At least now we can say that: "Previously, no test participants were able to find this form without difficulty. Now all users found it easily". A question, though: should I have timed the usability tests to get some quantitative results?

I guess the other benefit of doing the tests is that it makes the problems concrete, which is handy when trying to convince management to resource their resolution...

Posted by: James Robertson on August 15, 2002 10:15 AM

I have never done quantitative usability testing. I usually do qualitative, and like the freedom to discuss issues as they arise.

Quantitative testing has its place, and I suspect that it is best suited to things where you want to see a measurable improvement in task completion time (rather than an improvement in information retrieval/task success). If you do quantitative, you have to be careful not to do anything to impact on completion time.

Posted by: Donna Maurer on August 15, 2002 11:24 PM