Bad user testing beats no user testing

Jakob Nielsen has noted that it’s now twenty years since he started what he calls the ‘discount usability movement’.

This might be egging it a wee bit, I’m not sure there is such a ‘movement’ apart from that which Nielsen promotes.

It’s true that major companies use discount usability tactics – I noted before how last.fm used it when their site went through major changes. But ‘movement’?

Moving on …

Nielsen presented a paper entitled “Usability Engineering at a Discount” at the 3rd International Conference on Human-Computer Interaction in 1989.

It was born out necessity, he says, as he simply didn’t have the budget of the IBM User Interface Institute where he’d previously worked.

The paper advocated three main components of discount usability:

• Simplified user testing, which includes a handful of participants, a focus on qualitative studies, and use of the thinking-aloud method. Although thinking aloud had been around for years before I turned it into a discount method, the idea that testing 5 users was “good enough” went against human factors orthodoxy at the time.
• Narrowed-down prototypes — usually paper prototypes — that support a single path through the user interface. It’s much faster to design paper prototypes than something that embodies the full user experience. You can thus test very early and iterate through many rounds of design.
• Heuristic evaluation in which you evaluate user interface designs by inspecting them relative to established usability guidelines.

Nielsen says he was stoned in the market square as a heretic and I can well believe it.

I had a similar experience when, discussing issues with LocalDirectgov’s usability offering, I proposed that council web teams should use discount testing methods. This provoked nigh on outrage and a swipe at Nielsen by the usability company Nomensa. I like to think I moved them on from their initial horror to grudging agreement but you can make your own mind up in the debate, as it spilled over several posts and onto a Nomensa worker’s blog.

• Postscript: More birth pangs of government 2.0
• Usability and culture
• AlastairC – Usability myths and professionals

Nielsen even has the nerve, to some people’s delicate sensibilities, to say:

Discount usability often gives better results than deluxe usability because its methods drive an emphasis on early and rapid iteration with frequent usability input.

As well as, the horror:

Discount usability methods are robust enough to offer decent results even when you don’t use perfect research methodology.

In other words: Bad user testing beats no user testing, every time.

He cites a team that ran a usability study of MacPaint 1.7 (an early drawing program) in 1989 who each tested three users.

Better usability methodology does lead to better results, at least on average. But the very best performance was recorded for a team that only scored 56% on compliance with best-practice usability methodology. And even teams with a 20–30% methodology (i.e., people who ran lousy studies) still found 1/4 of the product’s serious usability problems.

Nielsen claims that “my 20 years of campaigning for discount usability have certainly not been in vain, [but] I can’t yet declare a win” — and nowhere is this more evident than in government, where cheap-but-effective methods of finding website errors would, you would think, have most resonance.

Both the US (usability.gov) and UK (usability.coi.gov.uk) official government usability advice contain no reference to discount methods.

~~~~~~~

Here’s a presentation I gave on discount user testing called Cheap’n’easy usability first in 2006.