Just ask one person

Something else from the 988 Suicide Help Line story.

Remember, they're trying to pick some hold music that will help callers stay on the line long enough to reach a counselor.

They’ve narrowed it down to four choices, and now they want some input on which one is best.

So how did they do it?

They walked around on the National Mall in Washington DC, and asked people at random.

From the RadioLab episode:

STEPHANIE [TECH LEAD FOR 988]: ... we had people listen live through our phones and vote on which one they liked the best. ... and by and large everyone really agreed on the same music choice. ...

[NARRATOR]: With one massive caveat.

STEPHANIE: ... we're limited to talk to nine people.

[SHOW HOST 1]: That's ridiculous!

[SHOW HOST 2]: That's it? Like, the mental health of millions of people depends on these nine strangers on the mall?

Yeah.

Nine people.

Doesn't sound like much, does it?

But you know what?

One of the show hosts took the same method to New York's Time Square

... and asked about twice as many people

... and got the same results.

Here's the thing:

Sample size in survey-based research is a big complicated topic that I won't get into here.

For extremely precise and rigorous research, you'll want a sample between 100 and 1000 people.

But do you really need that “extremely precise and rigorous?” if you're starting with virtually zero information, asking even a handful of people can be very useful to point you in the right direction.

Or, as Steve Krug puts it in his book, Don’t Make Me Think: A Common Sense Approach to Web Usability, “Testing one user is 100% better than testing none.”

I’ll go one further: Mathematically speaking, testing two users is 100% better than testing one. But testing one user is infinitely better than testing none.

All the best,
A.