A word of caution about online surveys

On the internet, nobody knows you’re a dog. Or in this case, that all you want is a dog. A cautionary tale about believing what people tell you in online surveys…

Yesterday I was sitting on a sofa with a friend of mine called Lara. She was using her iPod Touch to fill in a survey about her spending habits. Her only real motivation was the points it would give her towards getting an Andrex Puppy* stuffed toy.

Nothing wrong with that – most online surveys need some kind of motivator or nobody would complete them. The problem was, Lara is twelve years old.

She was thinking out loud as she completed the survey (without any prompting from me) and actually did an admirable job of completing it truthfully. Even the question about her age, which she entered as “under 25.” After she was done, I talked more with Lara about her behavior. One of the things she said was “all my friends do it.” She was very aware that she wasn’t part of the target audience for the survey, but she had few other ways of getting the “puppy points” she needed for her reward.

Sure, lots of household discretionary spending is controlled by kids, but if the marketers behind the survey make decisions based on how Lara told them her family spends, they’ll be in big trouble.

So, when you run your next online survey or online usability test, think carefully about the recruitment mechanism you (or the survey site) use to qualify participants. One of the reasons that in-person usability tests can get away with using smaller sample sizes is because they don’t have to deal with the noise caused by people like Lara.

*For readers outside the UK, Andrex is a brand of toilet tissue, marketed by adorable puppies.