planning user studies

Different locations for different measurements

There are times when “ecological validity” is important, and there are other times when you just want to know whether users can make their way through your interface. Save money by choosing the right location to run each type of study.

Early research work for customer development and user needs is lead user studies and ethnography; during development measure effectiveness and efficiency with RITE testing and analytics; close to release, measure utility and delight with analytics and direct observation
Get out to users’ locations for discovery work. Early UI can be tested in your office. Once the code is marginally stable, take it back out to the real world where people would normally use it.

Some types of user research only really work if you observe users in their natural environment. If you want to discover user pain points or determine product requirements, of if you want feedback on satisfaction, delight or utility it helps to observe natural behavior.

Other research can be run in less realistic settings, for instance in your office. In fact, if you are looking for data on effectiveness or efficiency, it helps to run sessions in a distraction-free environment that you can control.

Where are you in the cycle?

Typically the types of data you want to collect correspond to where you are in your shipping cycle – are you investigating new ideas, studying early builds, validating something just before you release, or capturing feedback on an existing product?

  • Early in the project, get out in the field. Make sure you’re developing something that people need
  • Once you start development, stay in the office
    • Create paper prototypes to set product direction
    • Use mock-ups or early code to measure interactions
    • For studies in your office, you may have to “engineer” any events like interruptions or incoming messages that you need to observe
  • Once it’s good enough (but with enough time before shipping to fix stuff) get back out to the field for validity testing
    • Make sure you have (relatively) stable code for user studies – you’ll want to install on users’ machines and devices
    • Instrument your code or use server logs for metrics-based longitudinal work

Even if you’re on a continuous delivery cycle, different pieces of functionality will be at different stages at any one time.

What’s the minimum viable environment?

Think hard about when to invest in real-world studies and when to save money by staying in the office. You want to answer your questions as quickly and cheaply as possible. If you can do that without requiring an “ecologically valid” (real-life) environment, you’ve saved time and money for the occasions when you have questions that truly require real-life observation.