There are times when “ecological validity” is important, and there are other times when you just want to know whether users can make their way through your interface. Save money by choosing the right location to run each type of study.
Some types of user research only really work if you observe users in their natural environment. If you want to discover user pain points or determine product requirements, of if you want feedback on satisfaction, delight or utility it helps to observe natural behavior.
Other research can be run in less realistic settings, for instance in your office. In fact, if you are looking for data on effectiveness or efficiency, it helps to run sessions in a distraction-free environment that you can control.
Where are you in the cycle?
Typically the types of data you want to collect correspond to where you are in your shipping cycle – are you investigating new ideas, studying early builds, validating something just before you release, or capturing feedback on an existing product?
- Early in the project, get out in the field. Make sure you’re developing something that people need
- Perform field or site visits, user need analysis, or Lead User studies to determine user needs
- Evaluate the usability of competitor products that are already on the market
- Once you start development, stay in the office
- Create paper prototypes to set product direction
- Use mock-ups or early code to measure interactions
- For studies in your office, you may have to “engineer” any events like interruptions or incoming messages that you need to observe
- Once it’s good enough (but with enough time before shipping to fix stuff) get back out to the field for validity testing
- Make sure you have (relatively) stable code for user studies – you’ll want to install on users’ machines and devices
- Instrument your code or use server logs for metrics-based longitudinal work
Even if you’re on a continuous delivery cycle, different pieces of functionality will be at different stages at any one time.
What’s the minimum viable environment?
Think hard about when to invest in real-world studies and when to save money by staying in the office. You want to answer your questions as quickly and cheaply as possible. If you can do that without requiring an “ecologically valid” (real-life) environment, you’ve saved time and money for the occasions when you have questions that truly require real-life observation.