Online testing can give you fast feedback for very little financial outlay. The results might be less trustworthy than face-to-face sessions, but the technique fits well as a complementary tool.
Remote usability testing tools let you measure the usability of any website from any computer, with any users anywhere in the world, asynchronously. What’s not to love?
Pros and cons of online tools
Crowdsourcing your usability studies may seem like a great idea until you get your first results back and realize that the wisdom of the crowd is a myth. You can get great results using online testing tools but only if you set your expectations correctly and plan accordingly.
Positive points
- Works well for short tasks that can be completed in around 10 minutes. Participants aren’t being paid enough to pay attention for longer, but this is enough time to get answers to burning questions.
- Works really well for competitor testing – test your site and one or more competitors using the same task list. Compare the results to see how well you’re doing. Cheap enough to do as a regular benchmark study.
- Widens the net for recruiting participants. Allows you to keep your local participants fresher, using them only for face-to-face studies.
Stuff to watch out for
- Most online tools are best for testing web sites rather than applications, although some allow you to upload screens or other images to create a prototype flow.
- Non-representative participants. You don’t always get a good indication of whether the participants were representative users of your site. In fact, often their comments reveal that they aren’t representative.
- Some tools need software downloads in order to work. Making participants download software leads to drop-out and if things go wrong, they may blame you or engage you in technical support issues.
- Some tools need you to add code to your site. This can be time-consuming, and prevents you from testing competitor sites.
Task-based tests with users recruited for their demographics
Tools in this category normally give you “video” playback of the task, with an audio track of comments made by participants as they think out loud. Only one (Userlytics) offers video images of the user too (sometimes useful for gauging reactions). Part of the per-user cost goes to pay participant gratuities. This figure is normally around $10, so don’t expect users to be engaged in your tasks for more than about 10 minutes.
- Usertesting.com provide a narrated video of the participant’s screen showing mouse clicks and text entry. Most users in USA, some Canada and UK. Responses “within hours.” $39/participant.
- TryMyUI is a very similar service, for $35/user.
- Userlytics a similar offering, except they include a recording from the user’s web cam and can capture the whole user desktop, not just the browser window. Can host prototypes. $59/user. Will recruit outside USA/Canada/UK for extra $
- WhatUsersDo GBP30/user, UK based. Nice list of task templates for different types of questions if you don’t feel confident creating your own.
- Gaze Hawk – recruits testers to do eye tracking studies using webcams. The output is heatmaps and gaze replays. I’ve not used them, but if this works it’s a lot cheaper than traditional eye tracking setups. Note that eye tracking heatmaps give you different and potentially more useful information than clickmaps. 10 participants cost $495.
Task-based test hosting sites
Unlike the previous list, with these tools you recruit your own participants. More hassle, but at least you know who is participating.
- UserZoom provide a raft of tools. Online user testing, surveys, card sorting and tree testing, heatmap click testing, as well as mobile (iOS and Android) testing apps. Are you sitting down? $9,000 for one year’s access to a partial suite. There are cheaper options out there.
- Openhallway.com provides a way for you to give participants tasks (“hallway tests”) and then captures video of the participant’s screen and audio of their narration. They host the video for you to watch and analyze. $49/month for unlimited tests. Other plans offer more storage space and allow download of the videos.
- Intuition HQ takes your screenshots, wireframes or sketches and lets you run single page, A/B comparison, navigation layout, or wording tasks. They give you back basic stats such as heat maps and time on task. $9 per test, and tests can be live for as long as you like.
- Verify is a platform for running multiple types of tests, including preference and click tests. $9/month starter plan.
- Usabilla give you click paths, task completion times, user comments for images you upload or URLs (they take a screen shot of the URL). They have a mobile-friendly presentation format for iPhone and iPad testing. Free 10-participant test, $49/month for 50 participants.
- Optimal Workshop have two tools specifically for creating an information architecture (card sorting) and then testing the navigation hierarchy you develop (reverse sorting). OptimalSort and Treejack are each $109 per month for unlimited studies. The limited free versions aren’t sufficiently flexible for running true studies.
Follow the actions of real visitors to your live site
Tools in this category normally require the addition of code to your site’s pages, because they are tracking behavior behind-the-scenes. Most of these tools report data as heatmaps. Some show the flow between pages.
- userfly shows you “video” of users’ interaction with your site. It’s free for 10 captures/month, their basic paid plan is $10/month for 100 captures and also allows you to download recordings. They have a beta solution that supports AJAX calls during the playbacks. The technology requires a code addition on your site
- Crazy Egg requires you to add code to your site. in return, they give you heat maps, scroll maps (how far down the page people scroll) and an overlay showing you the number of clicks per element (like Google Analytics). Most interesting is their confetti view – clicks segmented by referrals, searches, etc. $9 – $99/month (10-100 active pages, 10k-250k visits).
- Warning: Crazy Egg harp on a lot about eye tracking but they are not tracking eye movements. They claim an 88% correlation between mouse and eye movement, but their interpretation that mouse clicks give you 88% of the data you’d get from eye tracking is something written by marketing people. Take it for what it is: heat maps.
- Clicktale tracks mouse position, clicks, scroll behavior, and conversion. $99/month for 20k pageviews, playback and heatmaps (up to 990/month for more of everything). Free plan allows 400 pageviews, heatmaps of the most popular page, recording of the first 2 pages of each session.
- Clickheat is an open source heatmap generator that you install on your own server. Higher initial cost in getting the thing sorted out, but then you have heatmaps for life.
- ethn.io help you to perform intercept recruiting on your site and then direct users to a survey or other research technique. They partner with usabilla, optimal workshop, usertesting.com but you aren’t limited to those tools. Be aware that recruiting from your own site will give you selection bias.
Quick feedback on a design element
If you just need fast feedback on a single screen or flow, there are several sites that will host your images and manage the question and data capture hassle for you. Some also recruit participants for you, but there is no real demographic profiling so you get random Web weirdos.
- Navflow path and conversion analysis – how do people navigate around your site/app? Upload prototype images to the site. Variable pricing: Do studies for karma points redeemable against tests, or pay $20 per 100 responses (higher rate plans get higher priority). One of the UsabilityHub services (fivesecondtest, navflow, clicktest)
- Fivesecondtest – shows participants your design for 5 seconds, then asks for their first impressions via questions that you set. Free if you earn Karma points, or plans starting from $20/100 responses. One of the UsabilityHub services (fivesecondtest, navflow, clicktest)
- Clue – Another five second presentation tool for message recall testing. Free to create the test, they give you a link and handle responses, you recruit participants. The little brother of Verify.
- Feedback Army – show your interface to users, get their responses to questions you ask. 10 responses for $15. Can get 10 responses in a couple of hours, test is posted for 8 days max. Tests are posted to Mechanical Turk, and because reviewers choose which tests to accept, a short test (3-6 questions) works best. Geolocation to US, Canada, Australia, UK. There is NO demographic profiling, so people who respond are unlikely to be your target audience.
- Mechanical Turk – create a Human Intelligence Task. The ramp-up time means you’re probably better off using Feedback Army or similar interfaces into The Turk unless you think you’ll be doing this often.
Heat mapping for clicked areas, using uploaded images
See an aggregation of where users clicked in order to complete a task. The aggregate information is normally displayed as a heat map.
- Chalkmark – creates heat maps in reaction to guided user tasks. One of the Optimal Workshop services.
- Clicktest – heat maps/click overlays on your interface images. Free if you earn Karma points, or plans starting from $20/100 responses. One of the UsabilityHub services (fivesecondtest, navflow, clicktest).
Surveys
This space is relatively mature, because online surveys have been around for quite a long time. Of the many tools out there, these are probably some of the most popular of the cheap/free options.
- Wufoo. Free for limited responses (3 surveys at a time, 10 questions per survey, 100 participants). You find the participants. Funky but interesting survey creation interface, very customizable design to fit with your site, easy export to Excel for analysis (limited online analysis). Now owned by SurveyMonkey, but seemingly maintaining independence.
- SurveyMonkey. Free for limited responses. Well-known. Have a recruiting option to help you reach a targeted audience. Annoying number of upsells to the paid platform.
- Google Docs. Free. The form you create can be e-mailed or embedded in your site. Results are delivered as simple online analysis or in a spreadsheet for further manipulation.
- Blog software. Most of the leading blogging platforms have free plug-ins for simple one-question surveys that you can embed in your site. If your blog has good reach, this might be a suitable method to grab relatively fast feedback on a key point (with the accompanying sample bias).
Compare designs to test conversion
A/B testing tells you which of two interfaces makes you the most money. Multivariate testing takes this a step further by showing you which combination of several individual elements is the most persuasive.
- Unbounce lets you run A/B studies of landing pages, using pages built on their site from customizable templates. Also useful for lead generation. Starts at $25/month for 1000 visitors, 2 custom domains.
- Optimizely “scrapes” every element of your site’s pages and lets you re-configure their layout without changing the underlying code. Then you run a/b tests on the resulting redesigns. $19/month for 2k visitors
- Google website optimizer offers free A/B testing, multivariate testing, and conversion metrics. The trade-off is that it involves some script and tag changes to your page code.
- Loop11 offer path analysis so you can measure conversion. Although they also offer the more standard task-based testing options, they seem to want to appeal more to conversion-measuring marketers who get worried by small sample sizes because their $350 per-“project” fee includes 1000 participants (who you recruit yourself), and an unlimited number of tasks and questions.
Repurpose other tools
You might find other tools can be bent to your will if you need quick feedback with no outlay.
- Twitter, Facebook, Instagram, e-mail, other social media – be careful what you ask for, because you’ll get some weird feedback if you ask questions using these channels. Probably best as a way of recruiting users that you can then qualify before using them in online studies.
- Zurb are a design agency who have built and released several free tools to assist in the design process. A couple are listed above. They also offer:
- Axe – use your iPad to comment on web pages by scribbling and adding notes
- Bounce – takes a screenshot of any URL, allows you to add comments and send the result to someone
- Reel – upload a PowerPoint presentation, a PDF or sequential images. Create names/descriptions for each slide if you want, then get feedback (thumbs up/down) for each slide. Think about uploading a slide deck of your task flow…
- Spur – Tools for Web page critique. Point the site to a URL or upload an image. Apply seven tools that highlight design issues (blur, contrast, mirror, rotate, zoom, intersections, greyscale)
- Google Forms can be used for more than just surveys. For instance, create an animated GIF of your interface design that blanks to white after five seconds and you have an automatic message recall test. Not perfect, but passable.