Testing out Online Testing
I recently had the opportunity to test out some of the remote, unmoderated tools now being offered for user testing. Although I am an avid spokesperson for in person interviews, I also realized that setting up and running 1:1 interviews is time-consuming and expensive. We don't always test things we should because it takes too much time and money. That's where the beauty of the unmoderated, remote sessions come in.
For this experiment, I was actually able to schedule testing using two different tools concurrently with one-on-one interviews. This made sure that the team didn't lose any time or information during the process, and also allowed the opportunity to weigh the pros and cons of each process.
I spent several weeks looking at a number of online tools. The ones I landed on are Loop11.com and usertesting.com.
Recruiting
Loop11 allows a quantitative stab at user testing. You set up a series of questions and tasks. After each task you can ask survey type questions of the users. You can invite up to 1000 participants to view your website and subsequent tasks. You can choose demographic criteria, but the more criteria you choose, the more expensive the test is and too many criteria makes it impossible to recruit. Setting it up is a little tricky. Loop11 could take a lesson from Survey Gizmo and allow you to more easily move your questions and tasks around. my 200 person tasked with a relatively broad demographic cost around $800. Each project is $350 if you do them without a yearly package but then you add on the recruiting costs.
Conversely, usertesting.com encourages lower numbers, I tested only six individuals, but successfully recruited six Human Resources representatives which I wasn't sure would be possible. That particular demographic was not available on Loop11. Usertesting.com charges around $35 per recruit and the test cost no more than a few hundred dollars.
It's important to remember that you have a lot less control of the distribution of demographics when using the online tools, but the flipside is that you don't have to spend a week writing a screener and paying for that in both time and money.
If you think about a typical recruit for 1:1 moderated sessions, 5-10 individuals at $150-$200 apiece plus remuneration, a typical study usually runs at least $3000-$4000. Selling the online tools as a cost-saving measure is an easy win. But what about the results?
Setup
Both tests were relatively easy to set up. Loop11 allows you to work directly in their interface prior to collecting payment. This was definitely a trickier experience since I had to put in starting URL, success URL and any questions I wanted to ask on each landing. I found it difficult to keep track of and the numbering system doesn't differentiate between tasks and questions so your whole study runs together. (Ideally, each task would have a letter, and then associated tasks/questions might have a number like A1)
User testing.com was frustrating because I couldn't set up anything in their online tool until I paid and it took weeks to get through our organization's red tape to be able to set up payment. So I had to set it the test protocolup in Microsoft Word and then cut and paste when we were ready.
Results
The results are where you need to pay the most attention because the output is significantly different. Loop11 delivers much more quantifiable information. You survey/test hundreds of individuals and get large amounts of feedback on your site. We asked questions like whether people thought the site was easy to use or cluttered. In return, Loop11 graphs the responses that were delivered for each question. Loop11 also has heat maps which are great for the team to see a gave us great insight into where participants are focusing. Not only do you see the successes, but you also begin to see the other places on the page where participants clicked. Loop11 also provides click streams, but our beta was not set up in a way that this was valuable. In the future, I would push for better test URLs to be able to track the click streams. Without a live site and discrete URL's for each page - that is difficult. If you want to jump through a few more hoops, you can get audio and video from a collaboration between Open Hallway and Loop11.com. I ran out of time and energy and decided I didn't need video from 200 people. You also get a limited amount of information about each participant. I did find myself wanting a better way to export the graphs of feedback data in some way other than through a screen capture. There may be a way, but I didn't find it.
Usertesting.com delivers raw video along with demographic information, browser information and other key statistics for understanding your participants. The videos I got back were no longer than about 12 min. and I was able to watch them all and take notes within about 90 min. After you take notes in the area provided on the website, you can click a button to export and immediately get a spreadsheet outlining all the demographic data, the answers to the 4 written, open-ended qualitative questions at the end of this test as well as all of the notes you took as a researcher. It's a pretty sweet way to see everything all at once and the dream compared all of the cutting and pasting many of us do after a series of interviews. Because this is a much smaller set of participants, the data is clearly not quantifiable, but in some ways it is a bit richer.
Recommendations
Both of these tools have significant merit for use at the right time in development. Loop11.com seems like a preferred tool for when you have very focused questions and want to understand some limited and specific behaviors on your site and make sure people are finding what they need.
Usertesting.com is probably closer to a real world, 1:1 moderated session. You are provided with conversational feedback that is valuable a little further forward in the process when you're still making decisions and trying to see how people think when using your site.
The best thing I found in doing this is that the feedback was corroborated across platforms. The team was afraid of that we would get one set of feedback online and one set of feedback in person. I'm happy to report that both the online users (not typical users of our service) and the 1:1 participants (known users of our service) all had feedback that lined up and pointed in the same direction.
For our organization, I hope to be able to use these tools in the future to be able to include the voice of the customer more consistently and for a lower cost; not at the expense of talking with individuals 1:1, but in addition to that as touch points during the development process that keep us agile.
For this experiment, I was actually able to schedule testing using two different tools concurrently with one-on-one interviews. This made sure that the team didn't lose any time or information during the process, and also allowed the opportunity to weigh the pros and cons of each process.
I spent several weeks looking at a number of online tools. The ones I landed on are Loop11.com and usertesting.com.
Recruiting
Loop11 allows a quantitative stab at user testing. You set up a series of questions and tasks. After each task you can ask survey type questions of the users. You can invite up to 1000 participants to view your website and subsequent tasks. You can choose demographic criteria, but the more criteria you choose, the more expensive the test is and too many criteria makes it impossible to recruit. Setting it up is a little tricky. Loop11 could take a lesson from Survey Gizmo and allow you to more easily move your questions and tasks around. my 200 person tasked with a relatively broad demographic cost around $800. Each project is $350 if you do them without a yearly package but then you add on the recruiting costs.
Conversely, usertesting.com encourages lower numbers, I tested only six individuals, but successfully recruited six Human Resources representatives which I wasn't sure would be possible. That particular demographic was not available on Loop11. Usertesting.com charges around $35 per recruit and the test cost no more than a few hundred dollars.
It's important to remember that you have a lot less control of the distribution of demographics when using the online tools, but the flipside is that you don't have to spend a week writing a screener and paying for that in both time and money.
If you think about a typical recruit for 1:1 moderated sessions, 5-10 individuals at $150-$200 apiece plus remuneration, a typical study usually runs at least $3000-$4000. Selling the online tools as a cost-saving measure is an easy win. But what about the results?
Setup
Both tests were relatively easy to set up. Loop11 allows you to work directly in their interface prior to collecting payment. This was definitely a trickier experience since I had to put in starting URL, success URL and any questions I wanted to ask on each landing. I found it difficult to keep track of and the numbering system doesn't differentiate between tasks and questions so your whole study runs together. (Ideally, each task would have a letter, and then associated tasks/questions might have a number like A1)
User testing.com was frustrating because I couldn't set up anything in their online tool until I paid and it took weeks to get through our organization's red tape to be able to set up payment. So I had to set it the test protocolup in Microsoft Word and then cut and paste when we were ready.
Results
The results are where you need to pay the most attention because the output is significantly different. Loop11 delivers much more quantifiable information. You survey/test hundreds of individuals and get large amounts of feedback on your site. We asked questions like whether people thought the site was easy to use or cluttered. In return, Loop11 graphs the responses that were delivered for each question. Loop11 also has heat maps which are great for the team to see a gave us great insight into where participants are focusing. Not only do you see the successes, but you also begin to see the other places on the page where participants clicked. Loop11 also provides click streams, but our beta was not set up in a way that this was valuable. In the future, I would push for better test URLs to be able to track the click streams. Without a live site and discrete URL's for each page - that is difficult. If you want to jump through a few more hoops, you can get audio and video from a collaboration between Open Hallway and Loop11.com. I ran out of time and energy and decided I didn't need video from 200 people. You also get a limited amount of information about each participant. I did find myself wanting a better way to export the graphs of feedback data in some way other than through a screen capture. There may be a way, but I didn't find it.
Usertesting.com delivers raw video along with demographic information, browser information and other key statistics for understanding your participants. The videos I got back were no longer than about 12 min. and I was able to watch them all and take notes within about 90 min. After you take notes in the area provided on the website, you can click a button to export and immediately get a spreadsheet outlining all the demographic data, the answers to the 4 written, open-ended qualitative questions at the end of this test as well as all of the notes you took as a researcher. It's a pretty sweet way to see everything all at once and the dream compared all of the cutting and pasting many of us do after a series of interviews. Because this is a much smaller set of participants, the data is clearly not quantifiable, but in some ways it is a bit richer.
Recommendations
Both of these tools have significant merit for use at the right time in development. Loop11.com seems like a preferred tool for when you have very focused questions and want to understand some limited and specific behaviors on your site and make sure people are finding what they need.
Usertesting.com is probably closer to a real world, 1:1 moderated session. You are provided with conversational feedback that is valuable a little further forward in the process when you're still making decisions and trying to see how people think when using your site.
The best thing I found in doing this is that the feedback was corroborated across platforms. The team was afraid of that we would get one set of feedback online and one set of feedback in person. I'm happy to report that both the online users (not typical users of our service) and the 1:1 participants (known users of our service) all had feedback that lined up and pointed in the same direction.
For our organization, I hope to be able to use these tools in the future to be able to include the voice of the customer more consistently and for a lower cost; not at the expense of talking with individuals 1:1, but in addition to that as touch points during the development process that keep us agile.
Comments