The Return on Usability Testing

By Subha Subramanian

Usability testing is the qualitative measurement of user experience with a product or service, such as a website. Usability testing is for anyone who wants to make their website, software, device, etc. easier to use. Whether you employ thousands of people such as Microsoft or fall into the small business category, usability testing can play a significant role in giving you direct feedback from your customers.

In a typical usability test, users representative of the target demographic interact with the design as one or more people observe and take notes. The goal is generally to identify any obstacles the user may encounter by measuring task completion rates, problem areas, and user satisfaction. Usability has been a buzzword of the past few years; however, not everyone has jumped on the bandwagon.

For many companies, one of the reasons for not performing usability testing is the cost. One-way mirrors, cognitive psychologists and usability experts—these things can add up quickly to a big number on the debit side of the balance sheet. But is there a way to measure return on usability testing?

Costs

First, what are the costs associated with usability testing? The factors that affect the cost of usability testing are the type of testing (formal or informal), the number of test participants, the area of the site to be tested (entire site vs. smaller content areas), the complexity of the project and the number of tests you do throughout the lifecycle of the project.

Usability testing consists of three parts—Planning, Testing, and Analysis & Recommendations. Each of these has costs associated with it such as the cost of the time spent by the individual or the team, equipment and facilitites costs and payment in cash or kind to the participants. Consider a testing scenario where you test 8 users and pay them $100 each for an hour of their time. Placing an ad on craigslist will set you back $75. Add the cost of the time spent drafting the test plan and recruiting. Say rent on equipment and facilities for a two-day period comes to $3,000. Factor in the cost of a moderator and notetaker for the 8-10 hours of testing. Recommendations and analysis may take another 80 hours. The total bill? Around $16,000. Two such tests will set you back by around $32K. Steep? Maybe. But what are you getting in return?

Return on Investment

While usability testing just makes good sense, it can be measured in terms of a return on investment. ROI can be calculated in multiple ways—how many people use a particular feature, how much time they spend on your website, how many sales are closed, etc.—all particular to your website. The question is: what do you want to measure?

Goal-specific ROI

Goal-specific ROI measurement involves setting a goal, and benchmarking before and after the redesign.

Let's say the goal is to reduce calls to the support team. A usability test can point out glaring problems with the site. Fixing those problems will reduce the number of calls to the support team, thus reducing overhead. The calculation for the ROI in this case is the cost per call (the number of original calls to support - the number of calls after implementing the changes).

Task-specific ROI

One way to measure ROI for a task (as suggested by Claire-Marie Karat) is to measure the time taken by users to perform the task, such as calculating mortgage for a $500,000 home using the online mortgage calculator (on a bank website). Calculate the average time spent by users performing the task before usability testing (t= 150 seconds, say). Test again after the recommendations from the usability testing are incorporated, say for an improved time of t=115 seconds. Find the time saved per user. For this example it comes to 35 seconds per user on average. Assuming you have 10,000 hits per day — that's 350,000 seconds a day that you have saved. Multiply this value converted to hours by the average hourly salary ($40 for this example) of the people visiting your website. Calculate this amount for a year (say 300 working days a year). Compare this final amount ($1,166,666.66) to the cost of usability testing. As in this example, chances are you didn't nearly spend a quarter of that amount on usability testing. This is particularly relevant for an intranet site where time saved directly equals money saved.

Another way of measuring ROI for a task such as online purchasing can be done by measuring task completion rates prior to and after the redesign based on usability testing.

"You can increase sales on your site as much as 225% by providing sufficient product information to your customers at the right time. In our recent research, we found that the design of product lists directly affected sales. " (UI Engineering, 2001)

Each of these methods requires that data for the current system is available for comparison against data that will be gathered in the future.

Other Benefits

Usability testing can identify major showstoppers prior to launch or settle differences of opinion cheaper and faster than endless cyclic meetings. It can also tell you which features support the user experience and which ones detract from a satisfactory user experience.

Even if you do not perform ROI calculations, rest assured that usability testing would indeed bring you rich returns such as increased use and increased customer satisfaction and loyalty. Don't forget about word of mouth advertising as satisfied customers relate their experiences to others — the very best form of advertisement. There are three pairs of shoes from Zappos.com worth $399 in my closet that bear testament to the power of this kind of advertising.

Observing the user experience is the most direct kind of feedback you can get from your customers. To paraphrase MasterCard, usability testing: $16,000 dollars. The ability to get feedback from actual users: priceless!

Additional Reading: http://www.rashmisinha.com/useroi.html