Usability testing 101

This document provides a detailed overview of usability testing, focusing on its essential components and best practices. It covers;

1. The core elements of usability testing

2. Types of usability testing

3. The importance of well-structured task scenarios

4. “Think out loud” method

5. Post test questions

Core elements of usability testing

Usability testing revolves around three key elements

1. The facilitator
   Guides the session, delivers tasks, and observes without bias. A skilled facilitator ensures participants feel at ease, enabling honest and natural interactions.

2. The tasks
   Realistic, actionable, and aligned with user goals. Tasks mimic real-life scenarios, like paying a bill or reviewing transactions, revealing how users truly engage with the product.

3. The participant
   The heart of the test. The participant should be a realistic user of the product or service being studied. That might mean that the user is already using the product or service in real life. 

Alternatively, in some cases, the participant might just have a similar background to the target user group, or might have the same needs, even if he isn’t already a user of the product.

Types of usability testing

Qualitative vs. quantitative

Usability testing can be either qualitative or quantitative.

• Qualitative usability testing focuses on collecting insights, findings, and anecdotes about how people use the product or service. Qualitative usability testing is best for discovering problems in the user experience. Qual testing involves a small number of users, usually 5–8 users is recommended. 

• Quantitative usability testing focuses on collecting metrics that describe the user experience. Two of the metrics most commonly collected in quantitative usability testing are task success, number of errors and time on task. Quantitative usability testing is best for collecting benchmarks. Quant usability testing is based on a large number of participants, often more than 30. 

Task scenarios

Task scenarios in usability testing are realistic activities participants might perform in real life. They provide the necessary context for the task, ensuring participants understand why they are engaging with the interface, mimicking real-world use cases.

Poorly written tasks often focus too much on forcing users to interact with a specific feature, rather than seeing if and how the user chooses to use the interface. A scenario puts the task into context and, thus, ideally motivates the participant.

Examples;

1. Personal finance management

   • Poor task - Transfer $500 to your savings account using the "Quick Transfer" feature.

   • Better task - Imagine you’re saving for a vacation and want to move $500 into your savings account. How would you do that?

2. Loan application

   • Poor task - Apply for a personal loan of $5,000 using the app's loan section.

   • Better task - You need to borrow $5,000 to cover an emergency expense. Explore the app and apply a loan that matches your needs

3. Investment goals

   • Poor task - Use the app to invest $1,000 in a mutual fund.

   • Better task - You’ve decided to start investing $1,000. Use the app to find a suitable mutual fund that aligns with your long-term goals.

What are the key components of task scenarios?

1. Realism - Create tasks that participants might perform outside the test environment.

2. Context - Situate the task in a short scenario to motivate users and make the interaction feel natural.

3. Actionability - Tasks should prompt participants to take actions.

4. Neutrality - Avoid giving hints or priming participants to follow specific steps or use certain features.

Why context matters

Scenarios help participants relate to the task, engage more naturally, and reveal genuine behaviors. For instance:

• Poor Task: View your account balance.

• Better Task: It’s the end of the month, and you want to check if your salary has been deposited into your account. 

This shift from instruction to contextual storytelling ensures the participant interacts with the interface as they would in real life.

The Importance of user goals in creating scenarios

User goals serve as the foundation for designing effective task scenarios. They define what users aim to achieve and provide the necessary context for writing realistic, actionable, and unbiased tasks. 

Without clearly identified user goals, task scenarios risk becoming abstract, irrelevant, or overly directive, failing to replicate real-world interactions.

User goal - Check recent transactions.

• Poor Task - Locate the "Transactions" tab and view your last 5 transactions.

• Better Task - You’re reviewing your spending habits for the month. Find your most recent transactions and identify any large expenses.

“Thinking out loud” method

Participants are often asked to think out loud during usability testing (called the “think-aloud method”). The facilitator might ask the participants to narrate their actions and thoughts as they perform tasks. The goal of this approach is to understand participants’ mental model and why they are behaving in a certain way. 

Downside of this
Unnatural situation. Most people don't sit and talk to themselves as they perform such tasks. This makes it hard for test participants to keep up the required monologue. But, try encourage them to. 

Basic rules of good experiment design

External validity

External validity ensures that the study mirrors real-world conditions and involves participants representative of the target audience.

Eg. Testing a fintech app designed for small business owners by recruiting corporate employees instead of actual small business owners lacks external validity. The two groups have different needs, behaviors, and constraints.

or

Evaluating a mobile banking app on a desktop simulator undermines external validity because real users would access the app on their smartphones. 

Internal validity

Internal validity ensures that the study setup is fair and does not introduce biases that could influence outcomes.

Eg.

Testing a loan application process on two different designs, where Design A is tested with experienced participants and Design B with novices, creates a bias. Differences in user expertise, not the design, could explain the results.

or

Comparing two mobile app prototypes but conducting one test in a quiet room and the other in a noisy environment may skew results. The environment could impact user performance and satisfaction rather than the app design.

Questions to ask after the usability test

Post-test questions provide valuable context and insights into the participant’s experience. 

Examples include;

• Overall Experience - How easy or difficult was it to complete the tasks?

• Pain Points - Were there any parts of the process that felt confusing or frustrating?

• Likes/Dislikes - What did you like or dislike about the interface?

• Suggestions - Is there anything you’d change to make the experience better?

• Comparison - How does this experience compare to other similar apps or tools you’ve used?

• How would you rate the difficulty of using this app on a scale? (1 = very difficult, 5 = very easy)

• On the scale from 1 to 5 (1 = very satisfied, 5 = very dissatisfied) how would you rate your experience with the product?

These questions help uncover user perceptions, validate observations, and gather ideas for improvement.

Enhancing usability testing with additional methods

While usability testing provides critical insights into how users interact with a product, combining it with other research methods offers a deeper, more comprehensive understanding of user behavior and design effectiveness. These complementary methods address different aspects of the user experience, helping to validate assumptions, uncover issues early, and improve the final design.