“Five users are enough”. This isn’t just any statement, but one of the most embraced tenets of user experience (UX) research over the past couple of decades. The statement remains true even today, particularly in qualitative UX research, as industry experts are convinced that five users are sufficient to uncover around 85% of usability issues. While the ‘five-user rule’ is predominantly accepted due to its efficiency, cost benefits, and practicality, product development has evolved rapidly since Jakob Nielsen, founder of Nielsen Norman Group, introduced this term in 2000. So, are five users enough?
Despite early reluctance and skepticism from UX development teams, remote testing has quickly become a common sight and widely accepted approach in the UX world. That said, UX teams still find themselves in an age-old dilemma: what is the ideal sample size for a usability test?
If this scenario sounds familiar to you, look no further. UserQ helps you determine the sample size for different remote usability tests, depending on your objectives, budget, etc.
Determining the sample size: why are UX teams confused?
“We are developing a huge website/application.”, “Our product has many features. “We are targeting an array of user profiles”.
These are some of the statements often thrown around to justify testing with larger user groups. In reality, larger doesn’t necessarily mean better.
UX teams are often confused while picking the right sample size due to several reasons listed below.
Test with five users vs larger sample size
The five-user test, due to its acceptance and familiarity, is the go-to usability research among UX teams. However, other sources recommend testing with a larger sample size, leading to confusion and uncertainty over the right approach.
In-person vs remote testing
Some product teams, especially inexperienced ones, aren’t able to differentiate between moderated and unmoderated testing environments and underestimate the way users approach usability tests in a remote and lab setting.
Qualitative vs quantitative
It is crucial to define whether you are looking for qualitative or quantitative insights. If the goal is to understand the rationale behind understanding user behaviors, qualitative research is the way to go. However, if the objective is to know how many users behave in a certain way, quantitative research is ideal.
However, UX teams aren’t able to clearly define this, leading to uncertainty about the right approach to conduct usability tests.
Why sample size matters in UX usability research?
The sample size of a usability test is determined by five major factors:
- Objective
- Type of remote test/research methodology
- Size to gain statistical confidence for quantitative research
- Complexity of the product, topic, and design-related issues
- Project’s budget
One of the key reasons why the sample size matters in UX research is to evaluate the credibility of UX research insights. However, choosing between a small user group and a larger sample size typically depends on whether you want to validate a few design-related decisions or address broader usability issues.
If the objective is to validate specific design decisions, a smaller group may be sufficient. However, if you are looking to understand broader usability issues backed by data, it is always a good idea to opt for a larger sample size with diverse personalities.
An ideal sample size will help you validate assumptions and discover significant usage patterns, which, eventually, help you make data-backed product-related decisions. Ultimately, getting the sample size right allows you to maintain a balance between your objectives and budget while making UX enhancements.
Rethinking sample sizes for remote UX tests
Until recently, product teams across the world believed that usability tests with a relatively smaller sample size (5-8 users) is enough to determine usability issues. However, the momentum is gradually shifting toward remote UX testing, which also means greater variability across the number of participants, cultural background, testing environments, and devices.
This, increases the chances of inconsistent behavior, which means that a larger sample size is ideal to uncover accurate insights across a broader user base. In short, while a smaller sample size may yield optimal results for lab-generated tests, a larger sample is well-suited for remote UX tests. Luckily, there are several tools that facilitate smooth and frictionless remote UX testing.
Benefits of using UserQ tools for remote UX testing
Let’s put the spotlight on why platforms like UserQ are perfect for remote tests.
Swift turnaround
Users can create tailored remote tests depending on their goals, deliver them to participants, and gather results within hours.
Cognisant of MENA’s regional diversity
There are 19 countries in the Middle East and Africa region with people from different ethnicities and cultural backgrounds. UserQ’s remote testing tools are designed to take into account the region’s diverse user base and vast area, allowing you to reach users across MENA.
Affordable
In-person is expensive than remote testing due to the various costs involved, including participant recruitment, renting the lab space, dedicating resources, and time to observe and facilitate tests. While in-person testing is not only logistically inconvenient, it also hinders your team’s bandwidth.
On the other hand, you can easily run tests with 10, 20, 30, 50 or even 100+ participants remotely, without worrying about in-person recruitment and moderation, making it an affordable alternative.
Getting the sample size right for different types of remote tests
It is worth noting that there are plenty of remote tests to choose from, and it is crucial to find answers to two fundamental questions: when should I use a certain remote test, and what should be the sample size?
Find all the answers in this section.
Just a heads up, start by identifying larger issues, and continue to refine your product as you move toward its launch.
Test type | Recommended sample size | When to use it |
Prototype testing | 5-10 users | Initial stages of usability testing and detecting major issues immediately |
Five-second testing | 10-20 users | Evaluating the first impressions and visual hierarchy |
First click tests | 10-20 users | To determine whether users are navigating to the right place |
Preference testing | 20-50 users | Understanding visual and design choices |
Tree testing | 30-50 users | Identifying issues in menu structures and navigation logic |
Card sorting | 30-50 users | Validating content grouping and information architecture design |
Surveys | 100+ users | Acquire quantitative insights and understand behavioral data |
Choosing between 5 or 500 test participants: use case scenarios
Now that you have a fair understanding of the different remote UX tests, the ideal number of participants, and when to use each test, let’s explore some use case scenarios to help you determine whether you need 5 or 500 test participants.
Go with 5-10 users when
A small sample size of 5-10 users is typically recommended when the focus is on exploratory research and early-stage and moderated testing sessions. If you are looking to validate a new navigation flow or an early wireframe, it is a good idea to work with a smaller sample size, as you can find a majority of usability issues before venturing deeper into development.
Additionally, choosing a small sample size works well when UX teams want to test and make frequent design tweaks to enhance a particular feature of the product.
Go with 30-50 users when
UX teams should consider expanding the sample size of their study if they want to uncover data-backed patterns. For example, determining which call-to-action (CTA) button text resonates more with users.
A sample size of 30-50 users is also ideal when UX teams want to understand preference trends and information architecture structures, including how users interact with new categories or navigation menus.
This sample size is sufficient to get a fair idea of how users across the MENA region view or perceive the product’s terminology, layout, interface, etc.
Go with 100+ users when
Testing with 100+ users is perfect for surveys and conducting product-market fit analysis. The idea here is to analyze the responses from different types of users, which are typically segregated by age, region, language, and other factors. For example, comparing responses of the millennial population in Qatar and the UAE, understanding the language preferences of users in Egypt, the UAE, and Bahrain, etc.
Final words
It is clear that although the five-user rule is relevant even today, the arrival of remote UX testing is proving to be a game-changer in the UX world. Remote testing has not only compelled UX teams to rethink their user testing strategy, but also helped them save resources and optimize their product development budget.
The sample size of a remote UX test depends on several moving parts, including audience segments, research goals, budget, and complexity of the product and this is where UserQ enters the scene.
Platforms like UserQ allow UX teams to scale their remote testing efforts efficiently without overstepping budgets while reaching their target audiences across the MENA region. The testing tools serve as stepping stones and a smart framework to determine the right sample size and therefore, gain accurate user and behavioral insights.
Related Post
The benefits of user research
User research is typically a key part of product development and business growth all around the world, but here at
Best practices for conducting remote
In this guide, we’ll let you in on UserQ’s top tips and tricks to utilise our selection of remote user
Start, Grow, Lead: A Guide
A career in which you can make a real difference to the everyday lives of customers and clients alike is
Subscribe to our
product newsletter!
Receive emails about UserQ updates, new features,
offers and latest trends.
