Keep in touch, subscribe to our newsletter
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
→ Are 5 users enough?
→ Number of tests insights
There is no doubt that user testing has proven to be insightful. It is an effective manner to receive quick insights into the user experience of an interface’ design or digital product, or concept in general. Whereas the terms user and usability tests are often used interchangeably, there is a difference. In short, usability tests are a necessary step to ensure you are building an effective, efficient and a pleasant experience. Whilst, user tests are often believed to be the process of confirming the demand for a product or service. At Spark Works due to the nature of our work, we mostly focus on the latter. We generally use user testing as a way to obtain valuable feedback in regards to the key value proposition(s) of a concept. Essentially user tests serve as an ideal tool to validate early stage concepts and ideas for digital products and/ or services.
These past few months we have ran several interesting user tests which proved useful in both testing and improving initial concepts and designs. First, last month we launched our new website, for which we carried out small scale user tests with our team, partners and close customers. This resulted in clear improvements for the website’s UX and overall visual concept and design. Check out our new website design here.
Additionally, for one of our latest projects with a motorcycle manufacturer, we were asked to support with the creation of a new mobile-based customer experience. The objective was to centralise different features and additional services in one platform.
We tested this mobile-based concept in two ways. First, we co-created the initial concept together with the internal stakeholders through collaboration and continuous feedback. Due to the current pandemic, we had to do so virtually. This was challenging at first, but in the end produced great results. Second, after further developing the initial outcome and refining, we tested the new mobile-based product with potential users. However, instead of randomly choosing a sample of 5 - 10 users, this client specifically asked us to base our test sample on four pre-identified personas in two different markets. This resulted in a group of twenty test users - each persona represented equally by five participants - who underwent the testing phase.
Although it is widely accepted that five participants suffice for gaining trustworthy and insightful knowledge about your digital service or product design, and while it is clearly beneficial from a budget standpoint, from our point of view five users are not always providing enough insights. The way we see it, the “5 users is enough” assumption can be challenged on two points:
1. Five users are not always ensuring enough diversity in the sample test group
2. Number of tests depends on the stage of the project and the objective
The two points can be explained as follows: In the aforementioned case there was the request to test with four pre-defined personas. In theory, this means that there are four different types of target users. Nonetheless, each of these pre-defined user groups still require some internal variety in order to validate the user feedback. Consequently, based on the mobility case, we could conclude that even if there was a variety of personas, you should still ensure that there is a large enough sample size per persona, in order to receive a diverse and satisfactory set of user insights. For this case, an ideal testing group size was between 12 - 20 users.
Second, from our point of view, it is important to determine what the is objective of the project phase we are in. Again, in our case the tests were meant to validate the value proposition of some very early value proposition concepts and their specific features. As a consequence, the user tests were rather exploratory and served as a tool to ascertain the direction of the project. In other words, the tests were part of the testing phase, instead of the implementation phase.
So in this early stage of the project, in order to best define the most valuable offering, we recommended not only user testing, but also running so called Smoke tests, to assess interest at scale through a mockup website powered by Google Ad Words. The results would provide not just details on specific features and user lead recommendations, but a clear quantitative analysis on interest by users at different levels of commitment to the new value proposition.
Therefore, the question remains, “Are five users enough to do a user test?”, and your answer will most likely lean towards, “it all depends on the context of your user test.” But then again… We are consultants after all :)
Source: Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379-383.
Source: https://www.hotjar.com/usability-testing/
IMG Source: Jakob Nielsen
