From N=40 to N=1000

A lot has changed and improved in the past 7 years at WUA!, in the area of research. We went from 40 respondents in our own research lab to 200 respondents in an online research environment. But that’s not all…

These days, WUA! conducts research with a minimum of 200 respondents per device, and that research takes place online. For the study of health insurance policies in The Netherlands, we recently announced that we’re increasing the number of respondents to 1000.

A validated marketing model

For us, the research we conduct online is the key to adding value for our customers, and that value mainly comes from the WUA! Benchmark. We believe it isn’t about what your customer thinks about you or your competition, it is about what the customer thinks about you relative to that competition. At WUA! we call this “Think Outside Out”. We’ll get back to that 😉

In the past, before 2008, at WUA! we carried out the occasional study with the five of us, one by one we went online to search with the same online orientation assignment in the back of our heads, in order to see which providers we would end up with. Those early experiments laid the foundation  for the WPS model. Based on customer feedback, we continued to improve our solutions, and based on our online DNA and the visions of all online professionals that are working for WUA! we designed our own models. Those models were tried, tested, and validated by the best researchers in The Netherlands, in order to ensure that we’re really comparing apples with apples in our growing benchmark.

Our WPS model has a predictive value of over 70 percent, the WUA! benchmark is the largest worldwide website database with over 12,000 reviewed websites and apps. And the world needs to be conquered even further: currently WUA! are active in 10 countries, and plenty more need to be added to that list!

The WUA! Digital Sales Scan Model

From n=40 to n=1000

At WUA!, we’re always on the move, online innovation and entrepreneurship are in our blood and also for us, “good” isn’t good enough. Together with our customers we’re building on the most extensive, high-quality performance benchmark in the world to guide our customers in their roadmap to Digital Excellence. Every study we carry out makes our benchmark better, and our customers benefit from that on a daily basis.

Over the past few years we’ve learned that increasing the number of respondents (from n=40 to n=200 as a standard) in our benchmark studies contributes in a positive way to the quality of the online orientation studies we deliver. At the same time, the law of diminishing returns applies: for the Dutch market, we don’t need to move to 10,000 respondents per study right now in order to be able to fully map the market and guarantee a good benchmark. That would be too costly anyway…

The 200 respondents review 4 websites per study, that is part of the orientation assignment. So that’s a total of 800 reviews. Statistically, those 800 reviews provide a decent picture of the total market and the comparisons to the most important players in that market. Think about what happens when n=1000. Then you’re talking about 4000 unique website reviews. On average, 1 website review consists of 75 answers, which means we’re talking about 300,000 answers per study!

Two reasons to increase

1 Further away from the lower limit
At WUA! we regard 20 reviews as the statistical lower limit. In practice, this means that in a study with 200 respondents, a provider needs to be found by 5 respondents we give 4 reviews each. If we increase the number of respondents, we get a more qualitative feedback about the websites found, and we are much further away from the statistical lower limit of 20: best practices will then surface and become “harder”.

2 Acquisition and retention capacity of digital touchpoints
The second reason for increasing the number to 1000 (or more), is that with a high number of respondents, you can make a distinction between the digital experience of customers and non-customers. Then you can also make a prediction that is indicative of the website’s acquisition or retention capacity. We find that really interesting. In order to be able to do this, you need a larger number of respondents for every provider you want to make statements about. Imagine what this means for research in a “switch-sensitive” market like health insurance and in the telecom industry!

Example: Health insurance

So our study of health insurance policies in The Netherlands is an example of a study we increased the number of respondents for. Standard we used to carry out this study with 700 respondents already. A large player like Achmea, for instance, is found more often with different labels during online orientation studies than, say, DSW or HEMA.nl. Not illogical, brand awareness plays a role, and so does the size of the SEA budgets in Google to send traffic to online domains for a fee, of course.

For the healthcare market it is very clear that the market of providers is so extensive that it is worthwhile to not just look at what the number 1 is doing, but also at what the number 15 yields in best practices. Therefore, it is logical that we started working with more respondents, so that companies outside the top 5 are sufficiently reviewed as well. Additionally, when researching health insurance, it is very interesting to be able to look at the acquisition and retention capacity of the websites of various providers…

The WUA! benchmark as a KPI

At WUA! our ultimate goal is to establish our benchmarks of online marketing and service departments as a KPI. We know that some companies are already doing this: online teams are given the task to gradually improve the Digital Sales Score of the online domain, and this is strongly encouraged by the management. If our research is applied this structurally, we owe it to our customers to provide the best benchmark in the world. We believe it is likely that the number of respondents per WUA! study will only increase.

For if we look back at the past 8 years, we see that the quality of online domains increases and that we thus need more details (statements from respondents) in order to conduct decent research. Additionally, internet behaviour is becoming more personal and target audience specific measurements are becoming increasingly important, think about selections based on postcode, gender, age, et cetera. These types of measurements need larger numbers of respondents too…