May 2010

Is Your Quantitative Research Statistically Robust?


Dear Reader

When you need objective, statistically-robust information about your product, service or market, quantitative research is often called for. As its name suggests, it's used for measuring things – who's buying your product or service, for example, and why. And, perhaps even more important, who's not buying and why not, and who's buying from your competitors.

Whatever you need to find out, this edition of the newsletter offers helpful pointers to make sure your data is as robust as possible. And don't forget – at Research Insight, we don't just gather data; we analyse and interpret it too, to ensure you get the maximum value from your investment.

Happy reading!


Martin Holliss

t: +44 1235 812 456
m: +44 7931 376501

So what is the key to robust quantitative research results?

How many interviews do you need?
It is rare that you will need more than 1000 interviews of any one type of respondent (delivering data accurate to +/- 3%), and often 400 is fine (delivering data accurate to +/- 5%). A common misconception is that the number of interviews you need depends on the size of your market or ‘universe' – ie the number of people of the type you are interviewing. In fact whether your universe is large (eg people who watch television) or small (eg Aston Martin owners), the number of interviews needed for robust data is the same.

How do you get a representative sample?
Make sure the sample you use is representative of your universe. If your sample is not representative, it's most unlikely that the survey data will be either. So, for example, if your universe is the population as a whole – 51% of which is female – you need a similar percentage of women in your sample. Also, ensure that your sample is drawn randomly from the universe.This will ensure it is representative.

How do you get enough interviews in a subgroup?
If you need to break down your sample into different subgroups, e.g. different age bands, quotas can help you ensure that data is representative or that you have enough of each type of respondent for valid subgroup analysis. A simple rule of thumb is to conduct at least 100 interviews in each subgroup. For example, results between two subgroups, each with 100 respondents, must differ by 14% before they can be claimed to be significantly different. Results between two subgroups, each with 400 respondents, must differ by just 7% (ie half that for 100 interviews) before they can be claimed to be significantly different.

How do you prevent bias in your subgroup data?
At this point you may be thinking: “If quotas are set, won't the results be biased?” Yes, they will, which is where another statistical technique – ‘weighting' – is useful. Put simply, this moulds survey data back into a representative profile, which makes the resulting percentages meaningful. There are several different weighting techniques (eg ‘Rim Weighting') and some pitfalls, which is why you should only entrust your research to an agency with a proven record in this area.

How do you identify similarities/differences between subgroups?
Crosstabulations are invaluable when analysing survey data – and well worth the modest additional cost. As well as providing topline results, they also highlight similarities and differences between subgroups. For example, crosstabs would enable a confectionery manufacturer to see not only how many bars of chocolate men eat per week compared to women, but also whether there are differences in the amounts consumed by men and women of different ages. And one final tip: do make sure your crosstabs have stat tests built in – these will highlight subgroups with significant differences and help you draw statistically – robust conclusions.


To download a free ‘statistical significance test' spreadsheet (which is also a sample size calculator), click here

And if you'd like to find out more about quantitative survey samples, and much more besides, you can download our 25-page Market Research: What's it all about? introduction by clicking here

What else do we do?

Everything we do is tailored to our clients' precise need. So, as well as quantitative research, we have a range of other research techniques in our toolkit. Here are just three of them:

  • Desk research – ideal when you want to build up an overall snapshot of your market size/trends, or of your main companies/competitors, whether focusing on a single country or international/export markets.
  • Executive depth interviews – these work well if you have high-value, senior-level customers. We establish an atmosphere of trust, which encourages your customers to share their concerns and suggest solutions… in a wholly positive manner.
  • Focus groups – a common qualitative technique designed to explore those all-important What, Why, When, How, Where, and Who questions. Focus groups are perfect for uncovering and exploring a rich seam of views, opinions, perceptions and experiences.

Please email Martin Holliss or call him on +44 1235 812456 to find out more.