Representativeness = Quality?
What Does Representativeness Mean
and How Does It Influence Quality
in Market Research?
In 1994, Der Spiegel reported on irregularities in the market research industry – dubious figures from carelessly obtained survey results.
Even then, there were calls for stricter quality controls and more diligent work. In the following 25 years, little seems to have changed. For several months now, Der Spiegel has been continuously reporting on questionable practices in the market research industry under the heading “The Market Research File” (e.g., data from never-conducted interviews, falsified results, fake samples, etc.).
At the same time, German market research institutes are engaging in media debates about methodological approaches and their statistical quality.
For example, the startup Civey claims to have found a completely new approach in market research. It presents its own results as representative, even if the underlying sample is not representative.
Since then, the topic of quality, and particularly representativeness, has moved strongly into the focus of discussion in relevant media.
In the context of quantitative market research, the term “representativeness” is very often cited as the central quality criterion. However, representativeness is, of course, no guarantee for the validity of market research results. Many other components influence this much more significantly. The focus on representativeness is far too narrow.
What Does "Representativeness" Mean?
“Representativeness” is a topic in market research primarily because it is rarely practical to collect information from all members of the population. Therefore, a subset of this population (the sample) is used, which is hoped to “represent” the population as accurately as possible.
It is often suggested that the process by which a sample is selected influences representativeness. A random sample is frequently cited as the ideal. This is a misunderstanding. A random sample can (by its very definition) also produce highly unrepresentative samples. For example, a sample might consist of 100 women, even though the population contains an equal number of men and women. Or the sample consists of 100 people from Vorarlberg, even though the population includes all Austrians.
The main advantage of a random sample is that the distribution of the sampling error can be calculated, thereby increasing confidence in the precision of an estimate. A sample does not become representative through random selection, although a larger sample reduces the probable estimation error. However, representativeness is also unlikely with other selection methods, as this would ultimately mean that the sample exactly matches the population in all characteristics of interest.
Representativeness Does Not Exist – Or Does It?
A population can be characterized by a multitude of features. However, in a finite sample, it is impossible for all these feature distributions to be equally represented. And even if they were, we wouldn’t know it: many of the characteristics of interest are unknown, and this lack of knowledge is precisely what necessitates the study.
Therefore, representativeness can only be demanded (and verified) for those characteristics that are observable in the sample and known for the overall population, such as gender, age, place of residence, or education. Quota samples ensure representativeness regarding these aspects but must, of course, be viewed just as critically, because the “correct” quotas are not known in advance. Especially for the characteristics underlying the study objective (e.g., attitudes towards new products, brand awareness), the distributions are unknown.
Even online surveys that rely on registered online samples can only ensure representativeness for those factors where the distribution in the population and the characteristic of each survey participant are known. These can then be specifically selected or weighted. Of course, online samples do not contain all conceivable groups of the total population, such as people who do not use the internet or who have not registered as potential respondents. If one of these factors shows a systematic correlation with the characteristic to be investigated (e.g., internet users are positively disposed towards a certain brand), then sample representativeness is unlikely.
When a results report refers to representative samples or representativeness, the sample type and size, the collection method, the weighting procedure, and the exact question with response categories should also be specified. This allows anyone reviewing the results to form their own judgment on whether and how representative the results are.
Are Representative Quantitative Studies Wishful Thinking?
This question cannot be answered with a general yes or no, but three aspects must be clarified between the client and the executing agency within the framework of market research projects:
Jointly define the relevant population clearly (e.g., general population aged 20 to 65, or more specifically, all individuals between 30 and 50 years old who have a particular interest in a certain product category and have purchased products from this category at least once in the last year). Without clarity about the population, no approach can be designed that even approximately considers the issue of representativeness.
Jointly clarify what role representativeness truly plays: Is the client actually concerned with representativeness, or with the precision and confidence in the results? In the latter case, (only) a random sample allows for the determination of the confidence interval, which becomes smaller (i.e., more precise) with a larger sample.
Clarity must be achieved together regarding the best channel to reach the target group (e.g., online, offline, or a combination thereof). In certain cases, it makes sense to approach participants via social networks or online panels. In all these cases (and this applies to a large proportion of studies in practice), one is far from a random sample and only representative by chance (with regard to the characteristics to be investigated).
Services
