Quality in Quantitative Market Research
A Few Ideas for Quality Assurance
“The Market Research File” (Spiegel) is on the table. For several months now, reports of manipulation in the field of market research have been piling up. As described in the last blog post, the term representativeness is frequently mentioned in this context as an indicator of quality. We have already discussed various misunderstandings regarding “representativeness” and today we want to address specific process steps in quantitative market research and present ideas for quality assurance in each case: (1) in the conceptual design, (2) in the methodological approach, (3) in the questionnaire design, and (4) in the analysis, interpretation, and preparation of the results.
Quality in the Conceptual Design
Surprisingly often, the objectives of market research are not clear to the clients themselves. They find it difficult to name their decision-making problems and derive specific information needs from them. There is also a lack of a concrete idea regarding the required accuracy and the population relevant to the objective. However, clarity about goals and information needs is crucial for all further planning steps in a market research project.
We repeatedly experience how important this phase is and how often attempts are made to save time here when under time pressure. We also frequently observe that companies have already decided on a specific method without first clarifying fundamental issues (e.g., objectives and decision-making problems, information needs). Therefore, it is important to us to have clarity about the actual problem and objectives before starting market research. The goals must be discussed, questioned, and approved together with the client in the initial meeting. Only this clarity allows for the derivation of research questions through to the selection of suitable methods and the design of the survey instruments.
Quality in the Methodological Approach
To ensure quality, we decide which type of data collection makes sense for the specific case, depending on the research question and the population. In the case of quantitative survey designs, it is also clarified how the target group can best be reached (e.g., online, in writing, by telephone, or a combination).
In the next step, we determine the sample sizes, and together with the client, it is decided whether usable internal data (e.g., customer database or other internal contact data) is available or whether external sources must be used (e.g., purchase panels, acquisition via web banners, Facebook, MTurk, etc.). Depending on the information needs and the population, it may be necessary to restrict the sample and pre-select suitable participants using screening questions.
Whether an anonymous survey is more effective and thus delivers better results cannot be answered in general terms, but depends on the objective. However, the GDPR is increasingly forcing the client to be identified at the beginning.
Quality in Questionnaire Design
In our opinion, a high-quality questionnaire must meet the following requirements:
The length of the questionnaire and the type and number of questions must take the target group and survey situation into account (e.g., online survey at a trade fair vs. written questionnaire at home). The questions must be formulated clearly and understandably, and redundant questions must be avoided. Open-ended questions should be used sparingly. All questions included in the questionnaire should be relevant to answering the research question(s).
Based on our experience, a written survey (online or paper) should not last longer than a maximum of 15 minutes. Telephone surveys should also not be longer, unless the respondent was informed in advance or has a high level of interest in the topic.
For every questionnaire, we think intensively about the appropriate order of the questions. The order in which questions are asked has a major influence on the interpretation and evaluation of subsequent questions. We ensure that a questionnaire also provides a certain degree of variety so that participants do not become fatigued and are not tempted to click through automatically due to overly monotonous questions and the same repeated answer options (e.g., the same form of scaling).
Depending on the target group and the reach of the survey, translations into different languages may be necessary. To meet these requirements, we work with native speakers and translation agencies that operate according to the four-eyes principle.
Mobile-ready is becoming increasingly important for online surveys. Since the intensive use of smartphones and tablets in surveys, we have seen usage rates of up to 70% for mobile devices in B2C surveys. In B2B surveys, it is currently only around 20%, but the trend is rising. This brings new requirements for the graphic design and type of questions and requires adaptation to the frequently used user interfaces.
Quality in the Analysis, Interpretation, and Preparation of Results
Even before the analysis, we check the quality of the data by monitoring the response times and the response behavior of the participants. If a survey was completed unusually quickly or if participants provide contradictory or inconsistent answers, this leads to exclusion.
During the analysis, the principle of double-checking applies, meaning at least two people analyze and interpret the data.
After the analysis, a decision is made in close cooperation with the client as to which internal target group the results should be prepared for and in what form (e.g., pure information vs. decision support). It does not always make sense to present all results strictly according to the sequence of the questionnaire and in full. We deliver not only the data but also key statements so that the focus on the essentials is not lost. Based on the results, we discuss the consequences together with the client and derive measures for action.
CONCLUSION: OUR UNDERSTANDING OF QUALITY
Quality therefore goes far beyond the collection and preparation of data. Ultimately, quality is demonstrated by the fact that overarching strategic questions can be answered and the results provide concrete instructions for action.
In our work, we place great value on linking different data sources, filtering out similarities and differences, recognizing patterns, comparing them with information from existing studies, and deriving strategic recommendations for action. For us, this is where the true quality of market research is revealed. This quality depends also, but not only, on the data quality. Quality depends on the expertise of the people involved and the partnership-based cooperation between the market research provider and the client.
In our opinion, the components mentioned are necessary for high-quality market research. Representativeness plays a role, but on its own, it is neither necessary (because it is not verifiable) nor sufficient for high quality.
Services