Theory and experience teach that commercial market research
can be of very poor quality. What does that mean
for regulators and utility managers?
How can regulators and...
results that are at odds with your own organization's internal knowledge base. Disparities often arise when outside firms do not know the range in which research results "ought" to fall. Accordingly, always avoid communicating expected results in advance to researchers.
Missing baseline measures. Comparing research results to actual operating data provides a valid and handy check on research validity. Firms producing high-quality work product have an
incentive to provide such "baseline" measures. In contrast, firms providing low-quality work product have an incentive to prevent such checking, and typically will produce these measures for every firm in the market except the client.
Form over substance. Superfluous, eye-catching charts mark a potential source of concern when they appear calculated to draw attention away from the substance of the research itself.
Data sets of unknown size. Research firms can add volume to reports by providing tabulations based on data subsets (e.g., demographic breakouts). These subsets necessarily contain fewer cases than the overall sample, and so have correspondingly wider confidence intervals. It always is appropriate for researchers to report the number of cases included in a tabulation. Low-quality research, in contrast, is characterized by snappy charts and graphs alone, which promote an inference that the offered data "cuts" are statistically significant when, in fact, they may not be.
Too few firms. The prospect of profits is sufficient to attract market research firms to newly competitive markets. A relative paucity of firms, therefore, can indicate a "lemons" problem, although it might also characterize a market that is simply too thin to support more than a few firms. If a "lemons" problem is suspected, consider working inhouse.
Arithmetic indices of consumer choice. Statistical and econometric procedures can be used to quantify the significance of consumer choice variables. Simple arithmetic can be used to create frequency distributions, averages, proportions, and ordinal rankings of these variables, but cannot disentangle overlapping and interrelated effects. Accordingly, be suspicious of elaborate arithmetic indices.
Related service offerings. Database management and custom tabulations are legitimate extensions of market research work. "Consulting" services, however, can raise potential conflicts of interest. For example, it is reasonable to be suspicious of research firms that offer to provide marketing advice that is "guaranteed" to increase market share (em as gauged by the next round of its own market research. Weigh the incentives at play.
Mitigating excuses. Research firms may justify low-quality work by asserting that the numbers represent "the best information available." Maybe so, but be aware that "best available" does not mean "valid and reliable."
Who Owns the Data?
Some firms decline to make underlying research data available to purchasers of compiled research results, usually by asserting a proprietary claim to the data. The issue arises not only when clients seek a check upon the competence and integrity of researchers, but also when clients suspect that
researchers have not extracted all of the information contained in the data. This issue forms a potential source of friction between research firms and their clients.
Some market research is based upon data that clearly is owned by the research firm, as