Theory and experience teach that commercial market research
can be of very poor quality. What does that mean
for regulators and utility managers?
How can regulators and...
their characterization here as "red herrings."
Potentially Misleading Indicators
All research firms rationally stress those factors that tend to cast their work product as "search" and "experience" goods, and that tend to draw attention away from the "credence" nature of research work product. Experienced users of market research recognize these indicators as being imperfect proxies of research quality, which can be manipulated independently of the work product itself.
These indicators are not necessarily unimportant. However, they are to quality what sizzle is to steak. Utility managers must recognize these factors as being imperfect and potentially misleading indicators of quality. Research firms that stress these factors, but which otherwise provide no valid indications of quality, reasonably deserve to viewed with suspicion.
Name and reputation. Established research firms make a significant investment in brand. However, circumstances can allow these firms to build profits by "harvesting" their reputation and chiseling on quality. Moreover, reputation alone does not imply the institutional knowledge needed to analyze specialized markets properly.
Firm size and staff expertise. A large firm comprising a distinguished staff does not mean that a particular piece of work necessarily is (or will be) valid and useful. Moreover, apparent size and expertise can represent nothing more than carefully staged illusions.
Professional association memberships. Legitimate associations are scarce in the area of market research, and tend to exist only where market forces already are sufficient to ensure quality.5
Warranties. All professional firms purport to stand behind their work, but this stance is of little practical significance for "credence" goods, whose quality and value cannot be ascertained readily.
Price. High-quality research is costly to prepare. Price, however, is not necessarily a valid indicator of underlying cost. Rather, it can be a reflection of how much a prospective client is willing to pay.
Potentially Useful Indicators
Market research firms that provide high-quality work have a pecuniary incentive to signal that fact. Thus, research quality can be assessed in part by interpreting the signals that firms send out. Reasonable suspicions are raised when these signals are inconsistent with the self-interest of a firm that produces high-quality work product. Confidence in the quality and value of research work product reasonably decreases as the level of skepticism triggered by these signals increases.
Nondisclosure of survey questionnaires. A well-designed survey instrument is a good predictor of research quality. Any reluctance to disclose questionnaires (em valid concerns over trade secrecy aside (em raise suspicions.
Poorly described research method. Market research necessarily involves many procedures, data collection techniques, calculations, and assumptions. Clients are entitled to know exactly how research results are developed from these foundations. Sketchy and incomplete descriptions of method raise suspicions.
Proprietary claims to underlying data. Some research data truly is proprietary. However, clients can and should acquire property rights in job-specific research data whenever possible. A firm's reluctance to furnish this data raises suspicions.
Irreproducible results. Be suspicious of firms that promise to measure market effects that seem impossible to detect and quantify. Be doubly suspicious of firms that actually deliver on such promises.
Disparities with internal estimates. Be suspicious of research