Theory and experience teach that commercial market research
can be of very poor quality. What does that mean
for regulators and utility managers?
How can regulators and...
it measures what it purports to measure. It is reliable if its results are reproducible. It is these two attributes that provide the foundation for rational decisionmaking by regulators and managers.3
The substance of "bad" market research can range from data manipulation that is intended merely to increase the significance level of statistical tests, to the outright fabrication of raw data and research results. Such "research" is neither valid nor reliable, and so cannot provide an appropriate basis for decisionmaking.
Suspicions concerning the value of commercially prepared research are heightened by the capacity of bad research to squeeze good research out of the market. This phenomenon is referred to generically as the "lemons" problem and, much less precisely, as the workings of Gresham's Law.4
The theory of "lemons" is driven by the asymmetry of information between sellers and buyers. Sellers know the quality of the goods that they are selling. Buyers, on the other hand, cannot always recognize a "lemon" when they see one. This asymmetry proves especially significant for credence goods whose quality, by definition, is not readily ascertainable, either before or after purchase. Therefore, it is in the self-interest of buyers to assume the worst about quality, and to offer a correspondingly low price. The price discount that arises under these circumstances causes products of superior
quality and value to be withheld from the market which, in turn, further lowers the offered price. And so on. The market eventually will be dominated by "lemons" unless sellers break the cycle by discovering ways to signal quality.
In the context of market research, the "lemons" problem similarly would lower price and drive high-quality research from the market unless firms found ways to signal the quality of their work product. It is in the self-interest of research firms to signal high quality when it is present, and to drag red herrings when quality is low or absent. The differences between the signals that firms send out make it possible to distinguish between high- and low-quality work product.
Distinguishing between quality signals and red herrings can be a tricky task. Experienced managers and regulators develop an instinct for doing this. Decisionmakers working in newly competitive industries, however, can be too fresh in their jobs to have developed this instinct. There is also some casual evidence to suggest that managers will purchase and study whatever research is available, regardless of quality, in order to satisfy the institutional expectation that this is what managers are supposed to do.
Quality: Sorting Value
Research firms that provide high-quality work product have a pecuniary incentive to signal this fact. By so doing, the risk of being driven from the market by "lemons" is reduced, and price can be kept high.
In contrast, firms that produce low-quality work product must avoid providing any information that would signal this fact. Accordingly, these firms have an incentive to provide only generic information about themselves and their work, information which, upon close inspection, actually reveals little or nothing about work product quality. These signals, taken alone, can be grossly misleading, hence