To the Editor:
We read with great interest Timothy J. Considine and Frank A. Clemente’s recent article on the evaluation of the Energy Information Adminstration’s natural-gas forecasts (see “ Gas-Market Forecasts: Betting on Bad Numbers ,” July, 2007) . We, too, recognize the influence that these forecasts have on both business and policy decision making and, like many others, have taken an interest in evaluating their long-term performance. We take issue, however, with one conclusion of the Considine and Clemente article. They assert that the EIA’s forecasts are subject to “systematic bias.” We are more cautious and present evidence to suggest that bias is potentially much harder to identify than it may appear.
First, however, we should be clear about areas of commonality. We have not independently studied the EIA’s forecasts of production and import activity and therefore, have no reason to disagree with Considine and Clemente’s findings (which have also been found by others 1). We limit our comments solely to the forecasts of wellhead prices. With respect to wellhead prices, we are also in agreement that the EIA’s forecasting errors are significant. Natural-gas prices, however, exhibit comparatively very high volatility, and any discussion of forecasting “error” must be considered in that context.
We do not, however, believe that the EIA wellhead price forecasts are subject to systematic bias. We believe instead that the “continuing optimism” in the EIA forecasts is cyclical in nature, being preceded by what they might have termed “continuing pessimism,” had Considine and Clemente’s analysis considered the period before 1998. Further, we believe that a more subtle force is at play here—one that is not easily accommodated by the econometric tools used in their analysis. We discuss each issue in turn.
Consider the data in Table 1, taken from the EIA’s own annual analysis of their forecasting performance 3. [Despite being published in April 2007, the EIA’s report only includes actual performance through 2005 (thus, for example, the limit for the 2004 Annual Energy Outlook [AEO] is a 1-year forecast). We limit ourselves to the EIA’s data.] We have reconfigured the table to make each column reflect a specific year-ahead prediction ( e.g., one year ahead, two years ahead), rather than a prediction for a specific calendar year. The EIA provides this data going back to the 1982 AEO, although we begin our collection of the data in 1989 (post deregulation).
We assert first that Considine and Clemente’s analysis has selectively examined a very limited period of history in looking only at 1998-2006. As Table 1 illustrates, the annual forecasting errors for the post-1998 period are indeed substantially negative (-43.5 percent), indicating that the EIA significantly underpredicted wellhead prices. However, if one were to examine the 1989-1997 period, the exact opposite conclusion would have been reached—a story of “continuing pessimism”—as the EIA persistently forecasted higher prices than actually materialized (average error of +15.7 percent).
Over the entire period, however, the average error is small: 3.0 percent. Moreover, the average error over all years and across the majority of forecast horizons is less than 10