While oil and gas prices now are falling after the latest experience with fuel-price volatility, the Global Energy Decision fuels team is focused on modeling an integrated world-wide system of...
Gas-Market Forecasts: Betting on Bad Numbers
Why predictions from the Energy Information Administration may contain systematic errors.
mean squared error components reported above in Table 3. Indeed, as Figure 4 illustrates EIA substantially over-estimated LNG imports in each of the preceding three years.
As the independent research branch of the Department of Energy, the EIA forecasts for NG possess an imprimatur that stretches across the panorama of energy policy and analysis. Thus, the socioeconomic implications of systematic bias are profound indeed.
Several important conclusions can be drawn from this research. First, the NEMS model used by EIA to generate the AEO forecasts tends to over-estimate NG production and to under-estimate NG consumption by electricity producers. While EIA forecasts of NG imports from Canada fare somewhat better, projections of LNG imports are over-estimated substantially. These errors are associated with significant under-predictions of market prices. Hence, the overall optimistic picture of ample NG supplies, and growing consumption with either falling or constant real prices has not been supported by actual experience.
Moreover, an error-decomposition analysis demonstrated that the variation in EIA’s forecast errors generally are not reflective of random chance but instead contain evidence of systematic bias, either arising from a fixed, linear bias or from a systematic error coming from the model itself. This evidence of forecast bias arising from perhaps the most comprehensive energy market forecasting system in the world illustrates the enormous difficulty of forecasting these markets. The emergence of a natural-gas cartel will add even greater uncertainty to the forecasting.
These results offer several lessons and suggest certain concerns about current and future forecasts at EIA:
1. Gas Production. First, the consistent over-predictions of NG production in the United States should raise serious questions about the reliability of the premise that large supplies would become available with higher prices.
2. Gas Use for Generation. Second, the under-prediction of NG use in electric-power production even with unrealistically low prices suggests that other factors, such as sulfur-dioxide pollution permit costs, may be stimulating NG use in this sector. (This lesson suggests that the NEMS may not be adequately modeling factors that determine the electric-power sector’s consumption of NG.)
3. LNG Imports. Third, the large over-estimates of LNG imports suggest fundamental problems with the trade side of the model. Each of these three problems presents daunting challenges for energy market modelers.
4. A Bias Toward Optimism. Current EIA forecasts exhibit a continuing optimism. In the 2007 AEO, for example, NG prices are forecasted to decline over the next decade—despite the fact that wellhead prices have increased more than 100 percent in the last five years and that the EIA did not project the vast bulk of those increases. Further, the EIA forecasts that NG production will increase 11 percent by 2020. Yet the EIA has overestimated production substantially in virtually every forecast since 1998.
5. A Failure to Recognize the Problem. Despite the biased divergence between their NG forecasts and actual outcomes, the EIA has published virtually nothing on the question of asymmetrical error. In fact, EIA’s model evaluation methodology may itself camouflage the problem. For example, Auffhammer 2 has commented that, “The EIA conducts its own