While oil and gas prices now are falling after the latest experience with fuel-price volatility, the Global Energy Decision fuels team is focused on modeling an integrated world-wide system of...
Gas-Market Forecasts: Betting on Bad Numbers
Why predictions from the Energy Information Administration may contain systematic errors.
by more than 550 Bcf just one year out.
To shed light upon the question of bias, we conducted an error decomposition analysis of EIA NG projections of key variables—price, supply, and consumption—from 1998 to 2006. Error-decomposition analysis is used commonly to evaluate economic forecasting models by identifying those components of the forecast errors or the proportions attributed to bias, the model, or randomness. A reliable model would display random errors with no discernable pattern of consistent under- or over-predictions. Thus, the proportions of forecast errors attributed to bias and model components would be minimal.
In our case, we evaluated one-, two-, three-, and four-year-ahead forecasts made by EIA from 1998 to 2006 for six key variables: (1) wellhead price; (2) price to electric generators; (3) consumption by electric generators; (4) domestic production; (5) imports from Canada; and (6) LNG imports.
Selecting Data for Review
Bolinger and Wiser 5 provides a graphical illustration of how EIA wellhead-gas prices forecasts going back to 1985 track actual prices. Their graph clearly illustrates that price forecasts during the 1980s turned out to be too high while forecasts made during the early 2000s appear too low. Graphical techniques, however, do not quantify the size or systematic tendencies of these forecasts errors. This study attempts to extend their analysis by applying the error decomposition methods discussed above.
During December of each year, EIA publishes a forecast that forms the basis of the Annual Energy Outlook, or AEO,8 for the subsequent year. (Note: The EIA each year releases its reference case in December. Then in the following February, the EIA releases its full report, with sensitivity cases.)
So, for example, the 2006 AEO report released in December 2005 9 contains a forecast of 2006 prices. This study examines their forecasts published from 1998 to 2006 because EIA posts the detailed forecast tables on its Web site, which is accessible to the public. Auffhammer 2 uses a larger sample and finds that the EIA forecasts of NG consumption, production, imports, and prices do not exhibit the necessary conditions for rationality under symmetric loss. (Note: The EIA uses the National Energy Modeling System, or NEMS. See “Appendix: Methods of Forecast Evaluation,” describing our evaluation of EIA’s forecasting methods.)
While each EIA forecast extends 20 years or more, the maximum length of the forecast horizon examined in this study is four years. A three- to four-year forecast for prices is likely of most interest to industry because natural-gas-fired electricity generating plants take roughly three years to build. Moreover, going any more than four years out would not be meaningful given the small size of our sample. Given the sample of forecasts from 1998 to 2006, there are nine one-year-ahead forecasts, eight two-year forecasts, seven three-year forecasts, and six four-year forecasts. While comparing each published AEO forecast with actual data over its entire forecast horizon is insightful, economists typically stratify forecasts by length of time not necessarily when they are made. Hence, the forecasts are sorted by length of forecast horizon.
Evaluating the EIA Forecasts
To keep the analysis manageable