Why predictions from the Energy Information Administration may contain systematic errors.
Tim Considine is a member of the Energy and Mineral Engineering Department. Frank Clemente is a senior member of the graduate faculty at Penn State and former director of the university’s Environmental Policy Center. Contact Clemente at firstname.lastname@example.org. The authors would like to express their appreciation for suggestions from Professor Maximilian Auffhammer (UC-Berkeley).
The difficulties of predicting future trends in energy are widely recognized (see Reference4). Even the most sophisticated of forecasting models cannot account fully for a myriad of complex and generally uncontrollable variables. Thus, energy policy-makers necessarily must anticipate a wide range of possible outcomes in formulating energy plans.
The issue here, however, is not how difficult it is to predict energy prices, supply, and demand. Our question, rather, is whether systematic biases are built into forecasts, causing them to err repeatedly in the same direction. And the more visible the forecast (and the more likely also that it will be used), then the more likely it is that the error will be compounded in a variety of settings.
In the case of the U.S. Energy Information Administration (EIA), for example, natural gas (NG) data and projections are used widely in regulatory proceedings, energy planning, scientific research, investment decisions, litigation, and legislation. In such cases, systematic bias can have profound socioeconomic implications—not only within the United States but in other nations as well. Indeed, the National Energy Board of Canada regularly includes EIA NG forecasts in its projections. Even OPEC scholars use EIA projections as a benchmark in their research.
This widespread use of EIA forecasts follows the organization’s own view of its nature and purpose. In fact, the EIA has indicated that it designs its forecasts specifically to aid policy-makers by providing “a policy-neutral reference case that can be used to analyze policy initiatives.” However, while the EIA may strive to make its reference case forecasts “policy neutral,” the question still remains: Are they “substantively neutral” in a forecasting sense? In other words, are they removed from the sort of systematic bias in which predictions deviate from actual observations in a distinct pattern?
Over the past decade, it increasingly has become apparent that EIA forecasts for NG differ substantially from actual outcomes. Some commentators1 have suggested that EIA forecasts present a consistently “optimistic” view of NG that, for instance, underestimate price and overestimate supply. On the surface, this concern has face validity based upon forecasts from the EIA’s Annual Energy Outlook series:
• In 2002, the EIA projected the cost of NG to electric generators in 2006 would be $ 3.82 per thousand cubic feet (Mcf). Actual cost per Mcf was $7.15 (all in 2006 dollars)
• In 2003, the EIA overestimated domestic NG production in 2006 by almost 2 trillion cubic feet—more than the annual production of Oklahoma.
• In 2005, the EIA projected liquefied natural gas (LNG) imports would reach 1,140 bcf in 2006. Actual imports in 2006 were only 583 Bcf—off by more than 550 Bcf just one year out.
To shed light upon the question of bias, we conducted an error decomposition analysis of EIA NG projections of key variables—price, supply, and consumption—from 1998 to 2006. Error-decomposition analysis is used commonly to evaluate economic forecasting models by identifying those components of the forecast errors or the proportions attributed to bias, the model, or randomness. A reliable model would display random errors with no discernable pattern of consistent under- or over-predictions. Thus, the proportions of forecast errors attributed to bias and model components would be minimal.
In our case, we evaluated one-, two-, three-, and four-year-ahead forecasts made by EIA from 1998 to 2006 for six key variables: (1) wellhead price; (2) price to electric generators; (3) consumption by electric generators; (4) domestic production; (5) imports from Canada; and (6) LNG imports.
Selecting Data for Review
Bolinger and Wiser5 provides a graphical illustration of how EIA wellhead-gas prices forecasts going back to 1985 track actual prices. Their graph clearly illustrates that price forecasts during the 1980s turned out to be too high while forecasts made during the early 2000s appear too low. Graphical techniques, however, do not quantify the size or systematic tendencies of these forecasts errors. This study attempts to extend their analysis by applying the error decomposition methods discussed above.
During December of each year, EIA publishes a forecast that forms the basis of the Annual Energy Outlook, or AEO,8 for the subsequent year. (Note: The EIA each year releases its reference case in December. Then in the following February, the EIA releases its full report, with sensitivity cases.)
So, for example, the 2006 AEO report released in December 20059 contains a forecast of 2006 prices. This study examines their forecasts published from 1998 to 2006 because EIA posts the detailed forecast tables on its Web site, which is accessible to the public. Auffhammer2 uses a larger sample and finds that the EIA forecasts of NG consumption, production, imports, and prices do not exhibit the necessary conditions for rationality under symmetric loss. (Note: The EIA uses the National Energy Modeling System, or NEMS. See “Appendix: Methods of Forecast Evaluation,” describing our evaluation of EIA’s forecasting methods.)
While each EIA forecast extends 20 years or more, the maximum length of the forecast horizon examined in this study is four years. A three- to four-year forecast for prices is likely of most interest to industry because natural-gas-fired electricity generating plants take roughly three years to build. Moreover, going any more than four years out would not be meaningful given the small size of our sample. Given the sample of forecasts from 1998 to 2006, there are nine one-year-ahead forecasts, eight two-year forecasts, seven three-year forecasts, and six four-year forecasts. While comparing each published AEO forecast with actual data over its entire forecast horizon is insightful, economists typically stratify forecasts by length of time not necessarily when they are made. Hence, the forecasts are sorted by length of forecast horizon.
Evaluating the EIA Forecasts
To keep the analysis manageable and comprehensible, our decomposition analysis is conducted for three pairs of variables in the natural-gas market involving prices, domestic flows, and imports. The two prices are the average wellhead price and prices paid for natural gas by electricity producers. The flow variables include dry natural-gas production and consumption by electricity producers. The later was selected because the electricity sector comprises the most dynamic, market-sensitive component of natural-gas consumption along with industrial sector use. Imports include those from Canada and imports of LNG.
Prices. The EIA forecasts natural-gas prices in constant dollars. To establish a consistent basis for comparison, these constant price forecasts are inflated by the corresponding forecasts for the price deflator for gross domestic product (GDP). Once the forecasts are sorted, the prices are converted back to 2006 dollars using the latest GDP price deflator.
The forecast evaluation metrics for the one- through four-year-ahead forecasts from 1998 to 2006 appear in Table 1. On average, the one-year-ahead average percentage forecast error for the wellhead natural-gas price is 16 percent with an absolute error of $1/Mcf. These errors steadily rise and reach more than 45 percent with the four-year-ahead forecast and $2.60/Mcf.
The RMSE (root mean squared error), which penalizes large errors more severely than the average percentage error (see “Appendix,” for full explanation), is almost 35 percent for the one-year-ahead forecast. Like the average percentage error, it too rises with the forecast horizon, reaching more than 57 percent with the four-year-ahead forecasts.
The decomposition of the MSE (mean squared error) for the one-year-ahead wellhead natural-gas price forecast errors indicates that 54.7 percent of the errors can be attributed to systematic bias. This bias crests to almost 88 percent for the three-year-ahead forecasts. While random disturbances are substantial for the one-year-ahead forecast, the large proportion attributed to bias is noteworthy. A plot of the actual time series for wellhead natural-gas prices and the four different forecasts appears in Figure 1 and illustrates the tendency of the EIA price forecasts to systematically under-predict actual prices. The results for electric generator’s natural-gas costs are very similar to those for wellhead natural-gas prices.
Market Flows. Table 2 shows the forecast errors for natural-gas consumption by electricity generators and for dry natural-gas production. The forecast errors are much smaller than those associated with the forecast errors for prices, which is a common phenomenon. Price forecasting often is more difficult than forecasting demand and production series, which often contain a sizeable trend component or signal. Nevertheless, the forecast errors for these two key natural-gas market flows are substantial.
The EIA forecasts for natural-gas consumption in electricity generation consistently are below actual observations of gas use in this sector (see the average percentage errors in Table 2). This is somewhat counter-intuitive because given that EIA under-estimates prices paid for natural gas by electric generators, it would seem that lower prices would imply higher, not lower, natural-gas consumption, all other things held equal. One of the big changes affecting the electricity sector’s use of fuels has been the sulfur-dioxide emissions-trading program. That program has exerted a dramatic effect on the opportunities for fuel substitution in power generation, as shown by Considine and Larson6. Whether the NEMS correctly models the role of permits in power-sector fuel demand and fuel switching could be an important question.
The absolute error for the one-year-ahead forecast for electric generators natural-gas consumption is more than 900 billion cubic feet, which is more than 15 percent of consumption in this sector. In addition, the RMSEs are around 20 percent, nearly four times the errors found in econometric forecasting models of energy demand.7 Like prices, the error decomposition analysis for natural-gas consumption by electric generators reveals a substantial bias across all four forecast horizons.
The forecast errors for dry natural-gas production reveal further problems. As the average percentage errors indicate, EIA consistently over-predicts dry natural-gas production. The absolute errors are quite sizeable in relation to marginal supplies of gas, specifically imports of LNG. For example, the one-year-ahead forecast error for production is 590 billion cubic feet, which is about equal to LNG imports in 2006. The two- through four-year-ahead forecast errors exceed one trillion cubic feet.
The mean squared error decomposition for natural-gas production also reveals sizeable bias, especially for the three- and four-year forecasts. Unlike prices and consumption forecast errors, the model component of the errors is more than 40 percent for the one- and two-year forecasts. This fact suggests that the model itself is generating systematic errors for the near-term forecast horizon. The time path of each forecast depicted in Figure 2 illustrates that even though EIA has been scaling back its projections of natural-gas production, the model still portrays an upward track for production albeit from a lower base during each forecast year.
Imports. Another important factor influencing natural-gas markets is imports. The largest external source of natural gas into the United States is Canada, although EIA expects imports of LNG to become significant in the future. Among the forecast errors examined in this study, those associated with EIA’s projection of imports from Canada are the lowest. Similar to the other forecast errors, however, the forecasts contain either bias or systematic errors arising from the model.
The projections of LNG imports are not as accurate as those for Canadian imports. The RMSEs are quite large and, while the bias components are relatively small, the proportion of the forecast errors associated with the model remains substantial, especially for the first and third year-ahead forecasts. This finding could be associated with the rather idiosyncratic nature of the LNG import forecasts.
To understand what is happening in the LNG forecast error decomposition, a scatter plot of the actual versus predicted LNG imports appears in Figure 3. A perfect forecast in which the predictions are equal to the actual observations is plotted on the solid line. A “good” forecasting model should generate forecasts close to the line of perfect forecasts and randomly scattered around it. As Figure 3 illustrates, there are several very large over-predictions of LNG imports. The small number of these very large errors most likely accounts for the erratic swings in the mean squared error components reported above in Table 3. Indeed, as Figure 4 illustrates EIA substantially over-estimated LNG imports in each of the preceding three years.
As the independent research branch of the Department of Energy, the EIA forecasts for NG possess an imprimatur that stretches across the panorama of energy policy and analysis. Thus, the socioeconomic implications of systematic bias are profound indeed.
Several important conclusions can be drawn from this research. First, the NEMS model used by EIA to generate the AEO forecasts tends to over-estimate NG production and to under-estimate NG consumption by electricity producers. While EIA forecasts of NG imports from Canada fare somewhat better, projections of LNG imports are over-estimated substantially. These errors are associated with significant under-predictions of market prices. Hence, the overall optimistic picture of ample NG supplies, and growing consumption with either falling or constant real prices has not been supported by actual experience.
Moreover, an error-decomposition analysis demonstrated that the variation in EIA’s forecast errors generally are not reflective of random chance but instead contain evidence of systematic bias, either arising from a fixed, linear bias or from a systematic error coming from the model itself. This evidence of forecast bias arising from perhaps the most comprehensive energy market forecasting system in the world illustrates the enormous difficulty of forecasting these markets. The emergence of a natural-gas cartel will add even greater uncertainty to the forecasting.
These results offer several lessons and suggest certain concerns about current and future forecasts at EIA:
1. Gas Production. First, the consistent over-predictions of NG production in the United States should raise serious questions about the reliability of the premise that large supplies would become available with higher prices.
2. Gas Use for Generation. Second, the under-prediction of NG use in electric-power production even with unrealistically low prices suggests that other factors, such as sulfur-dioxide pollution permit costs, may be stimulating NG use in this sector. (This lesson suggests that the NEMS may not be adequately modeling factors that determine the electric-power sector’s consumption of NG.)
3. LNG Imports. Third, the large over-estimates of LNG imports suggest fundamental problems with the trade side of the model. Each of these three problems presents daunting challenges for energy market modelers.
4. A Bias Toward Optimism. Current EIA forecasts exhibit a continuing optimism. In the 2007 AEO, for example, NG prices are forecasted to decline over the next decade—despite the fact that wellhead prices have increased more than 100 percent in the last five years and that the EIA did not project the vast bulk of those increases. Further, the EIA forecasts that NG production will increase 11 percent by 2020. Yet the EIA has overestimated production substantially in virtually every forecast since 1998.
5. A Failure to Recognize the Problem. Despite the biased divergence between their NG forecasts and actual outcomes, the EIA has published virtually nothing on the question of asymmetrical error. In fact, EIA’s model evaluation methodology may itself camouflage the problem. For example, Auffhammer2 has commented that, “The EIA conducts its own forecast evaluation…[but] this type of evaluation ignores potentially persistent biases in the forecasting model.”
The analysis reported here suggests that considerable caution should be exercised when using EIA forecasts relating to the future price, supply, and consumption of NG. Similar caution should be exercised when using NEMS to assess the broader economic impacts of energy policy initiatives, e.g., carbon cap-and-trade programs.
Climate-change proposals currently before Congress3 depend heavily on predictions of the response of natural-gas supply and prices to carbon-permit prices. The actual capability of the NG supply network both here and abroad will be a critical factor in how economies adjust to such climate-change policies. Overestimating the supply capabilities of this network (as EIA has done over the past decade) could lead to underestimating the costs of carbon regulations.
1. American Association of Petroleum Geologists, “Natural Gas Supply Concerns,” Tulsa, Published 2000 – 2007, email@example.com.
2. Auffhammer, M. “The Rationality of EIA Forecasts under Symmetric and Asymmetric Loss,” Resource and Energy Economics, Volume 21, 102-121.
3. Bingaman, Jeff, U.S. Senate (Dem., N.N.): e.g., “Biofuels for Energy Security and Transportation Act of 2007,” “Carbon Capture and Sequestration Act of 2007,” “Energy Efficiency Promotion Act of 2007,” “Energy Savings Act of 2007,” United States Energy Storage Competitiveness Act of 2007,” S. 1321, introduced May 7, 2007.
4. Bezdek, Roger and Robert Wendling, “A Half Century of Long Range Energy Forecasts”, Journal of Fusion Energy, Volume 21, pp. 155-172, December 2002.
5. Bolinger, M. and R. Wiser (2005) “Comparison of AEL 2006 Natural Gas Price Forecast to NYMEX Futures Prices,” Lawrence Berkeley National Laboratory, LBNL.59233.
6. Considine, T.J. and D.F. Larson (2006) “The environment as a factor of production,” Journal of Environmental Economics and Management, 52, 3, 645-662.
7. Considine, T.J. (2000) “The impacts of weather variations on energy demand and carbon emissions,” Resource and Energy Economics 22, 4, 295-312.
8. Energy Information Administration, Annual Energy Outlook (AEO), U.S. Department of Energy, Washington, D.C., The AEO’s for each year from 1998–2006 were utilized, Tables 13 and 14.
9. Energy Information Administration, Annual Energy Outlook, 2005, U.S. Department of Energy, Washington, D.C., 2005, p.4.
10. Energy Information Administration (2007), Energy Market and Economic Impacts of a Proposal to Reduce Greenhouse Gas Intensity with a Cap and Trade System, U.S., Department of Energy, Washington, D.C., 2005 (January).
11. Energy Information Administration, Short Term Energy Outlook, U.S. Department of Energy, Washington, D.C., The Short Term Outlooks are issued monthly, Table 8a, 1998- 2006.
12. Ghouri, Salman, “North American Natural Gas Demand—Outlook 2020,” OPEC Review, Volume 28, pp. 1-26, March, 2004.
13. Maddala, G.S. (1977) Econometrics (New York: McGraw Hill).
14. Theil, H. (1966) Applied Economic Forecasting (New York: Rand McNally & Co.).
15. National Energy Board, “Natural Gas for Power Generation,” June 2006, Calgary, Canada.