Public Utilities Reports

PUR Guide 2012 Fully Updated Version

Available NOW!
PUR Guide

This comprehensive self-study certification course is designed to teach the novice or pro everything they need to understand and succeed in every phase of the public utilities business.

Order Now

Letters to the Editor

Fortnightly Magazine - September 2007

to measure systematic bias constituting two components, an intercept or linear bias and the slope or model bias. Is our finding of systematic bias an artifact of our sample selection from 1998 to 2006? To address this question, we re-ran our decomposition analysis from 1991 to 2006. (We could not go back to 1989 because our method does not allow discontinuities in the sample.) The results are reported in Table 1.

The average percentage errors range from 8 percent to less than 5 percent for the 1- to 4-year forecasts, respectively (see Table 1). Notice, however, the RMSEs are much larger, 29 percent for the 1-year ahead forecast and rising to more than 43 percent for the 4-year forecast. Also notice that the two error components, bias and model, reflective of systematic bias, remain substantial across all four forecast horizons. For the 1-year forecasts, 25 percent of the forecast errors arise from systematic bias, and this rises to 58 percent for the 4-year forecast. These results indicate that our previous finding of systematic bias is not an artifact of sample selection.

Our findings in Table 1 above would support the contention by Rode and Fischbeck that a proportion of the errors over the entire sample could be attributed to these market shocks. For example, the random error component accounts for more than 73 percent of the 1-year ahead forecast but this drops to slightly more than 41 percent for the 4-year ahead forecast. So clearly, market shocks can explain only part of the forecast errors but certainly not all.

A more important issue for understanding forecasting errors in the gas market involves structural change. Rode and Fischbeck plot the history of natural-gas wellhead prices, various market shocks, and EIA forecasts. EIA consistently predicted higher prices during the 1980s and 1990s, was proved wrong and reversed course in the mid-1990s, only to be proved wrong again. EIA and many others were predicting a “fly-up” in gas prices in 1985 due to deregulation. Instead, prices collapsed. EIA partially reversed course and lowered their price forecasts but again failed to see the gas surplus during the late 1980s and early 1990s. As the market swung back into balance and excess capacity was eliminated after 1998, EIA continued to forecast low prices when the industry was clearly struggling to meet supply and prices were escalating.

As Rode and Fischbeck illustrate, there was a clear structural break in the gas market around 1998 to 1999, when the market went from a period with low and stable prices operating with excess capacity to a period with high and volatile prices operating at or near capacity limits. Again, while forecasting errors and price volatility would be larger during these capacity constrained periods, they should be random, resembling white noise. Otherwise they are biased. The critical question is how the National Energy Modeling System (NEMS) performs during capacity constrained periods. As it turns out, our selection of the sample 1998-2006, which was originally dictated by the availability of data for the entire gas market forecast, actually provides a good test