THE SEPT. 1, 1998 ISSUE OF Public Utilities Fortnightly contained an article, "The Fortnightly 100," which promised to reveal America's "most efficient utilities." The authors used data envelopment analysis (DEA) to analyze historical operating and financial data for 140 utility holding companies. While DEA can be a useful tool for data analysis, used indiscriminately it can lead to misleading conclusions.
There are several rules of thumb to consider when benchmarking utilities, which were not incorporated in determining the "efficient" utilities from the ones who "misallocated" their resources.
Typically, when utilities benchmark against each other they create benchmarks for comparison that match generation costs to generation cost drivers (e.g., MWh produced), transmission costs to transmission cost drivers (e.g., mile of transmission lines) and distribution costs to distribution cost drivers (e.g., number of customers).
In their study, the authors only used one cost driver, MWh produced, to determine a company's efficiency. This methodology adversely biases the results of any utility whose company-owned production is relatively small compared to its transmission and distribution network. An example of this could occur in the case of a utility that divested the majority of its generation assets in favor for purchased power contracts.
The purpose of benchmarking, as the authors note, is to give companies an idea how much they can improve their current operations. Therefore, the analysis should be normalized for exogenous or uncontrollable factors. Some examples include:
• Asset Mix. The authors, without hesitation, compare all-fossil generation utilities against utilities with nuclear generation. An all-fossil utility will almost always look better than a utility with nuclear assets. Very little of that difference has anything to do with which utility is employing "best practices" as the authors suggest.
• Geographical Location. If the authors wanted to include fuel costs in their analysis, then it is difficult to see why they chose to compare utilities in producing regions against utilities in market areas. Transportation cost for fuel can be significant. Transport for coal can easily be as much or more than the coal itself for a plant not located near the mine mouth. Transport expense for gas in a heavily consuming region like the Northeast can reach 60 cents/ mmbtu (firm) or approximately 30 percent of commodity cost versus 10 cents/mmbtu for a power plant in Texas.
• Urban Versus Rural Utility. Urban utilities tend to have higher operating and maintenance and capital costs since maintenance and construction on a city street is much more expensive then trenching in a field. Also, an urban utility is likely to have more underground lines - more expensive than above-ground lines at
a rural utility.
While using historical databases for source data is convenient, there's always the chance of not fully understanding what is behind the numbers. For example, the authors included pension costs in their analysis. As a result of FAS 106 (an accounting rule requiring fully funded pensions), some utilities that were underfunded in the past will show a very high pension expense as they ramp their pensions up to fully funded status. The additional pay-in some utilities are making as a result of FAS 106 is significant, sometimes up to 10 percent of total labor costs.
Not adjusting for unique characteristic of some utilities is another instance of reaching questionable conclusions by using numbers without fully understanding them. For example, since reliability is seen as extra critical by the businesses in Manhattan (especially in the financial sector), Con Edison in New York has designed and operates an extremely high reliability system, which is relatively expensive to maintain. The authors ignore the special conditions companies like Con Ed operate under. They label the additional costs "misallocated."
While the SEC data the authors used is likely to be accurate and representative, caution should be used in drawing conclusions from FERC Form 1 data. While the FERC has set out detailed account descriptions and instructions for the population of these forms, there exists variability in the conformity of how utilities categorize their costs.
Additionally, and perhaps more significantly, each utility has discretion in what level of capitalization they apply to their overall costs to get the split between O&M expense and additions to capital (rate base). Differences in capitalization rates can significantly affect comparisons that include O&M.
Company benchmarking, at its best, exists in the form of benchmarking consortiums. In a benchmarking consortium, a number of companies in the same industry agree to provide very specific information to an impartial third party, such as an accounting firm or consultancy. The third party expends a considerable effort to assure comparability of the source data, normalize for one time or unusual costs, and select appropriate peer groups. The McKinsey Gas Pipeline Benchmarking Study and the Arthur Andersen E&P General and Administration Expense consortiums are good examples of this type of study.
Recognizing that limited resources may prevent such detailed analysis, large broad-based studies can be useful - only if these benchmarking "rules of thumb" are addressed.
David A. Foti
Economic Adviser
Enron Transportation and Storage (Gas Pipeline Division)
Enron Corp.
Houston
The writer has worked as an energy consultant for some of the "Big Five" firms and has performed more than a dozen utility benchmarking studies.
I WAS AMUSED TO READ THE ARTICLE ON UTILITY "EFFICIENCY" in your Sept. 1, 1998 issue. It reminded me of the weather reports we often get from so-called experts who haven't bothered to look out the window to see what the sky is doing. Econometric analysis is a difficult and challenging endeavor, one which can have great value in helping businesses and policy makers make the best decisions. The more elaborate and difficult the analysis however, the greater the danger in drawing conclusions that have no practical basis or use. This is in fact the case with the analysis presented - an elaborate and rather elegant theoretical study of "efficiency" in the utility sector, the results of which are totally useless.
The clue to this flaw is found in footnote 4, where the authors admit that "we may have introduced some bias against companies with large amounts of purchased power." In short, the study adopts as a key assumption the simplistic notion that each utility in the entire industry reflects an identical degree of vertical integration.
Talk about bias! It works against regions and companies with significant generation from independent power producers, and regions with very active bilateral markets. It also is biased against companies that use the competitive wholesale market as an efficient and cost effective way to meet power requirements. Finally, it counts against utility systems divesting of generation in response to state restructuring initiatives.
For example, Unitil is a small holding company system in New England, which operates three distribution systems, two in New Hampshire and one in Massachusetts. Unitil owns less than 10 percent of its generation requirements, having found that competitive wholesale purchases in New England are far more cost effective to meet customer requirements than building and financing generation. As a result, Unitil has among the lowest rates in New England. On the other hand, Unitil is also a very efficient distribution company. Analysis of the costs of distribution among New England utilities reveals that Unitil's distribution operations are the most efficient, and most cost-effective in the region. But you would never know it from this study.
It is likely that economics will continue to be known as "the dismal science" as long as its practitioners continue to forget the basic requirement of science - to observe the real world.
Robert G. Schoenberger
Chairman/CEO
Unitil Corp.
Hampton, N.H.
THE OBJECTIVE OF THE AUTHORS OF "THE FORTNIGHTLY 100" - to provide comparative measures of efficiency among electric utility companies - is a laudable one. We at Vermont Electric Power Co. have devoted considerable effort to this task because we agree that "Things that are measured tend to improve."
As a company that provides transmission only, we have been hampered by a lack of comparable data, so we have been limited to measurements of changes in (1) the price we charge for our services, and (2) the reliability and quality of service we furnish. During the past 10 years, the price we charge for our services has remained essentially flat, while the regional producer price index has risen more than 25 percent, and the consumer price index, more than 37 percent. Pursuing a zero defect policy, we also have seen a steady trend of increased reliability and quality.
We were astonished, therefore, to find ourselves posted at the very bottom of the article's top 100 companies. Notwithstanding their praiseworthy objective, your authors committed the error of failing to understand the data they used in their analysis. They define the output side of their efficiency formula as "total physical production in megawatt-hours produced and sold to all sectors." Data regarding production and sales were then taken from FERC Form 1. Owing in part to the inflexibility of FERC reporting requirements, and in part to some unique contractual arrangements, VELCO does report megawatt-hours sold to FERC. The fact is, however, that we are strictly a transmission company. We produce no power and, except via paper transactions as a pass-through entity, we sell no power. In short, the data employed by the authors of the article were, for VELCO, absolutely meaningless.
The authors may or may not have produced a useful tool for utilities to compare and improve their performance, but they have certainly not found a way out of the data age adage, "Garbage in, garbage out."
Richard M. Chapman
President/CEO
Vermont Electric Power Co.
Rutland, Vt.
THE AUTHORS RESPOND: We appreciate the interest our article generated. The impetus for this paper came from the authors' interest in and experience with productivity measurement and analysis over several years. With the advent of competition in the electric utility industry, and the obvious effects of productive efficiency on the competitive position of utilities, it seemed rigorous studies of productivity using the available tools would be needed more than ever. The two studies we published in the past year (Public Utilities Fortnightly, Sept. 1, 1998 and June 15, 1997), and the interest they generated indicate this is indeed the case.
Our analysis was intended as a macro-level general model to demonstrate how raw data envelopment analysis (DEA), a linear programming approach, can be used as an alternative to the econometric methods in measuring operational efficiency across a large group of enterprises. We received many favorable responses and a few critical ones. Potential bias, resulting from some inherent heterogeneity in the population of utilities that we studied was the main critical theme in the majority of unfavorable responses. Variations among utilities in the mix of generating resources, in geographic location, in consumer density, and in the proportion of purchased power, were the main sources of bias mentioned.
At issue is whether such heterogeneity invalidates efficiency comparisons across utilities. Indeed, heterogeneity, a fact of life, necessitates the use of techniques such as DEA, which allow us to make efficiency comparisons across different technologies and input mixes. The DEA approach produces a ranking of efficiency scores based on "relative" performance to "peers." That is, efficiency of any particular utility is examined vis-à-vis other utilities that are alike in terms of input mix. The final performance ranking across utilities is, to a large extent, a function of the technology and input mix.
We do acknowledge, however, as noted in footnote 4 of the Sept. 1 article, that our efficiency ranking is biased against utilities with significant amounts of purchased power. Unfortunately, data limitations prevented us from performing the appropriate adjustments to eliminate such bias.
IN "CHARGING KWhS AND BTUS ON CREDIT," (Public Utilities Fortnightly, Sept. 15, 1998), I was taken by surprise by the comments attributed to Mychelle Jackson of Austin, Texas. She mentioned that her utility accepts credit cards for customers who have been disconnected for non-payment. I work for Northern States Power based in Minneapolis and am responsible for payment programs. NSP has been advised that accepting a credit card "on any account that has been assessed a late payment charge," would be out of compliance with Visa banking rules. We had been taking credit cards on past due bills and Visa wrote to us and informed us we were out of compliance. Obviously we discontinued the practice and the result has been many angry customers that used to use the card. It is curious that some companies continue the practice.
Pat Boland
Product Manager
Northern States Power Co.
Minneapolis, Minn.
Articles found on this page are available to Internet subscribers only. For more information about obtaining a username and password, please call our Customer Service Department at 1-800-368-5001.