Evaluating smart meters and public backlash.
Mike Rutkowski leads the pusiness planning and performance improvement group in Navigant Consulting’s energy practice, and Todd Lester is a director in Navigant’s disputes and investigations practice. This article is based in part on Navigant Consulting’s Evaluation of Advanced Metering System (AMS) Deployment in Texas report. Standing alone, this article doesn’t provide a full understanding of the facts and analysis underlying Navigant Consulting’s observations and conclusions from that report, and shouldn’t be relied upon for such.
The envisioned smart grid includes the digital automation of the power supply system to improve the security, quality, reliability, efficiency, and safety of electric power, as well as to make the system more environmentally friendly. Advanced (i.e., smart) meters and the associated smart metering systems are an integral part of the smart grid. According to the Edison Foundation, 38 states are pursuing deployment of smart meters and almost 60 million smart meters are expected to be installed in the next 10 years—approximately 47 percent of U.S. households. The worldwide installed base of smart meters is expected to reach 302.5 million by 2015.1
The shift to smart meters and smart metering systems in Texas has been widely supported by both electric utilities and the legislative and regulatory bodies in the state of Texas. Much of that support derives from the significant perceived benefits to the utilities and their customers that are expected from the deployment of smart meters and metering systems, and the overall development of the smart grid.
The three largest transmission and distribution service providers (TDSPs) in Texas, under the regulatory authority of the Public Utility Commission of Texas (PUCT), began installing smart meters in late 2008. Oncor Electric Delivery was the first utility in Texas to begin mass deployment of smart meters to its approximately 3.4 million residential and non-residential retail electric customers. Oncor was followed by CenterPoint Energy, which began deploying advanced meters to its 2.4 million customers in early 2009. AEP Central Texas and AEP North Texas—collectively AEP Texas—began initial deployment of advanced meters to 1.1 million customers in November 2009. As of September 2010, these TDSPs have installed more than 2.1 million smart meters and expect to have 6.5 million smart meters installed within the next three years.
Dramatic change in the utility industry is occurring with the advent of smart meters and the development of the smart grid. Utilities around the globe see far-reaching potential through the deployment of smart meters and smart metering systems by enabling active participation by consumers, the creation of new products, services and markets, and the realization of new operational efficiencies and more.2 However, harnessing this potential and realizing the promise and perceived benefits of smart meters and the broader smart grid must first overcome certain challenges and key risks. To date, many members of the public haven’t embraced smart metering deployments and some have sought to halt the deployment of smart meters, including recent efforts to place a moratorium on smart meter deployments in California and Texas (see “Smart Grid Hits the Headlines”).
Concerns have been raised regarding the accuracy of smart meters, as well as the costs associated with smart meter deployments and system development, and the relative impact to consumers. Other issues have included questions over the security and privacy of smart meters, as well as public-health concerns surrounding the potential increased exposure to radio waves from smart meters using radio frequency to wirelessly communicate with the utility. However, the overarching concerns expressed in Texas related to what impact, if any, the deployment of smart meters and smart metering systems was having on customer bills, more specifically the accuracy of the monthly electricity consumption charges billed to customers.
Beginning in early 2010, the PUCT, as well as certain retail electric providers, noted an increase in the incidence of complaints from electric customers claiming higher electric bills, much of which was blamed on an abnormally cold winter in Texas. Similar complaints were also being lodged in California by customers of Pacific Gas & Electric (PG&E), which began installing advanced meters in 2006 and which leads the U.S. in the deployment of smart meters with more than 6.5 million now in service. The media and public attention however, focused on customers claiming a possible connection between higher electric bills and the new smart meters being deployed. At the time, both Oncor and CenterPoint were rapidly deploying advanced meters across their respective service areas in Texas.3
Certain consumer interest groups as well as a number of state legislators called for a moratorium on smart meter deployment that could have cost the utilities and their customers millions of dollars, as well as federal grant money. While smart meter deployment wasn’t stopped, consumer confidence in smart meters was shaken.
The PUCT, in an effort to quickly address rising public discourse over smart meters and any potential link to higher electric bills experienced by some Texas consumers, as well as the potential loss of consumer confidence, encouraged the retention of an independent third party to respond to the concerns raised by electric customers, the media, Texas legislators and others.
The PUCT, in conjunction with Oncor, CenterPoint and AEP Texas, engaged Navigant Consulting to conduct an independent investigation and evaluation of the accuracy and reliability of the smart meters and metering systems being deployed in the Texas competitive retail electric market.
Putting AMI to the Test
The independent smart meter study in Texas was designed to address four key questions about smart meters being deployed in Texas (see Figure 1).
1) Is electricity usage being accurately measured and recorded by the smart meters?
2) Is the recorded electricity usage being accurately communicated from the smart meters through the respective smart metering systems for use in customer billing?
3) Are customers with smart meters experiencing higher recorded electricity usage on average than customers with older electromechanical meters?
4) Are there any other potential factors or causes contributing to the observed higher incidence of meter- and billing-related customer complaints?
To address these questions, Navigant Consulting’s efforts were targeted across five key areas including: 1) the independent testing of the accuracy of a large sample of smart meters; 2) a review of complaints received by the PUCT from residential customers with smart meters who expressed concerns over increases in their electric bills; 3) an independent evaluation of the smart metering systems of each TDSP, including an evaluation of the transmission of customer electric usage information from smart meters through the advanced metering infrastructure (AMI) to that used for individual customer bills (i.e., meter-to-bill data flow); 4) a statistical analysis of historical electric usage; and 5) an evaluation of smart meter testing, deployment and provisioning processes and controls in place at each of the TDSPs.
The question of smart meter accuracy was the initial focal point of the independent study in Texas and was thoroughly addressed through a multifaceted approach including meter testing, evaluation of the meters’ communication capability and effectiveness, investigation of observed anomalies across certain smart meter attributes, diagnostics and event codes, and analysis of the historical electric usage recorded before and after smart meter deployment (see Figure 2).
Over 5,600 smart meters were accuracy tested, including: 2,400 smart meters before installation; 2,700 smart meters after installation (that were removed from service and tested in a laboratory environment); more than 500 smart meters in the field, using portable testing equipment; and side-by-side testing of 75 smart meters and 75 electromechanical meters, subjected to load and temperature conditions representative of those experienced in Texas.
The study also encompassed a review and comparison to the historical accuracy testing results for approximately 1.1 million smart meters deployed in Texas and more than 86,000 electromechanical meters.
The results of the accuracy testing indicated that the smart meters accurately measured and recorded electricity usage. Out of 5,627 meters, all but two (or 99.96 percent) tested were found to be accurate by the American National Standards Institute standards of +/- 2 percent. Side-by-side testing, as well as the review of historical accuracy testing results, indicated that the smart meters being deployed are significantly more accurate on average than the electromechanical meters they’re replacing (see Figure 3).
With the accuracy of the smart meters being established through initial accuracy testing, the focus quickly turned to questions about whether the advanced metering infrastructure in use was accurately and effectively recording, storing and communicating electric usage from the smart meters for use in customer billing. The independent study evaluated smart meter deployment and data management processes and controls, from the meter through the collection, validation and transmission of data for use in customer billing. This included assessing the processes, written procedures and controls developed by each TDSP to facilitate the successful deployment and use of smart meters in their respective service areas. Also it included a meter-to-bill data analysis of over 270,000 daily and monthly register reads (meter to head-end to MDMS to CIS); and on-demand register read verification for over 650 meters.
The evaluation focused on initial testing and acceptance of advanced meters; meter deployment, including physical installation of advanced meters and initial network connectivity; and data management, including collection, storage, and transfer of data recorded by the meter through advanced meter data systems.
Through the development and use of process maps that detailed the procedures and control points in place at each of the TDSPs, as well as through performing in-depth meter-to-bill data flow analyses and meter-to-back-end system verification checks, the independent study was able to provide reasonable assurance that the processes employed by the TDSPs were sufficient to ensure the accuracy and effective deployment of smart meters, and also that the smart meters were successfully communicating with the respective smart metering systems (see Figure 4).
Cause for Complaint?
The independent smart meter study analyzed a number of customer complaints related to high electric bills, as well as a small number of specific concerns expressed by customers in one of the TDSP’s service areas, to evaluate whether the complaints bore any relationship to the deployment of smart meters and their use in the customer billing process (see Figure 5). Customer complaints were analyzed in relation to a number of factors including: smart meter deployment; unusual weather or temperature conditions; historical electricity usage; billing cycles and durations; customer billing read type (e.g., estimated, manual, automatic); and media coverage.
Complaints also were correlated to customers with either smart meters or electromechanical meters in the respective TDSP service territories. It was observed that the frequency of complaints among customers with smart meters and customers with electromechanical meters didn’t change significantly in the first 12 months of smart meter deployment in Texas. However, customer complaints in certain areas of Texas increased significantly during the winter months in Texas at the beginning of 2010.
Upon detailed review however, the higher electricity bills observed in relation to the customer complaints appeared to be due primarily to significant changes in weather and electricity usage last winter. Heating degree days were significantly higher than normal in late 2009 and early 2010, due to an unseasonably cold winter (see Figure 6). From a heating degree day perspective, the recent winter was more than 56 percent colder on average than the previous year, translating to significantly higher energy usage.
While adverse weather conditions and the resulting increased demand for electricity appeared to explain a large number of the observed higher electric bills in Texas, various other factors also appeared to contribute to the unusually high bills some Texas consumers received. Many of these issues, however, had little to do with the smart meters themselves—e.g., billing differences in some cases were attributable to the length of customer billing cycles from one period to the next, and errors in the use of estimated and manual meter reads following installation of the smart meters.
Smart vs. Electromechanical
A key component of the independent meter study in Texas was to perform statistical analysis on the historical electric usage of customers with smart meters to evaluate whether customers with smart meters experienced different—higher—metered residential kilowatt-hour (kWh) electricity usage than they would have experienced without the smart meter. Significant variations were analyzed (i.e., regressed) against potential explanatory variables such as heating and cooling degree days, as well as differences in heating source (gas vs. electric), and structure type (apartments vs. single-family homes), among others, as necessary, to determine whether the installation of a smart meter had any relative impact.
The results of the analyses indicated no statistically significant differences between customers with smart meters and customers with electromechanical meters attributed to the installation and use of advanced meters.
Lessons Learned or Re-Learned
During the course of the independent investigation and evaluation in Texas, certain key points emerged that require special emphasis and continued attention.
• Educate Customers on Smart Meter Capabilities and the Deployment Process:
Despite diverse and focused effort by utilities in educating their customers on smart meters, more can be done. Various polls continue to demonstrate the lack of consumer knowledge about what a smart meter is, much less an understanding of the perceived benefits. The consumer backlash in Texas highlighted many of those concerns. However, the quick efforts by the PUCT to address customer concerns and questions about smart meters and any potential link to higher electric bills might have averted more widespread and deeper consumer dissatisfaction in Texas. Customer education activities should be conducted early and often in the deployment process (including before any smart meters are installed) to ensure an understanding of the smart meters’ accuracy and capabilities, as well as potential reasons for fluctuations in electricity bills. That understanding should continue to be reinforced through multiple channels throughout the deployment period.
• Design Robust Processes and Controls—and Continuously Improve Them:
Smart meters and smart metering systems are significantly more automated and complex than prior processes and systems. Regardless, smart metering systems still involve significant human interaction to identify, evaluate, analyze, and process information and, as such, there’s a possibility of error and oversight, as well as inconsistencies and deficiencies that can always be improved. It’s important in such systems under development that processes, procedures and controls exist to quickly identify and address issues as they arise, and as those challenges are addressed, that new processes, procedures and controls are implemented to prevent similar issues from recurring. The need for process design, documentation, and training shouldn’t be underestimated, nor should the opportunity to build efficiency and accuracy into these new processes. Utilities should focus on developing and honing an iterative process to create, test, and refine procedures and controls around key areas.
• Address Meter Communication Issues:
According to a recent poll, communication issues are one of the primary worries that keep many utility CEOs up at night. Rightfully so, as the communication capabilities of smart meters are one of the key selling points of smart meters and the gateway to many of the perceived benefits. During any deployment, smart meters might fail to communicate for a variety of reasons, reasons that might not indicate anything wrong with the meters themselves. However, smart meters with communication issues—especially smart meters that suddenly cease to communicate—should be evaluated quickly as these issues might be symptomatic of more serious issues affecting meter accuracy. In addition, a proactive role in evaluating the cause of failed communications can yield benefits down the road if those failures are caused by broader hardware or firmware issues.
• Establish a Systematic Process to Monitor and Evaluate Meter Event Codes:
As with any technology, meters sometimes fail or cease to perform to acceptable standards for a variety of reasons, and this is true for both smart meters and electromechanical meters. However, problems with electromechanical meters might go undetected for long periods, even years, whereas in many cases problems with a smart meter can be detected relatively soon, if not immediately, after the problem arises. If a smart meter isn’t performing to acceptable standards, the meter probably will communicate an event or error code (i.e., a red flag) signaling a potential issue. Utilities need to be well versed in the various event codes and logs recorded by smart metes and have processes in place to quickly evaluate the information.
• Develop Standardized Tools to Evaluate Smart Meter Performance:
While understanding a smart meter’s reporting diagnostics and event codes is important in evaluating its overall performance, there also is other important information. However, the sheer volume of information available from smart meters can make it difficult to efficiently and effectively extract the relevant information for review. As such, it’s important for an organization to establish standardized reporting and evaluative tools for quickly identifying and assessing smart meters that have experienced unusual events or that might be operating outside of expected performance standards. For example, GPS mapping of smart meter complaints, graphical displays of historical usage before and after smart meter installation, comparative results of previous accuracy tests, and frequency distributions of meter event codes, can all provide useful information in evaluating potential smart meter issues.
• Establish Procedures and Thresholds for Escalating Meter Concerns:
Not all smart meter related issues are created equal—some may be inconsequential, others intermittent, while yet others may be more catastrophic. The enormous amount of available information that can be used to evaluate smart meter performance, even with good standardized tools, might still need to go through a triage process to determine the appropriate level of response from the organization—e.g., What can be ignored? What needs to be monitored? When does the meter need to be replaced? When does the meter need to undergo an extensive root cause analysis? Having established procedures and thresholds around such decisions will help eliminate any guesswork. While many organizations have developed, or are in the process of developing, more robust procedures around monitoring smart meter performance, it’s crucial that such procedures be established early in the deployment process to identify and remediate smart meters that have inherent systemic issues.
• Perform Root-Cause Analysis on Smart Meter Failures:
Root-cause analysis is particularly important when failures seem to be concentrated in particular hardware or firmware versions. Like electromechanical meters, smart meters can fail for a variety of reasons over their expected service lives. However, the failure of each smart meter could provide useful information in diagnosing potential hardware or firmware issues that need further attention, as well as in assisting the utility in identifying other smart meters that might not be functioning properly. While the primary technology used with smart meters has been in existence for many years, and has been tested extensively, mass deployments currently underway have subjected smart meters to a multitude of conditions and stresses that might not have been anticipated or adequately simulated by current testing methods. A robust meter interrogation and evaluation (i.e., root-cause analysis) process should be developed with outlined procedures covering the parameters for the physical inspection of the failed meter, analysis of the meter’s historical event logs and codes, evaluation of historical electric usage patterns and trends, and potential additional testing by the utility and the manufacturer, including simulation testing in attempt to recreate the operating conditions and events leading to the meter’s failure.
• Evaluate Meter Performance Following Hardware and Firmware Upgrades:
As smart meter deployments ramp up across the world, innovations in designs, technology and manufacturing techniques are constantly emerging. Unexpected or unplanned operating conditions combined with evolving designs (especially hardware) and operating requirements (firmware), as well as expanded manufacturing to meet rising demand, the potential exists for unforeseen problems … problems that could be more quickly identified by evaluating and correlating smart meter performance across distinct meter attributes.
• Establish Cross-Functional Teams to Evaluate Smart Meter Performance Issues:
While the inherent technology in many smart meters being deployed today has the capability of identifying, recording and storing an enormous amount of information regarding the operations and performance of the meters, that information must still be captured, analyzed, evaluated and understood for it to provide any meaningful business intelligence. Relevant information can come from various systems within the organization, separate and apart from those responsible for deploying, provisioning and maintaining the meters and, as such, might require specialized knowledge to fully understand the potential impact that various operating conditions, diagnostic results and event codes can have on meter accuracy. Utilities should emphasize the development and education of cross-functional teams including representatives from the information technology, metering, billing, and customer service departments to coordinate and evaluate the potential impact of any issues related to smart meters.
The smart grid will be ineffective for utilities and consumers unless the key components, including advanced meters and associated systems, are properly developed, deployed, administered and monitored. Only then will customers trust the new technology to a point where they’re comfortable taking actions based on the information these advanced meters provide, and thus yielding benefits for themselves, their utility companies, and the environment.
While evaluation shows that the advanced metering infrastructure and associated business processes for each of three utilities in Texas are functioning accurately as planned, this technology is still perceived as new to many customers, and a significant ongoing effort will be required to establish and maintain the trust of the customer.
1. Source: Berg Insight Research
2. What is the Smart Grid?, Office of Electricity and Energy Reliability, U.S. Department of Energy.
3. At the beginning of 2010, AEP Texas had yet to initiate deployment of advanced meters other than a few thousand meters pursuant to a pilot program.