The most severe storm of the last 10 years hit the Chicago area on Aug. 23 and 24, 2007. A combination of major flooding, 80,000 lightning strikes, and winds up to 100 miles per hour left 634,000 of Commonwealth Edison’s 3.8 million customers without power. Six counties were declared federal disaster areas.
“The storm system was so massive it took 12 hours to clear our territory and locked out [or tripped] 400 main feeder circuits,” says ComEd President and Chief Operating Officer J. Barry Mitchell. “At one point, a tornado warning even required us to evacuate our central command center.”
Despite the storm’s severity, ComEd restored power to 75 percent of affected customers within one day. Further, 90 percent were restored within two days, and all were back online five days after the outage began.
Credit for this fast response extended throughout the company. Less apparent to customers were the significant procedural enhancements ComEd developed long before the storm of the decade appeared on weather radars.
ComEd’s effort to improve storm-response procedures occurred in two stages. The first began three years ago when ComEd undertook significant changes to reduce the duration of storm-related outages. Second, a sizable storm in October 2006 left the utility better prepared to respond when an even larger storm arrived only 10 months later.
To put these improvements in perspective, in 2003 and 2004 ComEd’s average length of an outage during typical storm situations (known as “storm CAIDI”) was 254 and 230 minutes, respectively. However, by the time of the August 2007 storm, ComEd’s storm CAIDI was down to 130 minutes. In other words, the length of ComEd’s average storm outage was cut nearly by half in two and a half years.
One of the most important changes implemented was the 2005 decision by former Chief Operations Officer John Costello to expand the company’s emergency response rotation from four to eight storm teams. Each team comprises up to 130 individuals who staff coordination roles in the central command center and four regional storm offices. These roles are in addition to the hundreds of frontline crews called out to perform field restoration work.
This change more fairly shared the storm burden across the organization and helped guard against individual burnout. However, implementation required training and drilling more than 300 new individuals, in addition to continued drilling for members of the previous four teams.
In rural areas of ComEd’s 11,300-square-mile service territory, it can still be hard to staff all eight teams with the ideal combination of experience and training. In some cases, personnel are trained for storm roles that don’t perfectly overlap with their daily duties, or they must travel more than an hour to distant regional reporting locations.
As ComEd works to correct these issues, the change has resulted in a significant drop in CAIDI during storm situations, in which minutes can mean everything. This has cut the average storm restoration time by about two hours and has improved how the company is perceived by customers and public officials (see sidebar, “Silver Lining: Customer Communications.”)
These weren’t the sole changes implemented in this period. For example, at the lineman level, ComEd established a “tiered staffing” approach to prepare proportionately for different weather situations. Instead of putting a large proportion of frontline employees on call for every approaching storm, the company now reserves “all hands” callouts for only the most serious weather events. This reduces “cry wolf” fatigue, while increasing the overall response rate to both large and small events.
In addition to the procedural changes instituted during the 2003 to 2005 period, ComEd took advantage of lessons learned from an October 2006 storm that left 481,000 customers without service. Given the even more powerful storm that would hit less than a year later, the importance of the October event cannot be overemphasized. ComEd hadn’t had an event affecting a half million customers in about 10 years. The inevitable result was a loss of institutional memory. People retired, left the organization or simply forgot how to best handle a storm of that scale.
In many ways, the October 2006 storm provided ComEd the best training possible for the August 2007 storm, allowing the company to test its people and processes, and develop corrective actions in ways that no drill ever could. This test paid off in August 2007, when ComEd was able to restore power to 150,000 more customers than were affected a year earlier—and to do it in five days instead of six.
During the October 2006 “lessons- learned” effort, one immediate finding was the need to better manage outside crews. ComEd hosted about 130 foreign crews in 2006, at nearly two times their normal salary. To control costs, it’s important to ensure those crews devote as much time as possible to patrolling lines, fixing poles, and restoring service.
What ComEd learned was foreign crews spent more time off-task than preferred. This included waiting to receive work pouches and extra time driving long distances to work sites. In some cases, duplicate work assignments were made, with ComEd and foreign crews showing up at the same location.
Building on these lessons, ComEd’s management of foreign crews improved significantly in August 2007. The company hosted about 275 outside crews from as far away as New York and Houston. Despite more than doubling the number of foreign crews compared to 2006, they were deployed in a far more productive way.
The August 2007 storm front hit on a Thursday night. Within three hours, ComEd personnel made the first requests for mutual assistance. The first foreign crews arrived Friday, and all were in the territory by Saturday, providing far more resources, more quickly than in previous large storms. By comparison, during the October 2006 storm, the company requested mutual assistance on the second day.
But as the October 2006 storm demonstrated, it’s not just having the manpower on property, or how well that manpower is deployed that makes the difference in restoring power quickly.
In its use of field command centers and staging sites, ComEd follows the Federal Emergency Management Agency’s model. In addition to the central command center and regional storm offices, the use of field locations provide more structure and organization in the field, with a vice president staffing each field command center to speed decision making.
A key learning from the October storm was the need to expand the number of field sites during large events. Consequently, in 2007 ComEd was prepared to establish two incident command centers in the hardest hit areas instead of the one center used in October 2006, and the company increased staging sites from two to five. The decision to do so resided with the emergency response directors, which for this storm included ComEd vice presidents such as Tim McGuire, Mike McMahan and Tyler Anthony.
In terms of foreign crews, going from two to five staging sites allowed ComEd to hand out work packages closer to the location of crews’ assignments. This reduced drive times for outside crews and increased their time spent patrolling lines, installing poles and wires, and restoring customers.
Additionally, ComEd increased foreign-crew productivity by improving assembly of work pouches, including work assignments, system maps, driving directions and supplies. This began with deploying double teams early on to do more damage assessment in the field and assist with work-package assembly. The supply department also set up multiple remote warehouses to expedite the delivery of materials to job sites. When foreign crews reported for their first shifts, not only had the company identified more work for them to do, but they spent little time waiting for work packages and supplies.
Carl Segneri, ComEd vice president of quality assurance, and Kevin Brookins, vice president of new business/work management, oversaw all foreign crews. They kept a separate “foreign crew war room” open for the duration of the restoration. The war room consolidated all issues related to the foreign crews, such as arranging and deploying truck rentals, securing hotel rooms, and eliminating the duplication of work assignments.
When all restorations were complete in Chicago’s south suburbs surrounding the Village of Frankfort, instead of receiving work packages, the next shift of foreign crews received full instructions and maps to drive 50 miles north to Northbrook. This allowed the utility to relocate 36 crews and an entire reporting center without missing a beat.
ComEd also implemented a new approach for attacking feeder lockouts. Called “feeder SPOC,” the tactic designates a single point of contact (SPOC) in the field as sole authority for managing all restoration work along the main stem of a feeder. Although it had yet to be finalized as a formal ComEd procedure when the storm hit, quick dissemination and implementation of the approach played a major role in the rapid restoration of many lockouts.
Normally, all restoration work along a distribution line is controlled and coordinated by a distribution dispatcher in the centralized operations control center (OCC). This ensures all work is conducted safely, and system configuration changes properly are recorded. However, during a large event like the August storm, the volume of work tickets and magnitude of radio communication into the OCC favors down-streaming some decision-making to a first-line supervisor.
Through the feeder-SPOC approach, a locked-out feeder is assigned to an experienced overhead crew supervisor knowledgeable in both system configuration control and the lock-out/tag-out procedure. The SPOC and assigned crews then make their way out from the feeder’s source at the substation. The SPOC is responsible for identifying and isolating all instances of main-stem trouble, directing all switching operations and the repair of damaged facilities, ensuring safe work zones, and managing the subsequent restoration of the main feeder stem. The SPOC also keeps the OCC dispatcher updated on configuration status and customer restoration progress.
This approach originated during the October 2006 storm. Jim Conway (then Southwest manager in construction and maintenance, now OCC director) oversaw several foreign contractors who worked hurricanes in Florida. They explained how Progress Energy takes a similar approach at the substation level. Although ComEd typically doesn’t lose entire substations in a storm, Conway thought the concept could apply at the feeder level, and coincidentally only a week before the August storm hit, he spoke with people at Progress Energy about their approach.
In the August storm, ComEd employed 25 feeder SPOCs overseeing about 100 crews. Foreign crews especially appreciated the tactic. Given their unfamiliarity with ComEd’s territory and system, they liked focusing on a single de-energized circuit underneath a local authority. From a customer perspective, it allowed the utility to restore the largest number of customers in the quickest, most resource-efficient manner.
To fully understand what worked and didn’t work, ComEd launched a thorough evaluation of emergency preparedness processes.
Typically, about 50 participants take part in a post-storm lessons-learned conference call. In this case, Emergency Response Managers Nichole Tillman and Bob Moyer led an effort that included 23 recognition breakfasts for line crews, at which frontline feedback was collected. Additionally, ComEd conducted dozens of interviews and team meetings, culminating in an all-day lessons-learned meeting involving more than 120 key stakeholders.
This effort produced a detailed understanding of what went well and, equally important, what new process changes could be incorporated into the 2008 training and drill plan. Some key lessons learned included the following:
• Customer Interface and Communication: Overall, ComEd communicated well with the public, the media and municipal officials. The company also improved its system for communicating local ETRs (estimated time to restore) to customers. However, feedback indicated customers still expect even more detailed updates, potentially utilizing online options, which the company will examine.
• Resource Management and Utilization: Despite streamlined communications from the feeder-SPOC approach, several crews reported delays reaching work dispatchers. Reducing communication delays will improve restoration work and efficient utilization of personnel. ComEd also is considering opportunities to further leverage staff from other groups.
• Emergency Response Team Proficiency and Qualifications: Despite efforts to improve the proficiency of all eight emergency response teams through training and drills, some team members continue to report being in unfamiliar roles and requiring extra time to reacquaint themselves with the tools on which they were trained.
The August 2007 storm provided a watershed moment for ComEd. It energized the company’s employees and built morale in ways that only excelling under extreme circumstances can allow.