ERCOT utilities approach CIP compliance from varying perspectives
Scott M. Gawlicki, a Fortnightly contributor based in West Hartford, Conn., has been writing about the power industry for nearly 20 years. Fortnightly Editor-in-Chief Michael T. Burr contributed some content for this story. Contact Gawlicki at firstname.lastname@example.org.
As proposed by the North American Electric Reliability Corp., the new critical infrastructure protection (CIP) standards charge utilities with identifying their own critical assets and related cyber systems.
This approach allows great flexibility for utilities to apply the CIP standards to their particular situations. This will help ensure that their efforts focus on securing critical assets, rather than on complying with an overly prescriptive set of mandates that might or might not yield a secure grid.
The same flexibility, however, is creating an unnerving level of uncertainty among utilities facing a looming compliance deadline.
“You’ve got every organization under the sun taking their own guess about what should and shouldn’t be considered a critical cyber asset,” says Darren Highfill, CISSP and utility communications security architect for EnerNex Corp., an engineering and consulting firm based in Knoxville, Tenn. “Until the standards are finalized and NERC starts doing audits, we’re speculating about where the line will be drawn.”
Under the current schedule, the new standards will become legally enforceable in 2009. Between now and then, however, the standards might evolve. In a recent Notice of Proposed Rulemaking (NOPR), FERC asked NERC to provide further guidance on how utilities should focus their “risk-based methodology” (see “Setting the Standard”).
“The regulated entity determines whether it has critical physical assets and assocated critical cyber assets,” says Joseph McClelland, director of FERC’s Office of Electric Reliability. “That discretion could lead to inconsistencies, and those inconsistencies could lead to vulnerability on the system. We’d like to see modifications to the standards and process to address those potential problems.”
Utilities can’t afford to wait for a refined set of standards. To ensure they are compliant when the standards become enforceable, utilities are working to define their critical assets today—even as they watch to see how their definitions might need to change tomorrow.
Oncor On Track
Since it’s up to each entity to develop its own way of identifying critical assets, their methodologies run the proverbial gamut.
“There are differences in what people consider critical and the strategies being applied,” says Bill Bojorquez, vice president of system planning at ERCOT, which has formed a CIP Advisory Board to provide guidance to its membership. “Substation duty in Houston will be viewed differently from a similar sized substation in a rural area. Some utilities are more in tune with the process of developing their methodologies than others. Our goal is to help their program engineers understand and meet their compliance requirements.”
Those requirements vary substantially within ERCOT, because of differences among various utilities, large and small.
Oncor, for example, is the largest electricity delivery company in Texas, providing power to more than 3 million customers and operating more than 115,000 miles of transmission and distribution lines. To identify its most vulnerable and critical assets, Oncor turned its transmission system planning and reliability experts loose to perform what it called “consequential studies.”
“Our transmission planning group conducts N-1 studies regularly to ensure the highest possible system reliability,” says Steve Martin, TMS project manager with Dallas-based Oncor, a subsidiary of Energy Future Holdings (formerly TXU Corp.) “These are heavy-duty reliability guys. They do N-1 studies in their sleep.”
The methodology evaluates a wide variety of scenarios, ranging from a cyber attack or outage affecting a single substation, to a series of substations, and the effect each would have on other assets, the service area and region. Each scenario revealed whether an attack could be isolated, or could spread quickly throughout the Oncor power grid.
“I suspect quite a few utilities are doing it this way. Rather than performing thousands of voltage-stability studies, we saw it as a calculation issue,” Martin says. “The problem is you can’t predict what state the (bulk power) system will be in when the event occurs.”
Oncor’s methodology proved most valuable when applied to the off-peak maintenance season—when equipment is being changed out, or certain generators are shut down for a planned outage. “During the peak-load season all the best generators are available and there aren’t any planned equipment outages,” Martin says. “But either way, it’s impossible to model every possible scenario.”
On the smaller end of the scale is the city of Garland, Texas, a municipal utility that operates three gas-fired generating plants, 133 miles of transmission lines and serves approximately 68,000 customers.
“We did most of it a year ago,” says David Grubbs, transmission manager for Garland. “The CIP is fairly clear in terms of what physical assets you should look at. You’ve basically got a checklist to go by (see sidebar, “Critical-Asset Checklist”). We’re using steady-state load-flow analyses, which basically show how overloading one asset will affect the others.”
But the guidelines can get a bit hazy, even for a comparatively small entity like Garland.
“The way I read it, if you have two computers three feet away from each other in a critical location and they communicate with each other by router protocol, that triggers the cyber security rules—even if the wire connecting the two never leaves the room. To me, that’s overkill,” Grubbs says.
Further, the standards do not apply to assets operating at the 69-kV level. That’s a problem because Garland relies on neighboring municipal Greenville Electricity Utility Systems’ 69-kV peaking unit for black-start support.
“A lot of black-start units are 69 kV because they’re smaller and easier to start,” Grubbs explains. “Greenville is designated to bring up our units if a blackout occurs. But they’re not considered critical under the CIP standards, even though their system is obviously critical to us.”
Fortunately for Garland, Greenville has begun applying the standards to its system as well, even though it’s not legally required to do so. “Actually if we all go dark, I can’t start our peaking generator without my diesel generator, so that’s the real critical asset,” says Greenville Power Supply Manager Rick Gillean.
Greenville, Texas, with roughly 13,000 customers, has spent nearly $250,000 on physical security measures thus far. Security-card access systems and roughly 40 security cameras have been installed at the utility’s peaking plant, substations and control rooms. Distribution system relays have been upgraded to improve distribution system reliability. And security checks are now standard practice for all employees.
Further, Gillean recently brought in a consultant to begin the process of identifying cyber assets. “We could sit on our laurels but we’ve chosen not to. Right now we’re excluded from the CIP, but that could change,” Gillean says. “The biggest thing I learned from my consultant is that the cyber guidelines are basically open to interpretation. There are a lot of gray areas, so everybody’s guessing.”
Such issues are bound to occur as the asset-identification process continues to unfold. “I don’t really view [the 69-kV issue] as a loophole,” Bojorquez says. “I see it as a potential improvement we will make down the road as we learn more. These are brand-new standards. There are a lot of implementation questions and there will certainly be additional work” (see “CIP Goes Live”).
For large utilities with multiple divisions, the focus needs to be on getting all the businesses on the same page.
“For a large-scale utility, the challenge is getting all the right people in the room,” says Will Tang of Digital Security Consulting Inc. of Glendale, Calif. “The transmission and distribution, corporate IT, and other engineering groups share computers, servers and overlapping network infrastructure. By involving the right people early on in the planning phase, you can be sure you’ve identified all the assets that require compliance.”
Most important, this collaborative process will allow utilities to identify other areas that are vulnerable to cyber attack. “Identifying the critical cyber assets is the hard part,” Tang says. “Once you determine where to place the fence and how big it needs to be, building it is much easier. Implementing security controls is something security companies have been doing since passwords were invented.”
Regardless of the methodology, the cyber-asset identification process will be an ongoing venture—as standards evolve, network systems change and technology advances.
“Right now you have some technologies that are good, but old; some that are newer; and some that are brand new,” Martin says. “As we go forward and the old technologies are replaced, the way we view each asset will change. New technologies will bring new levels of risk and designers will have to build in protective measures to mitigate that risk. So it will be a never-ending cycle. We’ll get smarter and the hackers will get smarter.”