A Fine Mess


CIP audits show utilities are just getting started with securing the grid.

Fortnightly Magazine - June 2008

Bad news from the front lines in the cyber-security war: Little meaningful progress has been made toward safeguarding the nation’s electric grid from malicious attacks. Initial cyber-security assessments and audits suggest few companies really are ready to implement the first wave of NERC critical infrastructure protection (CIP) standards, despite the fact the utility industry drafted the regulations.

“I honestly don’t believe the industry placed a high probability on the standards becoming law,” says Brian M. Ahern, president and CEO of security vendor Industrial Defender. The resulting anxiety has been a boon to companies like Ahern’s, whose phones are ringing off the hook.

“The industry is in a frenzy,” he says. “These regulations have passed. They have real teeth, to the tune of a million dollars a day in fines. They have some milestone dates looming, June 30th being the first. Yet very few utilities have done anything beyond assessing their level of risk.”

The initial CIP phase largely is procedural, covering personnel background checks, password protection and auditability. As fundamental as those things might seem, they’re big news to the industry, and the broad language of the rules is open to much interpretation.

“A number of utilities have gone back to NERC and FERC looking for clarification,” says Robert Sill, CEO and president of security firm Aegis Technologies.

Approaches used by utilities to handle security falls across a wide spectrum. At the most mature organizations the security function is managed enterprise-wide at the executive level. But more frequently security is pushed down the org chart, creating a kind of organizational balkanization, with predictable duplication of efforts and vulnerabilities.

Add to the equation the industry’s capital constraints, aging infrastructure and cultural resistance to change, and the result is a potentially dangerous cocktail. The only clear things are the many obstacles that stand between utilities and effective cyber-security.

The Enemy Within

Despite prevalent fears of terrorist hackers, or the pimply-faced otaku bent on taking down the grid for kicks, experts say the biggest threat is both more familiar and more pervasive.

“We had an incident last summer in L.A. where a disgruntled worker came in and basically slammed his fist down on the red button labeled ‘Don’t Push,’” says Darren Highfill, a security architect at EnerNex Corp. “Fortunately, it was on a Sunday and it didn’t cause anywhere near the problems it could have.”

Inside actors still present the greatest security risk. Ahern says only 10 to 15 percent of the total risk profile comes from external threats. The rest involves people who have intimate knowledge of—and access to—mission-critical systems. The examples of their impact are chilling.

“We’ve found laptops taped underneath desks, deploying malware attacks against servers with the malicious intent of taking down plants,” Highfill says.

At the other extreme are well-intentioned people who simply make mistakes. Ahern tells of an IT intern who inadvertently punched through a firewall and deployed antivirus software, a huge no-no.

“Risk comes in a number of forms,” Ahern says. “Everybody talks about external risks. The CIA reported in January four cities in Central and South America were actually attacked by outsiders and held for ransom. ‘Either pay us X, or we’re going to shut the power down.’ On the heels of that and the Aurora Project [in which a U.S. national laboratory demonstrated how a cyber attack could damage a power plant, see “Lessons Learned: Aurora Atack," January 2008], people are talking about it. The question is: can an outsider really get in and do that? My opinion is no.”

Highfill is similarly circumspect about the level of external danger. “It’s a serious threat, but there’s no shortage of sensationalism out there,” he says. What we haven’t seen is a massive blackout that has irrefutably been traced to a malicious action. Or if we have, it has been covered up in fantastic fashion.”

Utilities and government authorities avoid sharing information on security incidents. No one wants to be perceived as vulnerable, so data on security breaches remains closely guarded.

As a result, utilities and regulators are left trying to address perceived threats from terrorists and even foreign governments, as well as the broader exposure of their own employees, with only scant baseline data on the attacks the industry has faced to date.

Culture Clash

Ironically, one thing the industry has going for it is the result of its own slow-moving ways. Technologically, the waters are well-charted.

“The hacking industry is 25 years old,” observes Sill. “Other industries have learned a lot of lessons the hard way. So there are solutions out there that can be put into place very quickly. The path is known, but the challenge is culture.”

The industry’s control systems aren’t all that’s old-fashioned. It can be tough to get the necessary buy-in from systems operators who are set in their ways. Training employees and managing the cultural shift is a huge part of the puzzle.

“Raising awareness of vulnerabilities in the systems, let alone understanding the minds of hackers, is something that’s completely foreign to the people who are running systems today,” Sill explains. “There is a great deal of skepticism. These people have lived one way for decades, and now we’re telling them white is no longer white.”

There’s also a generation gap. Many operators are approaching retirement, with replacement workers in short supply. At the same time, IT professionals often jump to implement changes that can wreak havoc on 30 year-old control systems, without taking time to understand the tried-and-true technology.

In a typical enterprise environment, confidentiality of data is the highest concern, followed by integrity of data and finally the availability of data. In control systems, it’s the reverse: Availability is paramount, and until now confidentiality largely was ignored.

But with controls using internet protocols and becoming interconnected, the traditional security-by-obscurity model is changing.

“Interconnection, while it has been a huge boon to business, pretty easily allows a malicious entity to play hopscotch,” Highfill says. “They don’t have to compromise the latest equipment you’ve deployed, just the weakest link in the chain.”

Paying the Price

The industry has met the security challenge with much gnashing of teeth, largely because of the added expense at a time of rising cap-ex and fuel costs.

“Let’s not kid ourselves, this is one of the most capital-constrained industries in the world,” says Ahern. “I spoke to one utility that estimates it will cost $30 to $40 million over the next three years to become compliant with the NERC CIP standards. Where’s the money coming from?”

The key might be finding a financial upside in security investments. This would require utilities to view security as more than solely a compliance issue, and to put pressure on security vendors.

“One executive asked me, ‘how are you going to make this pay for itself?’” Sill recalls. “So we added a number of troubleshooting features. Since we see all the data and encrypt it, we also can do problem analysis. Then they don’t have to roll a truck to a substation to find out what’s going wrong. There’s an easily identifiable cost benefit to having this technology in place.”

There’s also a business case in being the best in class. After all, investor-owned utilities that look the sharpest on security also will look good to investors on Wall Street, because they are hedging against downside risk. In an industry moving toward greater interoperability, demonstrable security could prove a competitive advantage.

“Utilities take issues of operational stability very seriously,” Highfill says. “They’re proud of their track record, in terms of up time and level of service. If they can make some claims and guarantees on that, that’s a fantastic motivator in our industry. You want to be perceived as the rock.”