Plugging cyber security holes isn't as easy as everyone wants to think.

It's the elephant in the room in this post-Sept. 11 era. When asked, many in the know claim the energy industry's cyber security is fine, mostly, and the industry is working diligently to fix the few problems that remain. But some who say that the energy industry can no longer practice security by obscurity also caution against revealing even the most basic outlines of problems that confront the industry.

Research scientists at EPRI, Gas Technology Institute and Schweitzer Engineering Laboratories were willing to talk extensively to about the state of cyber security and energy infrastructure. The fact is, they say, there are significant vulnerabilities to the cyber infrastructure in the energy industry that, if left unaddressed, will continue to expose the grid to attacks. Some of the vulnerabilities cannot be fixed with any currently available technology-hardly a comforting thought. The good news is that much of what can currently be done to defend against attacks-in cyberspeak, to "harden" systems and networks-is either in place, or available at a fairly modest cost. And the technology that can plug the remaining gaps in energy industry cyber security may be on the market within a year or two.

Old Vulnerabilities and New Levels of Malevolence

Well before Sept. 11, there was concern in both government and industry about the vulnerability of the energy infrastructure to cyber attack. In 1997, President Clinton issued Presidential Decision Directive 63 (PDD-63), which named the energy industry as a key critical infrastructure component. PDD-63 also established the National Infrastructure Protection Center (NIPC), a federal agency housed within the FBI to provide warnings of threats to the nation's critical infrastructure. NIPC in particular works with the North American Electric Reliability Council (NERC), which was designated the lead agency for electricity by then-Energy Secretary Bill Richardson.

Part of the concern about critical infrastructure, in the energy industry as well as in other sectors, grew out of the need to address the problem of Y2K. And though Y2K was a hugely expensive headache for the energy industry, it appears that it will offer a silver lining two years later. "Y2K exposed a lot of infrastructure problems that companies might have had," says Jayne Brady, spokeswoman for Edison Electric Institute.

According to Jim Fortune, area manager for enterprise infrastructure security at EPRI, the industry learned a lot about its systems during Y2K remediation. "We also developed mitigation strategies, in case something did happen, such things as alternate forms of communication," he says. Preparations, like having alternative forms of communication and contingency plans in place, and making plans for which personnel to bring in to diagnose problems, he says, are directly applicable to a terrorist cyber attack. "Much of Y2K will pay dividends for this type of activity we're engaged in now," Fortune says.

Yet cyber security and infrastructure problems were being addressed at a far less urgent pace than Y2K had been. Fortune notes that "September 11 really did start to get the sensitivity at the senior level among utilities" on cyber security issues.

"Before Sept. 11, the industry didn't realize the malevolence of potential attackers," says Paul Oman, senior engineer, research, with Schweitzer Engineering Laboratories. "Now," he says, "we recognize that things are different. Terrorist attacks come on multiple forms. We must ask ourselves how hardened are our critical targets, and how can existing technology be adapted to make them less vulnerable?"

In response the question of whether cyber security has improved since Sept. 11, Fortune says there has been "no change." But, he says, it's a complex answer to a simple question. Cyber security in the energy industry "is unfortunately certainly less than desirable, but honestly the complexities of improving in that kind of time frame were simply-well, you couldn't do it," he says. "Since September 11 it would not even have been possible to start putting security into all these systems" that experts in the industry have known were vulnerable for some time.

Security by Obscurity No More

The biggest cyber vulnerability to the energy industry arises in many ways from the same cause as Y2K: systems not designed for the 21st century. The Achilles' heel of all utility systems, including electricity, gas, and water, are SCADA (supervising control and data acquisition) systems. SCADAs control supply and delivery systems, making decisions in about 4 milliseconds, rather than in the seconds or minutes that human decision-making used to take. And SCADAs are integral to utilities' ability to cut costs and be more competitive, according to William F. Rush, assistant institute physicist at the Gas Technology Institute. For example, on the gas side, city gate stations used to be staffed with operators. Now, he says, "with the combination of available and relatively low-cost telecommunications technology, those stations now are often unmanned. As time is going on, there is a heavier and heavier dependence on that telecommunication function. It's driven by competition."

The same is true for electricity. As the North American Electric Reliability Council says in its publication, , "[a]s deregulation and competition drive utilities to reduce costs and provide a higher quality of service, many utilities are automating substation operations-employing intelligent electronic devices-so equipment can be remotely accessed."

The problem, according to Rush, is that automation systems were built starting 100 years ago, and they were all designed to function fairly automatically, but with no communications. "Basically in gas, water, and electric, you're looking at a mid-20th century system that has a late 20th century communication overlay on it," he says.

And the vulnerability to the infrastructure is that communications overlay.

"All of these systems, when they were developed, were developed for their functionality," Fortune says. Functionality can be defined, in simplistic way, as speed and accuracy. "Since these systems were developed for functionality, there was no real consideration given to security," he says. "The assumption was that if you were using that system, you were supposed to be there. And that has persisted to today. We have these systems running with little, if any, real security on them. So in a sense, they are vulnerable to cyber attack. And that's sort of the bottom line."

The industry imposes very tight requirements on SCADA speed, around a 4 millisecond message rate. It's the very speed of the SCADA systems that makes the problem of securing SCADA systems difficult. "What works on IT systems-that don't have that speed requirement-wouldn't work on our systems," Fortune says. Packet-type encryption will simply not work on SCADA systems because it is simply not fast enough, according to Fortune. For many of the systems, hardware already operates at such slow speeds, e.g. using 286 processors, that if security protections were added on top, the processing time would be unacceptably delayed.

In the past, Fortune points out, SCADA systems have been relatively secure, because they had "security by obscurity." They were developed by vendors using proprietary protocols, software, and hardware, he explains. SCADAs were fairly isolated, and ran the system. Anyone who did try to penetrate a SCADA system "would have had a great deal of difficulty getting in, and if you did, understanding what was going on, because it was all proprietary. That is not the case today," Fortune says. With the deregulation of the utilities, grid operators and power marketers need marketing information that resides in such systems, such as what plants are running, whether they are running at full power, and whether they have been taken offline, among other things. So now, Fortune explains, there is a movement to tie what had once been isolated systems into corporate business systems. These corporate business systems are also Internet connected. "Now your threat has just escalated exponentially. The minute you tie into the Internet, there are over 200 million users, estimated, and you sort of have to assume there might be one or two out there that might have some malicious intent, right?" Fortune asks.

As Oman puts it, "if you have remote access, for example networks and telephone lines, you have vulnerability. Any time you have access, there is a threat of electronic intrusion."

No Quick and Easy Fix

Oman says that there is a lot of technology available on the market that can be used to harden systems to cyber attack. But many of these security methods and technology must be adapted and improved from commercial grade to industrial grade, he says. Most modems and virtual private networks (VPNs-a device that encrypts communications between set points) are commercial grade, not industrial. Industrial grade equipment requires a -40° C to 80° C temperature range, and shock resistance to withstand earthquakes, he explains.

Security varies across industry on SCADA systems, Oman says. Most of industry before Sept. 11 had some level of password access, he says, though maybe only 1 level. Many were catching up to 2-level password standard. Some of the newest devices, including Schweitzer's, have 4-level password security, with more authentication and security built in.

But password protection is simply not enough to protect SCADA systems. What is really needed is encryption and intrusion detection systems that operate at the same speeds as SCADAs. While there are many encryption algorithms available that suit IT and corporate enterprise system security needs, that is not necessarily the case for SCADA systems. Rush says "it is not a case that the algorithms don't exist, they do. The issue is to find one that is compatible with gas operations, water operations, and electric operations." What that means is the right combination of speed and security.

And as Fortune points out, the solution may not come quickly. "NIST [National Institute of Standards and Technology] has been working since the 1980s to develop intrusion detection systems for these types of systems, without success." On Oct. 1, NIST awarded a 2-year contract to Schweitzer to develop both procedures for hardening substations, as well as alogorithms for regional power control.

When PDD-63 came out-well before Sept. 11-GTI received a contract from the Technical Support Working Group, a government agency. That contract directed GTI to select an encryption algorithm that would meet the needs of the gas industry, but with an eye to water and electric, according to Rush. The contract also stipulated that at the conclusion of the research, GTI would give away the algorithms, the code, and everything that would help a manufacturer put such security into SCADA equipment.

Rush says he was thrilled at the prospect. "I sallied forth, so happy to be able to give people this wonderful new toy. Then, I just hit a brick wall. Manufacturers had no interest, even though we said it was free." Rush doesn't necessarily fault manufacturers. While the code was free, changing manufacturing and engineering systems was not without cost. And, more importantly, manufacturers told Rush that there was no demand from customers for such cyber security. It was 1998, not 2001, when Rush initially approached manufacturers.

Rush was not deterred. He went to the customers, specifically the gas utilities. Although receptive to the idea that cyber security was a concern, organizations such as the American Gas Association deferred consideration-Y2K was looming, and the need for the type of cyber security Rush was talking about did not seem particularly apparent. Ironically, Rush was scheduled to make a presentation to an AGA committee in October 2001. Instead of talking about the need for such security, the AGA wanted to know if it could be made available by the following May.

There is no doubt that like so many other things, after Sept. 11 the industry's view of security changed. While many, many discussions are occurring, the problem, according to Rush, is that most utilities will tell you that their system is bulletproof. Often, though, they aren't, he says. When he presses utilities on why they feel their SCADA systems are secure, Rush says a typical answer is, "I have a protocol that no one knows." Then he asks questions such as "could I get the manual from the company that makes the protocol?" Then utilities realize that the protocol is in the manual, available from the system supplier. "So what you quickly realize is that what you are hiding behind as a bulletproof shield is in fact fairly pathetic," Rush observes.

Rush says that there are lots of encryption algorithms-and the strength of the best ones lie not in the fact that they are secret. Basically, he says, what you want to do is pick an encryption system where the security lies in the key, not in the mechanism-like a lock. You can find out how to make that lock, he says, but that doesn't enable you to open any lock. "What you're really looking at is the key. Encryption algorithms are the same way." So Rush advises those in the industry to pick one that's been attacked by mathematicians for years and years. "If no one can crack it, that's a pretty good bet."

Rush says that in testing the algorithms chosen by GTI, he and others came to the conclusion that it makes a difference how they are implemented. "If you implement them just by sticking them on an existing SCADA system, it will probably slow it down too much," he says. What is needed, he says, is to modify the SCADA RTU-remote terminal unit-the computer that actually processes the information, by putting the algorithm on a separate chip. The chip, he says, costs about $5, "so it's no big deal-and it already exists."

The problem is, Rush notes, that utilities need to put the chip into their SCADA units. The cost for doing so? "The best estimate I can come up with is it would add something on the order of $50-100/RTU." Such cost would add little to the overall RTU cost of about $3000-5000. Rush estimates that a mid-sized utility system has around 100 RTUs. "So we aren't talking a huge amount of money," he says.

Some utilities consider those kinds of cost a sound investment. Alliant Energy has already spent between half a million and one million dollars to harden its systems to cyber attack, according to Erroll Davis, president, chairman, and CEO of Alliant. Davis has been speaking to the utility industry since 2000 about the need to defend against cyber attacks. At first, he says, the reaction was akin to his early Y2K education efforts-he was accused of preaching doom and gloom. He isn't getting that same reaction any longer.

Heads in the Sand?

Rush voices worries about the way IT departments and those operating SCADAs think about SCADA security. Although IT managers go to great lengths to harden utilities' corporate networks, Rush says they may overlook off-site SCADA systems, the backups for which are often connected to inexpensive telephone lines. "What happens is the SCADA system will be located several miles away from the facility. And, in order to provide robustness to the system, it's typical that they will have at least one communications backup system, and sometimes two. Which is good from the robustness of the system perspective, but it's bad from the perspective that it gives me three unsecured routes in which to attempt an attack." What that means, Rush says, is that there's a telephone number such that if someone dials it, they call into a utility SCADA system.

SCADA engineers often protest that an unlisted number that cannot be discovered protects their system. Rush then asks them if they've ever heard of a wardialer.

Wardialers are pieces of software that quite simply, dial every number in an exchange. All 10,000 of them. Specifically, wardialers dial an area code, an exchange, and then 0001, 0002, etc. Using a wardialer to find a specific person would be difficult. But wardialers aren't looking for people. If someone answers, or if an answering machine picks up, the wardialer hangs up. If the wardialer gets a beeping or hissing kind of sound, it responds with a fax tone. If nothing happens, it switches to modem sounds. If still nothing happens, the wardialer makes a record that says, essentially, the number at (what was just dialed) doesn't have voice, doesn't have a fax machine, and doesn't have a computer modem. What has happened, Rush says, is that the wardialer has winnowed 10,000 possible numbers down to a much more manageable set. Then, a hacker can start to test.

Rush says, "[i]f you think that it would be hard for you to hack, I suggest you sit down at your computer and type into a search engine 'hacking sites' or 'telephone hacking.' I'm sure you can find a wardialer, for free."

These hacking tools, Rush says, are all known not only to hackers, but also to people who worry about cyber security. Such tools are less familiar to people who operate SCADA systems, he says, who often operate outside the purview of IT departments. "They want to run their own systems, and they don't want IT messing with them. And they're completely convinced their system is bulletproof," Rush says.

As Fortune points out, when it comes to critical infrastructure protection, "we're not talking the script kiddies." Those who could perpetrate a serious attack would have knowledge of both the electrical system and computers, and would have a malicious intent, according to Fortune. "Now that's not just terrorists. There's a long list of folks out there like Iraq, China, and North Korea, nation-states who are developing asymmetric warfare capability." These nations are knowledgeable, Fortune says, because the very same vendors-Siemens, ABB, GE, and Westinghouse-who sell to the United States sell systems to those countries as well.

Share and Share Alike

The solution to cyber security boils down to sharing information-of the right kind. As Fortune points out, "security awareness is the key." That awareness, he says, needs to be not only at the top levels within companies, it also must be an education process throughout the entire infrastructure. Educating employees about cyber security issues is, he says, "the most bang for the buck you can get."

Indeed, companies can overlook the fact that despite all the attention paid to terrorists and hackers, disgruntled employees are the most likely source of attack on cyber security. ()

Of course, companies must get information from somewhere. EPRI has a very active information sharing program amongst the major utilities in North America and internationally, Fortune says. "The basic premise is that no one organization has all the answers, but collectively we can probably come up with some pretty good ones. What we're doing is sharing information on how to protect, and how to mitigate, and also developing R&D agenda that we can get funded and develop the technologies," he says.

Lou Leffler, manager projects, North American Electric Reliability Council, agrees. "The electricity industry is very complex, very diverse, with a lot of members in it," he notes. All the industry participants-big investor-owned utilities, government utilities, co-operatives and municipal utilities-must be communicated with, he says. "Every one of them is critical, some more so than others, but it might be some little co-op out in the middle of who-knows-where that serves a military base. They're all important, and we've got to get the word out to them" in the event of a crisis, Leffler says. He also points out that communication from the Electricity Sector Information and Analysis Center is only the beginning, because now the information has to ferret its way down through the organization to the action people in the organization.

While sharing information about cyber protection is encouraged, it's a different story when it comes to sharing real-time data between industry participants. Grid operators, in particular, accumulate critical information that could be valuable to evildoers with access to their systems.

One solution is to segregate critical operations systems from other business systems, which is what the California Independent System Operator does, according to Cal ISO public information officer Gregg Fishman.

The risks of not separating critical operations from corporate enterprise computers are serious. As Schweitzer's Oman says, "the minute you put them together, you no longer have reliability and security. Giving access to marketing information and infrastructure operations is opening the door to literally every hacker in the world." He applauds Cal ISO's approach. "You have to separate grid protection from grid management."

But there is often temptation to combine systems, Oman says. There is a lot of pressure to combine the two because executives want to look at metering information, he says. Instead of setting up a new technology scheme, i.e., a second system not connected to the Internet, with its added costs, executives say "let's just use one system," Oman says. "But the minute you do, everything's toast," he says.

James Sample, Cal ISO's Manager of Information Security Services, says that he believes that the utility industry has taken the right steps in the prevention of cyber attacks, but that sometimes those steps are very inconsistent. "A majority of utilities are privately held and due to cost issues and the lack of federally mandated requirements, security is often overlooked. In addition, there is a shortage of security professionals that understand the utility industry requirements." A good example, he says, is the healthcare industry. Until the Healthcare Information Privacy Protection Act, the health care industry was aware of the need for information security, but implementation across the industry was inconsistent and in some cases non-existent, Sample says. The same was true for the financial industry. "Until the Federal government develops requirements and standards, such as they did with the healthcare and financial industry, the security implemented in utility industry will lag and deployment will be inconsistent," Sample says.

Fortune is not quite so ready to call in the regulators. "I think, now that it's on the radar scope, [industry leaders] are starting to ask the right question, what does it cost to do this, to do that?" At the end of the day, he says, industry needs to develop the technology, including encryption algorithms that allow encryption of real-time operating systems, and intrusion detection technologies.

Fortune also says that the industry needs to redirect some of its thinking against different types of attacks. While often the focus is on physical attacks, a recent exercise conducted by EPRI using Department of Defense techniques to assess likely targets of the electric system revealed that eight out of the top 10 likely targets are subject to cyber attack.

But Fortune is by no means an alarmist. "The electric system is extremely robust, in a way. It handles hurricanes and natural disasters, so I have a large and abiding confidence that what gets thrown at us, we'll be able to develop work-arounds, because that's our history. I don't think any other industry has a 99.99 reliability factor ... so that fact that I tell you that we're vulnerable today-and I mean that-doesn't mean that if we got attacked, we couldn't quickly get back into service." Fortune says he is confident that industry will develop the technologies necessary to protect the system.

Code of Silence

In the wake of Sept. 11, even discussing cyber security issues can be difficult. GTI's Rush says, "[o]ne of the scary things about this is, in some cases people won't let you talk about security. The thing about it is, the people who have the technical capability to launch these attacks, know all about them, they know more about it than I do. But, I can't tell anybody who's a victim." Indeed, during interviews for this story, NERC officials asked that problems with SCADA systems not be discussed-although such problems are mentioned prominently in some of NERC's own documents that are available on the Web. NIPC declined to be interviewed, but its Web site clearly states that SCADA systems and OASIS communications networks present vulnerabilities.

A Google search for "SCADA systems vulnerability" conducted by Fortnightly in early February found 684 hits, many highly relevant and more technically detailed than any of the information presented here. Some might suggest the solution is to remove information from the Web, and indeed the Federal Energy Regulatory Commission (FERC) and NIPC have suggested that the energy industry closely examine what types of materials are available on their Web sites. FERC last October announced that it would limit access to public documents, mostly those detailing specifications of energy facilities licensed or certificated.

As Leffler points out, there's a lot of information out there. The question is, is there too much? In the last three months, he says, NERC has been looking at what data is available on Internet sites, and in other places. Information, he says, "should be made available to those who need it, and not at all to those who don't." Yet it is difficult to draw those lines, and the criteria being used to make those decisions is still evolving.

Leffler says, "[w]e're still learning that. But, for example, does the public need to learn the output of a generating station? Does the public need to know the transmission flows on key critical transmission lines? I don't think so." Both operators and markets also needs access to data so that they can make good decisions, he says, and such information is made available to them. But Leffler balks at items like industry maps detailing transmission lines, generating stations, and other key critical facilities locations being made available to the public. In addition, he says that items like future plans, bottlenecks or congestion areas in the transmission grid should not be available. "We don't want to highlight frailties, for the obvious reasons. We live in a different world now, and we've got to guard our information appropriately. "

FERC is indeed trying to help set such policy, but it's no easy task to balance the need to protect the critical infrastructure yet maintain the hallmark of an open society, easy access to a wide variety of information. So far, according to a senior FERC official, the agency has not restricted access to pending plans filed at the agency. One of the main points of contention about restricting such information comes down to environmentalists' and citizens' concerns about future plants and transmission line placement, along with concerns about the environmental impact of existing plants. Generating plants are essentially large chemistry sets, with supplies of chemicals that in the wrong hands could be misused. Easily available lists of specific types and quantities of chemicals could prove a tempting target to would-be terrorists. Yet at the same time, those who live near such plants want to know what could be affecting their environment. As Ellen Vancko, spokeswoman for NERC says, "The question is the degree of specificity. We're not saying that the public doesn't have a right to know what's going on in their community."

A Business Decision

As Rush says, "I think one of the things that really, really needs to be understood is that security is a business decision just like any other." What that means, he says, is that utilities need to make truly informed security decisions. In order to do that, you need quantified risk assessments. Figure 2 provides an overview of an information security program model that can be used to guide the risk management process.

Vancko agrees. "You have to look at cyber security as every business does, when it has a vested interest in protecting its assets, its shareholders, property, whether intellectual or physical, from disruption." The industry is going to do that, she predicts. The industry proved they will take the necessary measures to protect infrastructure during Y2K, she says. "I think there will be peer pressure at minimum, and if the government determines that there's weaknesses, then they may decide they need to do something." She rejects the notion that deregulation-and increased competition-means that security will be relegated to the back burner.

Alliant's Davis says that it is up to industry to protect itself. In a speech last March, he issued a warning that seems hauntingly prescient now: "The government cannot save us from cyber hackers. Our federal laws can't protect us from such attacks, because cyber terrorism can start from any country in the world. It has no boundaries. It is a global issue, and no one entity in the U.S., or in the world, can really protect us. We must take aggressive action to save ourselves."

Articles found on this page are available to Internet subscribers only. For more information about obtaining a username and password, please call our Customer Service Department at 1-800-368-5001.