Vulnerability of SCADA Systems Underscore Urgent Need to Secure Utility Supply Chains

The recent reports describing the profound risk of catastrophic power outages, the vulnerability of computer hardware and communications equipment to tampering during design and manufacturing, and the penetration of utility SCADA systems by Russian hackers highlight the criticality of supply chain integrity.

The combination of these revelations exposes dangerous gaps in regulatory oversight created by the fragmentation of authority between state and federal governments and discordance among federal agencies charged with protecting the nation's critical infrastructure from nation-state and other attacks. Technical and policy solutions exist but will require appropriate resources, political will, and regulatory courage to achieve.

In December 2018 the President's National Infrastructure Advisory Council, in the report "Surviving a Catastrophic Power Outage," warned that the power grid is a prime target and studied the consequences of a widespread power outage of two to six months.

To put the consequences of a long-term outage in perspective, ten years ago the Electromagnetic Pulse Commission predicted that a one-year power outage could result in the deaths of hundreds of millions of Americans; estimates range from two-thirds to ninety percent of the population.

While the consequences of a failure to prevent a prolonged and widespread power outage are clear, many of today's emerging threats are less obvious than a dramatic, but low probability, nuclear attack by a terrorist organization or rogue state. Today's threats can be delivered electronically from half a world away or surreptitiously baked into OEM hardware manufactured by Chinese foundries and shadowy subcontractors.

Brien Sheahan: No consistent comprehensive cybersecurity standards exist for distribution level utilities.

Last Fall, Bloomberg Businessweek reported that a unit of the People's Liberation Army compromised the Chinese supply chain of San Jose-based Super Micro Computer Incorporated (dba Supermicro) one of the world's largest suppliers of server motherboards, by inserting a covert microchip in products used in the network infrastructures of almost thirty U.S. companies, including Amazon, Apple, and an unnamed telecommunications company, for purpose of espionage.

While the alleged victims deny being compromised, Bloomberg has stood by its reporting. In testimony before the Senate Homeland Security Committee in October, Department of Homeland Security (DHS) Secretary Kirstjen Nielsen stated, "this is a particularly pernicious threat . . . because it is very difficult for the average citizen, company, or government entity to understand every component that was put into a part, piece of equipment, or network that they have purchased." While Secretary Nielsen said DHS had no evidence supporting the specific allegations, she expressed concern for the "very real and emerging threat."

The vulnerability of foreign, especially Chinese, made equipment is vividly illustrated by new revelations about telecommunications maker Huawei. Australian intelligence sources allege the company provided the Chinese government with backdoor access to foreign networks for spying.

According to Mike Burgess, Australia's cyber-defense chief, if high-risk vendor equipment is used anywhere in Australia's evolving 5G network, the future communications system underpinning Australia's water supply and electricity grid and health systems, even self-driving cars, could not be protected.

Dominic Saebeler: With the astronomical number of components in modern integrated circuits and products, it is possible to add covert components that are maddeningly difficult to detect.

Burgess warns that, "the distinction between core and edge collapses in 5G networks. That means that a potential threat anywhere in the network will be a threat to the whole network." The Chinese government's official response involved a not so veiled threat to Australian business interests.

More recently however, the arrest in Poland of a senior Huawei executive and a former Polish intelligence officer on charges of espionage underscore the severity and authenticity of the threat. Poland has called for NATO and the European Union to work jointly to restrict the sale of Huawei equipment.

The Mobile communications industry body GSMA has proposed that its members discuss the possibility of Huawei being excluded from key markets, and the EU is weighing a de facto ban on Huawei equipment due to security concerns.

A January 2019 report by Wall Street Journal writers Rebecca Smith (who was first to report the Pacific Gas and Electric Company Metcalf substation attack in 2014), and Rob Barry details how Russian operatives targeted the SCADA systems of utilities in half of U.S. states, and utilities in Canada and the U.K. by penetrating insecure systems of third-party service providers in utility supply chains.

Wei Chen Lin: Regulators and utilities are positioned to minimize these risks by promoting the use of domestically sourced components and supply chains for critical applications.

These attacks used well-known hacking tools and tactics including watering hole and spearfishing attacks. According to Robert Lee, CEO of cybersecurity firm Dragos, "Russian actors [have] gained the ability to disrupt or completely shut down power plants and other critical infrastructure in the U.S."

More than emerging, these threats are the new reality, and must be addressed by state and federal policymakers and industry with urgency and coordination. There is a long history of reported supply chain attacks by nation-states that illustrate the vulnerability and demonstrate the consequences of inaction.

Security researcher Brian Krebs recounts a decade old report from a well-placed source who informed him that a Chinese tech company actively sold a custom hardware component for internet connected printers that forwarded every document or image processed by the printer to a server allegedly controlled by hackers aligned with the Chinese government.

In 2007, speculation mounted that the successful Israeli bombardment of a suspected nuclear installation in Syria was aided by the failure of Syrian radar systems manufactured with digital backdoors to disrupts their function.

In 2010, the FBI announced results of Operation Network Raider, which resulted in more than seven hundred seizures of counterfeit networking products manufactured in China. The products included routers, switches, and network interfaces bearing Cisco brand names and model numbers. Some of the counterfeit hardware were reportedly shipped directly to the Marine Corps, Air Force, FAA, FBI, defense contractors, universities, and financial institutions.

There are countless other examples, and the utility industry is not exempt from the threat.

In December 2017, we co-authored an article in Public Utilities Fortnightly highlighting the potential for this type of risk. The article encourages regulators to ensure the security of global supply chains to reduce threats to the utility industry. While awareness has increased and some progress has been made, these recent reports show that the risk is a reality.

The Need for Standards

The Federal Energy Regulatory Commission recently approved NERC CIP-013-1 standards addressing cybersecurity supply chain risk management. These standards are an important step forward but are generic and may not result in practices that fully address the type of risks identified in these new revelations.

In addition, the new standards only apply to the bulk electric system. No consistent comprehensive cybersecurity standards exist for distribution level utilities. The Commission noted that a significant cybersecurity risk associated with the supply chain persists because the proposed standards exclude firewalls, authentication servers, security event monitoring systems, intrusion detection systems, and alerting systems.

Some believe these concerns are mitigated by the separation of operational networks that control the industrial processes involved with the delivery of utility services from business networks involved with the typical corporate business functions. However, these traditional airgaps and electronic security perimeters have also become illusory and outdated concepts, as connectivity has increased in the operational networks.

In fact, technicians regularly use jump boxes to access operational networks remotely, bridging the intangible castle walls. Vendors also regularly cross these divides: in January 2019, Duke Energy, one of the largest utilities in the nation with operations in seven states, was fined ten million dollars for a hundred and twenty-seven violations of NERC CIP Reliability Standards.

The filing notes the violations posed a serious risk to the security and reliability including repeated failures to implement physical and cybersecurity protections and allowing vendors without proper clearance to gain unescorted access to sensitive locations such as substations and server rooms.

The Regulators' Role

The authors did not find published cases before state regulators applying traditional prudence analysis to specific cybersecurity spending or related infrastructure investment. However, these questions are well within state regulatory oversight of reliability and cost.

Considering the clear and present threat posed by these events, regulators will, necessarily, begin to view utility business practices - including respons es to cybersecurity threats - according to traditional benchmarks of good judgement and best practices.

It is well established that a public utility should not be allowed to recover negligent or wasteful costs. For example, the Illinois Supreme Court has noted that, "an affirmative showing of the reasonableness of a utility's construction-related costs is necessary if a sense of confidence in the ratemaking process is to be instilled in those consumers."

And that, "the Commission may order a supplemental audit, take affirmative evidence concerning the reasonableness of the costs, or deny the costs altogether if they are not shown to be reasonable." "Only when the Commission is satisfied by . . . affirmative evidence that the costs incurred by a utility are reasonable . . . may those costs be included."

FERC has held that, even where the utility does not have the statutory burden to show reasonableness of an expenditure, as is the case in Illinois, when "some other participant in the proceeding creates a serious doubt as to the prudence of an expenditure, the utility has the burden of dispelling the doubt and proving the prudence of the questioned expenditure."

Prudence analysis is based on the standard of care a reasonable person would be expected to exercise under the same circumstances at the time the decisions were made, similar to negligence in civil torts, and hindsight review is not available.

Differences of opinion will occur over what is considered a prudent action. However, given the serious doubt placed on supply chain security of foreign procured equipment and software as in the case of Huawei, it is unlikely a utility would be able to overcome the burden of proof to convince a PUC to allow costs associated with imprudent use of foreign components without inclusion of evidence or supporting details that specifically explain the remediation of these potential malicious software and component risks. In fact, not only are utilities responsible for their own prudency, but also the prudency of its contractors and vendors.¹

Recent revelations involving ZTE (another Chinese manufacturer accused of creating potential risks to national security of other nations), Huawei, and Supermicro, along with consistent history of cyber espionage and sabotage between nation-states, arguably counsel in favor of an outright ban, where practicable, of foreign sourced components in any critical infrastructure or the inclusion of a very specific framework through which software and hardware components are vetted to ensure they do not include unauthorized modifications. An outright ban could also minimize inconsistencies and uncertainties inherent in ad hoc cyber procurement prudency analysis undertaken by PUCs.

While considering the potential of excluding foreign manufacturers from utility supply chains, other factors may be more difficult to control. These include ensuring that domestic suppliers are not infiltrated by malicious actors, as well as managing what will most likely be significantly increased cost of domestically sourced software and hardware components.

A shift toward requiring domestically manufactured components and equipment to be used in utility supply chains would require a significant transition period and an improved framework for the detection of unauthorized modifications in the design, manufacturing, and distribution of software and hardware components.

Continued reliance, merely due to cost considerations, on foreign supply chains based in countries with whom the United States has a tenuous relationship does not address the ongoing identified risk. In the case of China, organizations and individuals are legally required to "assist and cooperate with state intelligence work." Securing the technology supply chain is incredibly complex and expensive, especially "when products may have been intentionally compromised during some part of the manufacturing process."

Each individual modern device is composed of hundreds of chips, but it only takes one modified chip to subvert the security of the entire product. In 2008 researchers from the University of Illinois demonstrated the ability to alter microprocessors to create a stealth system that allows them to log into a network or computer.

With the astronomical number of components in modern integrated circuits and products, it is possible to add covert components that are maddeningly difficult to detect, and it is impractical and uneconomic to test every component.

Equally challenging is the belief by many commentators and security researchers that, even if the U.S. government and Silicon Valley somehow rallied the funding and political will to require that all products sold to U.S. consumers be made only with components made in the United States, ordinary consumers would almost certainly balk at buying the more expensive products at the security premium when a comparable product with the features they want and care about is available much more cheaply.

While the challenges are significant, regulators and utilities are positioned to minimize these risks by promoting the use of domestically sourced components and supply chains for critical applications. Unlike general consumer products, equipment for critical utility operations is often specialized and represents only a subset of products produced.

Requiring these components to be U.S. made and distributed through trusted sources would not require nearly the same kind of funding and political will as would be needed to do the same for all U.S. consumer electronics.

In addition, where protection of critical infrastructure is concerned, a premium is worthwhile and to be expected in the face of the potentially existential threat to the nation posed by failures in utility operations. These costs must be evaluated in context, as compared to the potential economic impact of a cyber-attack against the power grid; a report issued by Lloyd's of London in 2015 estimated that a major attack could cost between two hundred forty-three billion and one trillion dollars.

Looking Ahead

Regulators are entrusted with ensuring the public health, safety, and welfare. State and federal regulators, national security agencies, and utilities must increase the attention to these challenges and adopt aggressive risk management strategies that include establishment of trusted distribution paths, vendor security validations, and ensuring supply chain integrity.

In addition, performance of regular audits, implementation of change management and threat awareness programs, and securing software development and design while having products independently tested is critical. These strategies must be applied to the software, hardware, services, and personnel of each vendor throughout the supply chain.

Programmatic as well as technical controls will be required as well as strategic navigation of the introduction and support of increased costs and marketplace transition timelines toward a modified or entirely new supply model. It is a herculean task. Though, ensuring the safe, secure, resilient, and efficient delivery of utility services is also an essential task.

Utilities and regulators must not only consider the actions and prudency of the utilities themselves, but also the actions and prudency of vendors throughout the supply chain. The sooner a collaborative and comprehensive set of standards are developed for distribution utilities the closer the industry will be to reducing the potential impact of third-party originated (or facilitated) intrusion into the operations of U.S. utilities.  

Endnote:

1. In Re Pennsylvania Power Company, Public Utility Commission 1987. See also Illinois versus ICC, April 29, 1986.

Lead image © Can Stock Photo / fgnopporn