Regulators Must Ensure Integrity of Global Supply Chains

In pursuit of economies of scale, convenience, and other efficiencies, utility operations across all sectors are becoming more connected. Some estimates put the total number of Internet-addressable devices within the next decade at a hundred billion, with a large percentage of the grid becoming networked with integrated communications. As a result, greater vigilance by utilities, their vendors, and regulators will be required to limit the risk of threats originating from all parts of the supply chain.

The rapid expansion of the Internet of Things, including utility infrastructure combined with the international procurement of components and software, creates new challenges for ensuring the cybersecurity of equipment and systems. Regulators need to understand the vulnerabilities associated with global supply chains for mission-critical information and operational technologies, and ensure that utilities enforce risk mitigation practices throughout their operations.

In response to the proliferation of the Internet of Things, any supply chain management plan must proactively address and measure cyber supply chain risk.

Supply chain security is "a program that focuses on the potential risks associated with an organization's suppliers of goods and services, many of which may have extensive access to resources and assets within the enterprise environment or to an organization's customer environments, some of which may be sensitive in nature."¹ In fact, supply chain security is becoming an essential part of any overall business risk management strategy.

The complexity of this challenge is significant. Utilities must manage an expanding attack surface resulting from increased insertion of remote access functionality into operational technology components. Vigilance by utilities and regulators is essential to understanding where additional risk exists from vendor components and how to mitigate those risks.

Risks to Utility Industry Far from Theoretical

Since at least 2011, attackers known as the Dragonfly group have been targeting industrial control systems equipment providers, including those supplying the energy sector.² Rather than directly attacking their targets in the energy sector, the Dragonfly group "found a soft underbelly by compromising their suppliers, which are invariably smaller, less protected companies," according to a Symantec report.³

Included in the Dra­gonfly group's wide variety of tactics, are ambitious and successful direct penetrations of utility operations systems and compromises of legitimate software packages available for download from the websites of industrial control systems equipment providers.⁴ While it is impossible to completely eliminate all risk from the supply chain, there are strategies, tools, and methodologies available to significantly reduce the effectiveness of tactics frequently used by attackers.

One such risk exists in the code running in programmable logic controllers which contain information valuable to attackers. PLCs are ruggedized and specialized computers that are essential in the control of modern industrial processes, including utility operations systems. PLCs can translate analog sensor signals into digital data and translate digital commands into analog control signals, leading to physical actions in other components.

The programming used to operate PLCs can include source code, typically in programming languages readable by humans, or compiled code, which is derived from source code and readable only by the machines.

Brien Sheahan: Remote access Trojan software is an increasingly common approach used to breach essential utility operations.

Source code contains a wealth of information of great value to attackers, such as detailed configurations, disabled options, and sensitive intellectual property. Access to source code makes it easier for an attacker to make unauthorized modifications to the code and perform malicious actions.

One strategy to mitigate this risk is requiring source code to be removed from PLCs where appropriate, and instead requiring that only compiled code be included. Although compiled code can be reverse-engineered to reveal information about source code, including only compiled code still creates an obstacle to malicious access to source code.

Another risk exists in firmware updates. One strategy to limiting this risk is checking for cryptographic signatures, prior to a utility installing downloaded firmware version upgrades, to confirm the authenticity of patches from the manufacturer. Cryptographic signatures use mathematical algorithms to prove that the signed data originated from the signer and has not been altered since it was signed.

This prevents upgrades and patches with malicious code inserted by an attacker from masquerading as legitimate upgrades and patches offered by the manufacturer. Verifying patches by testing them in a non-operational cybersecurity laboratory environment is another strategy.

While these types of strategies for reducing risk are available, they are not consistently deployed. Public utility commissions have an opportunity and obligation to actively discuss and promote approaches to limit evolving threats. Requiring consistent, disciplined risk mitigation practices from utility suppliers will strengthen utility security and reliability.

Current State of Supply Chain Security Across Industries

In his article "Securing the IT Supply Chain,"⁵ Ziv Chang, director of cyber safety solution at Trend Micro, describes how "third-party contractors and suppliers have been used to compromise larger organizations." Remote access Trojan software is an increasingly common approach used to breach essential utility operations.⁶ As suggested by its namesake from Greek mythology, a Trojan is a computer program that appears benign but delivers other malicious content like viruses and worms.⁷

For example, "a 2011 hack on Lockheed Martin was blamed in part on information stolen from a hack on their vendor, RSA, that compromised SecureID tokens."⁸ In a more widely-reported case, the 2013 data breach at Target⁹ was traced to an HVAC vendor's remote connection used for electronic billing, contract submission, and project management.

Even the federal government has not been immune to serious cyber-attack. The June 2015 data breach disclosure¹⁰ by the U.S. Office of Personnel Management revealed that one of a series of data breaches began in May 2014. A hacker had stolen credentials from an agency contractor and exfiltrated proprietary data over the course of a year before being detected.

Dominic Saebeler: Though the utility industry is heavily regulated, the suppliers that utilities rely on are not similarly regulated. But this may be changing.

Recognizing the criticality of protecting the supply chain, and its role in maintaining secure government and private sector operations, the federal government is developing a multi-pronged approach for global supply chain risk management called "The Comprehensive National Cybersecurity Initiative."¹¹

According to the Initiative, acquisition decisions should incorporate consistent application of risk assessments that are a critical part of a comprehensive cyber supply chain policy. Policy creation and subsequent implementation should focus on awareness, enhancing skills, use of strategic tools, and regular testing of suppliers' adherence to security standards. Adoption of best practices and standards will help ensure that these types of controls are implemented across industries.

Existing Regulations, Standards, Guides, Best Practices Across Industries

Best practices related to supply chain risk management are being established in companies across the country. Some examples include: requiring source code to be provided for all purchased software, pre-qualifying components and sub-components; alerting vendors that failure to provide products that match specifications or providing products that turn out to be counterfeit will result in disqualification from current and further business; clearly stating security requirements in requests for proposals; and requiring software to have a security handshake.

While best practices are a step in the right direction, utilities, like other large, complex businesses, must continuously address and reassess the effectiveness of acquisition of equipment and materials through a supply chain strategy and assessment.

Meagan Pagels: Compared with other industries, utilities are uniquely critical, as they provide an enabling function across all other sectors.

Organizations like the EastWest Institute and The Open Group are also assisting with the creation of tools for supply chain management. The EastWest Institute's "Cybersecurity Guide for Technology Buyers,"¹² for example, focuses on three primary areas, including enterprise security governance, product and service lifecycle, and assurance.

See the box entitled Tools for Supply Chain Management.

The Guide provides a set of twenty-five questions that form the basis of a proactive vendor interaction strategy that increases security-based requirements. It puts vendors on notice that the buyer has higher expectations regarding suppliers' compliance with industry best practices for supply chain security.

In addition to the work of the EastWest Institute, other collaborative efforts are underway. One such example is the Open Group, a consortium composed of IT buyers, sellers, and academic researchers, which has developed a set of best practices¹³ for suppliers to follow "to mitigate the risk of tainted and counterfeit components."

Wei Chen Lin: PUCs should ask utilities and vendors tough questions in a meaningful dialogue to improve awareness and accountability.

The Open Trusted Technology Provider™ Standard¹⁴ was approved by the International Organization for Standardization and the International Electrotechnical Commission as ISO/IEC 20243:2015.¹⁵

Under the O-TTPS Accreditation Program, suppliers are evaluated by third-party assessors for conformance to the standard. The Open Group maintains a list¹⁶ of accredited suppliers.

Drawing in part from the O-TTPS, the National Institute of Standards and Technology addresses cyber supply chain best practices in Special Publication 800-161¹⁷ and summarizes the topic in the conference materials.¹⁸ Risks identified by NIST include compromised software, ineffective security practices of smaller suppliers, counterfeit hardware, information leakage, and many others.

SP 800-161 integrates controls from various publications in the SP 800-53 family to mitigate these risks, ranging from access control, identification and authentication of components, to the concept of provenance or change management.

Standards development is helpful in focusing efforts on a consistent approach to an intentional risk mitigation strategy to support an entire supply chain. The Electricity Subsector Cybersecurity Capability Maturity Model, ES-C2M2,¹⁹ is a voluntary model authored by the Department of Energy.

Components of the Model pertaining to supply chain security focus on identifying external dependencies and managing risks introduced by the dependencies. Though the utility industry is heavily regulated, the suppliers that utilities rely on are not similarly regulated. But this may be changing.

Current State of Supply Chain Security in Utility Industry

The utility industry, like finance, healthcare, and government, is confronting serious challenges to ensure the integrity of information and operations technology in an environment of increasing reliance on connected networks that provide remote access and management of physical and cyber assets.

PUCs should assess whether utilities are lagging behind, or leading other sectors, in implementing advanced management techniques that consider supply chain security. The utility industry recognizes the need for efficient and secure supply chains and appears to be making progress in this area.

A number of themes have emerged across the utility industry to mitigate cyber and supply chain risks, including active risk awareness, adoption of best practices, and implementation of consistent supply chain protection programs.

While these strategies have the potential to reduce unnecessary risks, some utilities are also drafting expanded requirements into their requests for proposals. They are demanding that suppliers develop their own best practice-based programs, and exploring methods for independent verification of their vendors' supply chain security.

The Federal Energy Regulatory Commission issued a notice of proposed rulemaking²⁰ in July 2015 that required the North American Electric Reliability Corporation to develop critical infrastructure protection rules for supply chain management.

FERC approved the NERC CIPv6²¹ in January 2016. Subsequently, in July 2016, FERC also issued an order²² directing NERC to develop standards to address "software integrity and authenticity; vendor remote access; information system planning; and vendor risk management and procurement controls" in the bulk power system.

In September 2017, NERC formally petitioned FERC²³ for approval of the supply chain risk management standards. The standards only apply to IT systems supporting or connected to high- and medium-impact bulk electric system cyber systems as defined by CIP-002-5.1a.

In the face of potential new regulations, and despite recognizing the importance of cyber supply chain security, some companies and suppliers do not view additional regulation as advancing cybersecurity.

This view is based on an assumption that regulations may result in criteria that are either too broad or too narrow, which could interfere with adoption of ever-evolving best practices. This view holds that additional regulations in the cyber supply chain may foster an environment where mere compliance with regulations is prioritized over actual increased security.

Compared with other industries, utilities are uniquely critical, as they provide an enabling function across all other sectors. Not only are utility assets a part of the critical infrastructure, they are an indispensable basis for the proper functioning of other critical infrastructure assets. PUCs must serve the public through strong oversight of utility cybersecurity, notwithstanding the inherent challenges of maximizing actual security while complying with standards.

PUCs can be catalysts to drive collaboration, knowledge sharing, and awareness of best practices and tools available for addressing supply chain security risks. PUCs can request reports from utilities that outline their progress in protecting operational systems against counterfeit or compromised components, as well as embedded software not properly vetted by suppliers.

However, it is important that utilities guard against becoming overly focused on compliance at the expense of actually improving security culture and protocols. PUCs must also balance their oversight role with the real possibility of creating new vulnerabilities. That is, PUCs may want to avoid becoming repositories of sensitive information that, if disclosed, could significantly compromise security.

PUCs are also strategically positioned to work with large investor-owned utilities to understand effective implementation, measure success through the absence of disruption, and promote sharing of best practices throughout the industry. Small investor-owned and municipal utilities, as well as independent power producers, may not have access to the same level of resources.

With assistance and collaborative direction from larger peers, smaller utilities and generators can also take the necessary steps to strengthen their supply chain security. PUCs should challenge and encourage the industry to carefully consider both the benefits and risks that accompany the rapid introduction of independent power producers, microgrids, and distributed energy resources.

Potential resiliency benefits exist through designs that include islanding of operations, black start and continued operations capabilities from DERs, should the broader network be compromised via cyber-attack. But the utility attack surface does increase with the introduction of additional vendors, devices, and entry points that may not be as stringently managed by smaller vendors, especially those supplying products that are remotely accessible by the vendor.

Questions for Utility Industry

Success in an organization's efforts to strengthen supply chain security requires appropriate culture, sufficient capabilities, and determination to follow through. PUCs should ask utilities and vendors tough questions in a meaningful dialogue to improve awareness and accountability.

See the box entitled Key Questions to Consider.²⁴

Ultimately, utilities and regulators must insist that cyber risk mitigation be part of the entire product development process of suppliers, from design to maintenance, delivery, and even disposal, and that cybersecurity becomes culturally integrated within their organizations, including vendors and suppliers.

Evolution of Standards and Incentives

There are two specific areas where PUCs can focus right now: embedded software and third-party audits. While many best practice models for cyber supply chain management are conceptually the same, they differ in small but significant ways to offer flexibility to the adopters to customize their strategies. However, the models consistently promote a focus on requiring embedded software that considers security risk.

It is critical that cybersecurity is addressed as an integrated attribute of operations and information technology solutions, rather than an add-on security feature. Hardware manufacturers can, and must, take steps to mitigate security vulnerabilities in product development.²⁵

Adding authentication capabilities to hardware devices and using write-protect switches on the device that prevent flashing the firmware can reduce the risk of the PLC accepting unauthorized command code.

When these risks are not properly addressed, vulnerabilities in vendor-embedded software form part of the chain of exploits that attackers use to ultimately cause the intended effect on a utility's operational environment, as security companies ESET and Dragos recently revealed²⁶ in the 2016 attack on Ukraine's power grid.²⁷ CrashOverride, a piece of malware central to the Ukraine attack, is the "second-ever known case of malicious code purpose-built to disrupt physical systems."²⁸

Rather than gaining control of an operator's workstation in a control center, as attackers did in the 2015 attack on Ukraine's power grid,²⁹ this new approach in CrashOveride contains a "logic bomb" that targets a bug in vendor-embedded software. The Windows-based platform leverages an existing vulnerability,³⁰ disclosed in 2015,³¹ in Siemens SIPROTEC devices widely deployed in power grids around the world.

In normal operations, such a device monitors grid components, sends information back to operators, and automatically opens circuit breakers if dangerous power levels are detected.³² "Once this vulnerability is successfully exploited, the target device stops responding to any commands until it is rebooted manually," according to a report by ESET, which effectively renders the device inoperable. Exploitation of this vulnerability is difficult to prevent, because the attack uses the communications protocols as they were designed to be used, without deviating from expected standards.

Vendors are in the best position to mitigate these risks when initially developing the code that runs these devices and while providing ongoing support for these devices. The ability to remotely analyze and detect threats and provide local overrides as part of a comprehensive threat analysis and detection approach is indispensable in secure software solutions.

The software development cycle requires basic hygiene and post-sale support that includes upgrades and software fixes. Secure methodologies must also be considered and incorporated early in the development process. Utilities need to ensure that vendors continually improve their practices while considering various approaches to improving the security of software solutions.

Information and communication technology providers outsource many of their internal components and manufacturing processes, including assembly of various components that arrive from multiple sub-component providers, the security of which is often assumed. Identification of distribution practices for components and devices, from design, to manufacture, to use in utility operations, requires a comprehensive risk management strategy. 

Although the supply chain for utilities is not directly overseen by state or federal regulators, the industry must adopt safeguards to ensure that supply chain cyber security, utility safety and reliability remain clearly within the purview of state public utility commissions. Third-party audits are a way to efficiently collect information and share progress assessments with utilities, their peers, and their regulators, without necessarily dictating rigid rules and compliance mechanisms.

Reasonable concerns surrounding third-party audits include the potential for inconsistent standards and interpretations, the possibility that sensitive information could become public, the technical bias of auditors, and the regulatory tendency to become overly-focused on process rather than actual security improvements, and ultimately, the cost to ratepayers.

While these are all valid concerns, an urgent need exists for independent oversight and evaluation to assure regulators and the public that the industry is appropriately prioritizing, investing in, and taking actions to secure critical infrastructure in a prudent and cost-effective manner. As the recently-revealed breach of credit reporting agency Equifax demonstrates, trusting that even large, well-funded corporations are taking all necessary and appropriate precautions may no longer be a prudent course of action.

The NERC CIP standards promulgated by NERC are a step toward incorporating a cyber supply chain section that will soon become part of the compliance program. Consideration should be given to the most appropriate method of incorporating use of third-party audits into a holistic assessment of cyber defense capabilities.

Conclusion

The challenge is considerable. The utility industry is actively pursuing solutions to improve resiliency, increase efficiency, and, at the same time, balance operational efficiencies while prioritizing risk mitigation best practices. But utility supply chains are large and complex. Tectonic geopolitical and technological changes are increasing the nation's exposure to cyber- attack as information technology and utility infrastructure advances, and more autonomous and connected assets are integrated into traditional system operations.

Embedded software and vulnerable components continue to be a risk that must be addressed early and throughout the process of integration of new technologies as they are increasingly relied upon for utility operations. Focusing on increased demand for integration of cyber defense best practices, and consistent approaches throughout the supply chain, will greatly reduce the risk of external interference with utility operations.

Public utility commissions have an opportunity to drive awareness throughout the industry and challenge utilities to demand their suppliers step up and keep up in reducing risk from the supply chain.

Endnotes:

1. Dave Shackleford, Combating Cyber Risk in the Supply Chain, SANS Institute, September 2015.

2. Dragonfly: Western Energy Companies Under Sabotage Threat, Symantec, June 30, 2014.

3. Ibid.

4. Ibid.

5. Ziv Chang, Securing the IT Supply Chain, TrendMicro, March 31, 2015.

6. Lucian Constantin, New Havex Malware Variants Target Industrial Control System and SCADA Users, PCWorld, June 24, 2014.

7. Murugiah Souppaya et al., Special Publication 800-83 Revision 1, Guide to Malware Incident Prevention and Handling for Desktops and Laptops, NIST, July 2013.

8. Ziv Chang, Ibid.

9. Target Hackers Broke in Via HVAC Company, Krebs on Security, February 5, 2014.

10. Eduard Kovacs, Leadership, Not Technology, Blamed for Huge OPM Breach, Security Week, September 8, 2016.

11. The Comprehensive National Cybersecurity Initiative, National Security Agency.

12. Purchasing Secure ICT Products and Services: A Buyer's Guide, Policy Report, EastWest Institute, September 13, 2016.

13. The Open Trusted Technology Provider Standard (O-TTPS) - Approved as ISO/IEC 20243:2015 and the O-TTPS Certification Program, The Open Group, January 30, 2017.

14. O-TTPS Certification Program, The Open Group.

15. SO/IEC 20243:2015, International Organization for Standardization, September 2015.

16. O-TTPS Certification Register, The Open Group.

17. Jon Boyens et al., Supply Chain Risk Management Practices for Federal Information Systems and Organizations, NIST Special Publication 800-161, National Institute of Standards and Technology, U.S. Department of Commerce, August 2013.

18. Best Practices in Cyber Supply Chain Risk Management, Conference Materials, National Institute of Standards and Technology, U.S. Department of Commerce.

19. Electricity Subsector Cybersecurity Capability Maturity Model (ES-C2M2), Energy.gov.

20. Michael Brooks, FERC Seeks Supply Chain Protection Against Cyber Threats, RTO Insider, July 20, 2015.

21. Jim Erlin, Hello There, NERC CIPv6, The State of Security, February 1, 2016.

22. Order No. 822, Revised Critical Infrastructure Protection Reliability Standards, 154 FERC. Section 61,037, 2016, to be codified at 18 CFR Part 40.

23. ERC News: September 2017, 6, NERC, 2017.

24. ISO/IEC 20243:2015, International Organization for Standardization, September 2015.

25. Kim Zetter, Why Firmware is So Vulnerable to Hacking, and What Can Be Done About It, Wired, February 24, 2015.

26. Alert (TA17-163A) CrashOverride Malware, US-CERT, July 7, 2017.

27. Andy Greenberg, 'Crash Override': The Malware That Took Down a Power Grid, Wired, June 12, 2017.

28. Ibid.

29. Kim Zetter, Inside the Cunning, Unprecedented Hack of Ukraine's Power Grid, Wired, March 3, 2016.

30. Anton Cherepanov, WIN32/INDUSTROYER: A New Threat for Industrial Control Systems, 15, ESET, June 12, 2017.

31. Advisory (ICSA-15-202-01) Siemens SIPROTEC Denial-of-Service Vulnerability, ICS-CERT, July 21, 2015.

32. Andy Greenberg, 'Crash Override', Ibid.