True story: Early one crisp spring morning, a damaged electric distribution line arced onto a natural gas pipeline that it was directly touching. The pipe began leaking and natural gas migrated down the trench, where it met a French drain leading to the cellar of a brand-new home. As the family slept above, the cellar filled with gas, reaching the correct mixture for combustion. When the house’s sump-pump switched on, the house exploded, throwing the two children out the windows and onto the lawn while the mother and father burned.
Such a tragic event could have been avoided by ensuring a minimal clearance between the electric line and the natural gas pipe.
The natural gas company began a forensic investigation of the incident. The house that exploded was one of 700 gas and electric installations that were already finished in a new subdivision of 3,000 homes. Initially the company dug up 30 installations, but from these investigators knew enough to dig up all the installations in the subdivision.
The cost of the incident was enormous in terms of the tragedy to human life, as well as in operating and construction costs, the company’s reputation, legal liabilities, regulatory oversight and inter-company coordination in new construction. Since the impacts can be vast in scope for any company, management’s first job is to know the risk by measuring it, and second, if needed, to fix it.
The process of understanding and fixing risk exposures traditionally requires asking a series of questions. Can such events reasonably be prevented in the first place? If so, how? And can it be made an on-going permanent process? What are the initial costs and subsequent benefits—in terms of cost savings, reputation, market competitiveness, regulatory oversight at state and federal levels, stockholder returns, workforce design, training programs, management longevity and tied-in compensations systems?
An entirely new and better approach allows companies actually to measure this kind of risk—that is, to measure the degrees of compliance regarding actual field practices versus written standards and procedures. This approach to measuring risk and compliance is drawn from five engagements conducted for natural gas distribution systems located in as many states, each with its own regulatory commission. The methodology was created and implemented for these companies and presented to each of the five state regulatory commissions as an ongoing system with which to continuously measure compliance going forward.
Compliance is particularly important in the gas industry because of potential risk to the public and customers, utility employees and property. In the late 1960s, following a series of explosions and fires involving natural gas and petroleum product pipelines, the National Pipeline Safety Act (DOT-192) was promulgated, mandating detailed and comprehensive standards and procedures for pipeline construction, operation, maintenance, inspections, incident documentation and emergency response. State regulators were charged with drawing up procedures mirroring the federal act, and incorporating these in state regulatory law. Gas distribution utilities consequently have developed procedures responsive to those of their respective state commissions.
A recurring issue throughout the industry has been, and continues to be, how best to assess compliance with regulatory requirements and, in the event of reportable incidents, how best to show compliance in order to reduce liabilities and penalties.
Since the inception of the gas industry with manufactured gas before the turn of the last century, through the widespread growth and use of natural gas after World War II, to the enactment of the Pipeline Safety Act and up to today, gas-system managers have uniformly had as key objectives the provision of safe, reliable service at reasonable cost to their customers. Safety, as the paramount concern, continues to be monitored as always principally through inspections or field audits of system conditions, construction in progress, and ongoing operations.
The process of inspecting gas systems hasn’t changed fundamentally since the industry’s beginning. Inspectors and field auditors are trained in the system’s written procedures, provided with copies of those procedures, and dispatched to a field site to witness and record compliance or non-compliance for whatever activity is happening. Sometimes inspectors are given sections or regions to cover. Sometimes they focus on a particular aspect of construction or operations. Sometimes inspectors follow specific contractors and company crews. Some companies tout, “Every job site is inspected.”
The technology, however, has advanced through the use of hand-held computers to log pass-fail results on project work steps for transmittal to central data bases, which devices also provide on-site call-up of applicable procedures, past contractor or company crew performance on like installations, acceptable or approved variations on standards, and other useful information. Other examples of technology gain are so-called “smart pigs” propelled through cast iron or steel pipes to produce videos or photos to examine conditions, new systems to radiograph welds in search of flaws, and much improved leak detection or sniffing equipment.
Technology advances notwithstanding, the information gathered in traditional approaches to inspection is at best only descriptive of what part of installations occur most often, but not necessarily what is most important in the installation for measuring risk, nor is it inferential about the system-wide or regional compliance—or, most important, protection of people and property. Information collected, however more quickly, says nothing at all with any precision about risk or compliance. It provides no statistically valid measurement of meaningful risk factors, and thus no basis for demonstrating levels of confidence for the entire system.
This kind of inference is critical for executive management’s oversight responsibility and risk management roles.
The objectives in designing and implementing an improved compliance-measurement system for the first endeavor of this kind were driven by most serious reportable incidents, and these have remained the objectives for subsequent client engagements.
These objectives are:
• Ensure, prospectively and with high levels of confidence, that actual construction performance from a system-wide point of view—for both company employees and outside contractors—is in compliance at all levels of federal, state, local and company procedures and practices.
• Gain and document information and insight into the construction performance of the several electric utilities and other utility systems with which the company together serves customers.
• Provide findings, conclusions and recommendations supporting opportunities for improvement with regard to the company’s construction management process, and as well for intercompany practices.
The newly designed risk-management and compliance program initially focused on four principal measures deemed to be most critical in avoiding failures and reportable incidents. However, in all, field auditors collected pass-fail information on 103 compliance factors at each inspection site. The four key and most critical measures are:
• Clearance between gas pipes and electric lines;
• Depth of cover over gas pipes;
• Distance of warning tape from gas pipes; and
• Continuity of locater wires.
Simply put, these are the elements in new-business residential construction that most initially believe, if non-compliant potentially can injure people and damage property.
But what about the other 103 compliance factors considered at each inspection? How do these affect risk? Must we keep up with the pass-fail performance on all 103 compliance factors? Are some more important than others; or is it just the four critical factors? Can we combine and use them all as one critical determinate of risk and installation compliance? Is it possible to measure separately the performance of crews and contractors, in certain locations, subdivisions, territories, regulatory jurisdiction or by type of customer served?
By using the new method, the information contained in all of the 103 compliance factors is brought together to capture the meaning in terms of risk management, providing a much more useful performance-management tool.
In addition to scoring each compliance factor on a pass-fail basis, it describes the risk of the entire installation by finding the individual contribution of each compliance factor to total risk. That is, instead of treating each of the 103 compliance factors in a single installation as if they had the same value, or equal weight, the new method determines the importance of each of the 103 compliance factors according to their contribution to the risk of the entire installation. If the inspection shows that an individual compliance factor fails, then rather than treating it as if it had the same value as any other factor that failed, it is treated according to its individual importance and contribution to total risk of the installation. The result is one measure based on weighted criteria, which provide the basis for a single risk-adjusted compliance score for each installation inspected (see Figure 1).
The company ranks each of the items of information gathered at an inspection as to their contribution to the overall risk of the installation. If the total risk of the installation could be considered 100 percent, then what portion of the total does each one compliance factor represent? The authors found that about 10 percent of the items counted for about 50 percent of the entire risk. So it’s important to concentrate on those items having more importance. The alternative wastes time and resources, and misreads the risk.
Determining the importance of each of the 103 compliance factors involves polling a group of company people at several levels and from different parts of the organization, with varying degrees of specific knowledge about construction, on which elements the group believes to be most critical if failure occurs. This yields a set of weighted criteria that provides the basis for a single risk-adjusted compliance score for each installation inspected. This has been found to be a surprisingly accurate predictive tool.
In the past, field auditors dropped in on an installation and observed a small portion of whatever was underway at the time. The new program is different in several important ways. Namely:
• Auditors observe installations from beginning to end;
• Sites are assigned auditors through a carefully randomized process;
• Should compliance problems emerge, auditors are instructed to record the as-found situation, require the crew to correct the problems, and make certain the installation is in compliance;
• Field audits are unannounced, addressing the principal elements associated with risk to people and property introduced by non-compliant construction practices.
Besides inspecting installations from start to finish, and assigning auditors to installations at random, the question arises: “How many installations need to be inspected to produce compliance measures that are highly confident?” Cost must be balanced with the degree of risk the company is willing to undertake. The risk is measured by the compliance level. The cost is determined by the required number of samples, which in turn are determined by:
• Size of the installation/construction program: In this example the client had 25,000 new installations annually;
• Confidence Level: Desired level of statistical significance (usually a 95-percent confidence level);
• Required precision: How close should the sample predict the true (but unknown) risk and compliance value for the system; and
• Inherent variation: The true value of risk or compliance for the system. This isn’t known in advance, but requires an initial experienced-based estimate (see Figure 2).
For the client with 25,000 new installations to be inspected annually, focused inspections on about 505 sample installations produces compliance measurements for the entire system at the desired 95-percent confidence level (see Figure 3).
To get a 100-percent confidence level—and 100-percent precision on the actual level of risk and compliance— it would be necessary to inspect all installations, a sample size of 25,000. Choosing a 95-percent confidence level and a 2-percent precision level allows very high confidence that the estimate truly reflects the actual level of risk, but at a substantially diminished cost.
The significance of this statistically valid confidence level is that it’s widely accepted by industry regulators, federal and state courts, as well as the military, because it is recognized as being as good as reasonable people acting prudently can achieve—i.e., the common-law “prudent-man rule.” The Nuclear Regulatory Commission accepts this standard on such things as stainless-steel welds in nuclear power plants, and the armed forces accept this on deliverables such as bullets. Many courts have upheld the prudent-man standard in civil litigation. For comparison, moving to a 96-percent confidence level would require inspecting a much greater sample, increasing from 505 installations to several thousand. Expanding the sample in this way isn’t reasonable, because it doesn’t balance risk, precision and cost. A 95-percent standard allows the company to know the level of risk it faces, and allows it to act reasonably on this information if it’s found to be unacceptably high.
The benefits of this new way of measuring compliance are major and many, and begin with greater assurances to all industry stakeholders, partners and participants that the level of risk and compliance is understood with a high level of confidence, and that all is being done that can reasonably be expected to understand the risk the company faces.
Specifically, the new way permits continuous improvement in attaining very high levels of compliance. Measurement and comparisons of compliance by area or subdivision, together with compliance performance comparisons of company and contractor crews, provide a basis for best-practice gains. Additionally, it allows inspection resources to be optimized. In all five engagements, fewer, not more, full-time field inspectors were required after system implementation, auditor training, and IT integration were accomplished.
Moreover, this approach vastly improves compliance documentation for regulatory authorities, as evidenced by the testimonials of all five regulatory commissions involved. It reduces liability and risk associated with punitive jury awards in civil cases, and potentially can reduce insurance premiums as a result of reduced risks to the utility and its constituents.
Finally, this approach provides a greatly improved platform for managing overall enterprise risk, with a solid basis for certificates of executive management under Sarbanes-Oxley, and a prudent-man defense for stockholder lawsuits against directors and management.
This way of measuring compliance is, of course, not limited to field construction activities or gas, electric or water utilities. In fact, it has direct applicability to measuring compliance among any set of procedures, field or office, with actual practice in carrying them out.