DOCKETED
Docket Number: 01-AFC-07C
Project Title: 01-AFC-7C Russell City Energy Company
TN #: 242769
Document Title: CEC Staff Report - Russell City Energy Center May 2021 Incident Root Cause Gap Analysis
Description: ***This document supersedes TN 242757***
Filer: susan fleming
Organization: California Energy Commission
Submitter Role: Commission Staff
Submission Date: 4/25/2022 10:16:38 AM
Docketed Date: 4/25/2022
ATTACHMENT III
California Energy Commission
STAFF REPORT
Russell City Energy Center
May 2021 Incident: Root
Cause Gap Analysis
April 2022 | CEC-700-2022-004
California Energy Commission
Geoff Lesh
Brett Fooks
Elizabeth Huber
Timothy Smith
Paul Trygstad
Ivan Clark
Primary Author(s)
John Heiser
Project Manager
Elizabeth Huber
Office Manager
SAFETY AND RELABILITY OFFICE
Geoffrey Lesh
Office Manager
ENGINEERING OFFICE
Shawn Pittard
Deputy Director
SITING, TRANSMISSION, & ENVIRONMENTAL PROTECTION DIVISION Drew Bohan
Executive Director
DISCLAIMER
Staff members of the California Energy Commission (CEC) prepared this report. As such, it does not necessarily
represent the views of the CEC, its employees, or the State of California. The CEC, the State of California, its employees, contractors, and subcontractors make no warrant, express or implied, and assume no legal liability for the information in this report; nor does any party represent that the uses of this information will not infringe upon privately owned rights. This report has not been approved or disapproved by the CEC nor has the Commission passed upon the accuracy or adequacy of the information in this report.
ACKNOWLEDGEMENTS
Acknowledgements
California Energy Commission
Jared Babula
Tom Chesy
John Heiser
Caryn Holmes
Shahab Khoshmashrab
Kenneth Salyphone
Sandy Ito
Kirk Oliver
Nicolas Oliver
Hung Pham
James Qaqundah
Dian Vorters
California Public Utilities Commission
James Cheng
Nika Kjensli
Chris Lee
Stephen Lee
Leslie L. Palmer
Naveed Paydar
City of Hayward
Alex Ameri
Garrett Contreras
Kelly McAdoo
i
West Peak Energy
Paul Trygstad
Ivan Clark
Kenny Smith
ii
GLOSSARY
AIR CONTAMINENTS — Dust, fumes, mist, smoke, other particulate matter, vapor, gas, odorous substances, or any combination thereof.
BLACK START — The process of restoring an electric power station or part of an electric grid to operation without relying on the external electric transmission network to recover from a total or partial shutdown.
BUSHING — A metal lining for a round hole, especially one in which an axle revolves.
CENTRIFUGAL — Moving or tending to move away from a center.
COMBUSTION TURBINE (CT) — A turbine driven by expanding hot gases produced by burning fuel, such as natural gas.
DISTRIBUTED CONTROL SYSTEM — A computerized control system for a process or plant usually with many control loops, in which autonomous controllers are distributed throughout the system, but there is no central operator supervisory control.
GALLING — A form of wear caused by adhesion between sliding surfaces.
GENERATOR — A dynamo or similar machine for converting mechanical energy into electricity.
HEAT RECOVERY STEAM GENERATOR (HRSG) — An energy recovery heat exchanger that recovers heat from a hot gas stream, such as combustion turbine or other waste gas stream.
It produces steam that can be used to drive a steam turbine.
ISOLATION VALVE — Stops the flow of process media to a given location, usually for maintenance or safety purposes.
LOCKOUT TAGOUT (LOTO) — A safety procedure used in industry to ensure that dangerous machines are properly shut off and not able to be started up again prior to the completion of maintenance or repair work.
MOTORING — A process when the steam turbine generator (STG) is connected to the electric grid but instead of outputting power to the grid, it is taking power from the electric grid.
PRIMARY CONTAINMENT — A tank, vessel, pipe, transport vessel or equipment intended to serve as the main container for or used for the transfer of a material.
REHEAT — The process by which additional energy is added to steam to increase the efficiency of the steam cycle.
SECONDARY CONTAINMENT — A control measure placed or built around a storage vessel to prevent its contents from corroding or polluting the adjacent environment. iii
STEAM ATTEMPERATOR — Located in the steam pipe work upstream of the steam turbine that allows very fine control of the final steam temperature by spraying precise amounts of water into the steam flow.
STEAM TURBINE (ST) — A machine that extracts thermal energy from pressurized steam and uses it to do mechanical work on a rotating output shaft.
STEAM TURBINE GENERATOR (STG) — A device that uses steam, produced from a heat recovery steam generator, to drive the blades of a turbine to produce mechanical energy that can then be used to produce electricity by causing rotation of the central shaft of a mechanically connected generator.
STOP/CHECK VALVE — A valve with override control to stop flow regardless of flow direction or pressure.
THRUST BEARING — A bearing designed to absorb thrusts parallel to the axis of rotation.
WATER INDUCTION — The process by which water finds itself entering the steam turbine.
WORM GEAR — A gear consisting of a shaft with a spiral thread that engages and drives a toothed wheel and changes the rotational movement by 90 degrees.
ZERO LIQUID DISCHARGE (ZLD) — A strategic wastewater management system that ensures that there will be no discharge of industrial wastewater into the environment. iv
ABSTRACT
On Friday, May 28, 2021, at 2:27 p.m., the California Energy Commission (CEC) was informed by Russell City Energy Center that it was in a forced outage because of a serious steam turbine generator incident at 11:47 p.m. on May 27, 2021. During Russell City Energy Center night shift’s normal shutdown procedures for taking the power plant offline, an incident in the steam turbine generator occurred causing an onsite explosion and fire.
The CEC’s Siting, Transmission and Environmental Protection Division maintains a comprehensive compliance monitoring and enforcement program to ensure that permitted thermal power plants are constructed, operated, and decommissioned in accordance with their conditions of certification and all applicable laws, ordinances, regulations, and standards. The CEC’s post-certification compliance monitoring and enforcement authority can be found in Public Resources Code sections 25532 to 25534.2 and Title 20, California Code of Regulations, sections 1751 to 1770, as well as in conditions of certification within facility licenses.
Under this authority, the Russell City Energy Center May 2021 Incident: Root Cause Gap Analysis was developed to summarize the CEC’s investigation into the factors that contributed to the May 27, 2021, incident and to determine what corrective actions would be required for the Russell City Energy Center to safely restart operations. In addition to determining the causal factors of the May 27, 2021, events, the CEC focused its investigation on worker safety, fire safety, hazardous materials, onsite physical security, and other conditions of certification as warranted.
v
Keywords: combined cycle, combustion turbine, control valve, explosion, fire, forced outage, incident, steam turbine generator, water induction, heat recovery steam generator, steam, lube oil, Hayward, Russell City Energy Center
Please use the following citation for this report:
California Energy Commission Staff, Russell City Energy Center May 2021 Incident: Root Cause Gap Analysis. California Energy Commission. Publication Number: CEC-700-2022-004. vi
TABLE OF CONTENTS
Page
Russell City Energy Center May 2021 Incident: Root Cause Gap Analysis.................................i Acknowledgements (Optional) ..............................................................................................i Glossary ............................................................................................................................ iii Abstract .............................................................................................................................v Table of Contents.............................................................................................................. vii List of Figures.................................................................................................................. viii List of Tables ................................................................................................................... viii Executive Summary.............................................................................................................1 CHAPTER 1: Russell City Energy Center................................................................................4 Introduction.................................................................................................4 Combined-Cycle Configuration ......................................................................4 Russell City Energy Center ............................................................................5 Background............................................................................................................................................................ 5 Incident ................................................................................................................................................................. 6 Investigation .......................................................................................................................................................... 6 CHAPTER 2: Incident Investigation ......................................................................................8 Introduction.................................................................................................8 Initial Site Inspection....................................................................................9 Follow-Up Site Inspection ...........................................................................11 Fire Department Response..........................................................................13 Compliance Inspection................................................................................14 Root Cause Analysis ...................................................................................14 CHAPTER 3: Technical Analysis..........................................................................................16 Incident Causation .....................................................................................16 Gap Audit Conclusions ................................................................................18 Equipment Maintenance and Monitoring Program.........................................18 Control Room Operator Interface and Training.............................................19 Inadequate Water Induction Protection .......................................................20 Public Safety..............................................................................................23 CHAPTER 4: Conclusions ...................................................................................................24 Corrective Actions ......................................................................................24 APPENDIX A: Joint State Agency Investigation Team Document Requests ..............................1 vii
LIST OF FIGURES
Page
Figure 1.1: Combined-Cycle Configuration ............................................................................5 Figure 1.2: Russell City Energy Center HRSG ........................................................................6 Figure 2.1: Steam Turbine Generator .................................................................................10 Figure 2.2: Debris Field .....................................................................................................12 Figure 2.3: RCEC Metal Piece.............................................................................................13 Figure 3.1: Russell City HRSG Reheater Schematic..............................................................17
LIST OF TABLES
Page
Table 2.1: RCEC Investigation Timeline ................................................................................8 Table A.1: Document Request Timeline ................................................................................1 viii
EXECUTIVE SUMMARY
On May 27, 2021, around 11:47 p.m., the Russell City Energy Center experienced a mechanical failure of the steam turbine generator that resulted in an explosion that threw dozens of metal pieces off the project site and resulted in an onsite fire requiring response by the Hayward, Alameda County, and Fremont Fire Departments. The steam turbine generator was severely damaged. In addition to the immediate public health and safety threat, this incident resulted in a loss of 600 megawatts (MW) of generating capacity from the grid.
Fortunately, there were no injuries and the lube oil mixed with fire suppression water was contained with no adverse impact to nearby waterways.
Structural Integrity Associates, an independent consultant retained by the project owner, Russell City Energy Center, LLC, a subsidiary of Calpine Corporation, performed a root cause analysis of the incident. The consultant’s root cause analysis was released to CEC staff on November 24, 2021. The root cause analysis concluded that there was only one cause of the incident: “The systems' inability to detect and drain excess water under pressure and at high temperature within the reheater system is the root cause of the STG drivetrain event at RCEC.”
To independently investigate the incident, both the California Public Utilities Commission (CPUC) and the California Energy Commission (CEC) inspection units established a Joint State Agency Investigation Team (JAIT). The investigation team conducted an examination and review of the power plant and associated documents, independently assessed the findings from the root cause analysis, and investigated gaps identified in that report.
The investigation team found that Structural Integrity Associates’ root cause analysis was limited in the scope of its analysis and restoration recommendations. Both the CPUC and the CEC concluded that further investigation to more broadly capture the causal factors to the incident was needed. The investigation team focused its site inspections not only on the power train involved in the incident, the steam turbine and electrical generator and associated heat recovery steam generator, but also examined facility operations, maintenance, and management practices that may have contributed to the causation of this incident.
The investigation team determined that there were three causal factors to the incident. The factors included: deficiencies in maintenance for some critical equipment, deficiencies in control room operator interface and training, and inadequate protection from water induction.
These factors led the investigation team to develop corrective actions to address the deficiencies that contributed to the incident, which are contained in Chapter 4 of this report.
Successful completion of the required corrective actions would directly address the causes of the May 27, 2021, incident and provide protective measures to further reduce the likelihood of future steam turbine overspeed due to water induction.
The activities of the JAIT were strengthened through collaboration between the agencies’ complementary approaches to execution of their respective regulatory authorities which added 1 both depth and breadth of technical knowledge and investigative expertise. This allowed the team to quickly uncover the causal factors that contributed to the incident and to provide the required corrective actions needed to reduce the possibility of a reoccurrence of this type of incident. Successful completion of the corrective actions will enable the Russell City Energy Center to return to operation safely.
2 This page intentionally left blank.
3 CHAPTER 1:
Russell City Energy Center
Introduction
The California Energy Commission (CEC) has jurisdiction and permitting authority for thermal power plants 50 megawatts (MW) and greater in California. This jurisdiction also includes infrastructure associated with thermal power plants, including electric transmission lines, natural gas lines, and water pipelines. The CEC’s permitting process ensures that proposed thermal power plants are designed, constructed, and operated in a manner that protects public health and safety, promotes the general welfare, and preserves environmental quality.
As a certified regulatory program, the licensing process is the functional equivalent of a California Environmental Quality Act review and includes coordination with local, state, and federal agencies to ensure that these agencies’ permit requirements are incorporated. There are 76 power plants operating under CEC licenses, totaling roughly 26,600 MW. Of these, 41 operate in gas-fired combined-cycle configuration.
Combined-Cycle Configuration
A combined-cycle power plant generates electrical power by the combination of a combustion turbine generator (CTG) burning fuel to operate its own electrical generator, and a steam turbine generator (STG) using the high temperature waste heat from the CTG exhaust to generate additional electrical power. Thermodynamically, the CTG operates on what is called a Brayton energy cycle, and the waste heat in its exhaust is captured by a heat recovery steam generator (HRSG) to make steam. The steam is then sent through pipes to the STG which operates on what is called a Rankine cycle. When both cycles operate simultaneously, the operation is called a combined-cycle configuration. Because the Rankine cycle makes electrical power from what would have been wasted heat in the CTG exhaust gas, the combined-cycle configuration is more efficient. It makes more electrical power from the same amount of fuel.
The overall efficiency of a combined-cycle power plant can be up to 60 percent more efficient than other fossil-fueled generating sources. The high efficiency of a combined-cycle configuration also reduces the overall air emissions per megawatt-hour.
In the combined-cycle process, the CTG compresses air and mixes it with fuel that is heated to a very high temperature. The hot air-fuel mixture moves through the CTG blades, making them spin. The spinning CTG drives a generator that converts a portion of the spinning energy into electricity. The HRSG captures waste heat exhaust from the CTG that would otherwise escape through the exhaust stack. The HRSG creates steam from the CTG exhaust heat and delivers it to the STG. The STG turns the generator drive shaft, where it is converted into additional electricity.
There are many different configurations for combined-cycle power plants, but typically each combustion turbine has an associated HRSG, and one or more HRSGs supply steam to a single 4 Combustion Turbine
Generator
Combustion Turbine
Generator
< < < <
< < <
<
HRSG
L
HRSG
L
I 2x1 Combined Cycle I
T
Steam
Turbine
Generator
Steam Condenser
.. ..... 1 Coo ling System I
steam turbine. For example, at a power plant in a 2x1 configuration, two combustion turbines/HRSGs supply steam to one STG; likewise, there can be 1x1 or 3x1 configurations.
The STG is custom-made to match the number and capacity for any combustion turbine/HRSG configuration (Figure 1.1).
Figure 1.1: Combined-Cycle Configuration
Combined-cycle configuration showing a 2x1 operation
Credit: California Energy Commission
Russell City Energy Center
Background
The Russell City Energy Center (RCEC) is a 600 megawatt (MW) natural gas-fired, 2x1 design, combined-cycle power plant located in Hayward (Alameda County). The project was certified in September 2002 and began commercial operation in August 2013. In February 2019, the CEC approved a project amendment allowing RCEC to install a 10 MW battery energy storage 5 system in response to the California Independent System Operator’s selection of the facility to provide black start battery energy storage capability should the grid go down and need a “jump start” to come back online.
The RCEC consists of two Siemens Westinghouse F-class CTGs; two HRSG’s; a single condensing GE D11 STG; a de-aerating surface condenser; a mechanical draft hybrid wet/dry plume-abated cooling tower. To control emissions of air pollutants, RCEC has gas turbines with dry, low nitrogen oxide (NOx) burners. The units use the best available control technology including selective catalytic reduction for control of NOx.
Figure 1.2: Russell City Energy Center HRSG
View of heat recovery steam generator from ground level Credit: California Energy Commission
Incident
Around 11:47 p.m. on May 27, 2021, RCEC experienced a mechanical failure of a STG that resulted in an explosion that threw dozens of metal pieces off the project site and resulted in an onsite fire requiring responses by the Hayward, Alameda County, and Fremont Fire Departments. The STG was severely damaged. In addition to the immediate public health and safety threat, this incident resulted in a loss of 600 MW of generating capacity from the grid.
Investigation
To investigate the RCEC incident, both the CEC and the California Public Utilities Commission (CPUC) inspection units established a Joint State Agency Investigation Team (JAIT). Along with the JAIT’s engineering and subject matter experts, third-party independent consultants, 6 Aspen Environmental Group, and West Peak Energy, were hired to support the JAIT’s 10- month investigation. The JAIT conducted:
Comprehensive inspections of the site on 12 different occasions including a three-day onsite gap-audit.
Weekly JAIT meetings.
Site tours for the five CEC commissioners, the CEC executive director, and the CPUC President.
Independent review and analysis of Structural Integrity Associates’ (SIA) root cause analysis report commissioned by the project owner.
Formal requests for information for more than 100 documents including maintenance reports, operation records, and other agency site visit reports such as from the Occupational Safety and Health Administration.
In-depth assessment of the documents and reports.
Interviews of several onsite witnesses and first responders.
The JAIT and the independent consultants conducted examinations and independent reviews of the facility and assessed the findings of the root cause analysis and supplemented gaps in the RCA Report. The JAIT’s focus included the equipment involved in the incident, the HRSG system, and any facility operations, maintenance, and management practices that may have contributed to the potential for this incident to occur. 7 CHAPTER 2:
Incident Investigation
Introduction
Under Public Resources Code section 25532 and Title 20, California Code of Regulations, section 1770, and the conditions of certification of CEC-issued facility licenses, CEC staff oversees a compliance monitoring and enforcement program. This includes inspection and enforcement activities to ensure that all CEC-jurisdictional electric generating facilities are operating in compliance with air and water quality, public health and safety, and other applicable regulations, guidelines, and conditions adopted or established by the CEC or specified in the license’s conditions of certification. Because of the seriousness of the RCEC incident, the CEC inspection team and multi-agency leadership visited the facility on several occasions, as summarized in Table 2.1.
Table 2.1: RCEC Investigation Timeline
Visit Date Agencies Represented Event
June 7, 2021 CEC and CPUC inspections
team
Initial inspection of the explosion and
fire site and interviewing employees,
first responders, and witnesses
August 3, 2021 CEC and CPUC inspections
teams
Establishing coordination with City of
Hayward. Met with Hayward Fire
Chief onsite to discuss the incident
and future coordination
August 5, 2021 CEC, CPUC, Hayward Fire
Department (they are the
local Certified Unified
Program Agency), and City
of Hayward’s City
Manager’s Office (COH)
Meeting to present the results of the
CPUC’s 2019 Audit, corrective actions
identified and implemented, discuss
RCA process and timeline, and site
inspection
August 16, 2021 CEC Chair David Hochschild
and former STEP Lead
Commissioner Karen
Douglas, Hayward Mayor
and Fire Chief, and CEC,
CPUC and COH staff
Overview of the CPUC 2019 Audit,
tour of explosion and fire site. Also
toured the Hayward’s Navigation
Center (transitional facility for the
unhoused) where a metal piece of
the STG penetrated the roof
August 19, 2021 CEC Commissioners
Andrew McAllister and
Overview of the CPUC 2019 Audit,
tour of explosion and fire site. Also
8 Patricia Monahan, Hayward
Mayor and Fire Chief, and
CEC, CPUC and COH staff
toured the Hayward’s Navigation
Center (transitional facility for the
unhoused) where a metal piece of
the STG penetrated the roof
August 27, 2021 CEC Vice Chair Siva Gunda
and former CPUC President
Marybel Batjer, Hayward
Mayor and Fire Chief, and
CEC, CPUC and COH staff
Overview of the CPUC 2019 Audit,
tour of explosion and fire site. Also
toured the Hayward’s Navigation
Center (transitional facility for the
unhoused) where a metal piece of
the STG penetrated the roof
October 19,
2021
CEC inspection team A compliance inspection of plant to review whether RCEC was in
conformance with the Conditions of
Certification
November 30,
2021
CEC, CPUC, CUPA, and
COH staff
Presentation from Structural Integrity
Associates on the Root Cause
Analysis Report and recommended
corrective actions, Q&A, and site
inspection
January 3, 2022 CEC and CPUC leadership
and staff, CUPA and COH
staff
Briefing on Structural Integrity
Associate’s Root Cause Analysis
Report and Corrective Actions, and
site tour and inspection
February 7 – 9,
2022
CEC and CPUC inspections
teams and expert
consultants from Aspen
Environmental Group and
West Peak Energy
Comprehensive onsite investigation
and gap-audit of the explosion and
fire site
March 21, 2022 CEC and CPUC inspection
teams
Corrective actions and timeline
presentation to Calpine’s RCEC
management
Source: California Energy Commission
Initial Site Inspection
In response to the May 2021 incident, the CEC staff initiated its investigation with an onsite inspection on June 7, 2021, to assess the damage from the STG failure and subsequent fire. 9 The CEC staff started at the STG structure deck and observed that the thrust bearings for the STG were exposed, and lube oil released.
Figure 2.1: Steam Turbine Generator
General Electric Model D11 Steam Turbine Generator Graphic Credit: Technical Training Professionals
During the initial onsite inspection, the CEC staff observed that the STG shaft was fractured at the exit point of the intermediate pressure (IP) section to the low-pressure section, and the shaft was ejected from the STG. The CEC staff examined the turbine shaft that entered the low-pressure (LP) section and found that the shaft was twisted. The metal casing of the LP section had separated, breaking bolts in the process.
The black charring and soot in the area around the casing were clear evidence that there was a fire at the exit of the LP section to the steam turbine’s generator. The LP section casing was severely damaged from the overspeed event. The drive shaft connecting the LP section to the generator was fractured at each end and had been thrown from the enclosure.
The various equipment on the STG structure deck also had extensive fire damage. The lube oil feed and return lines were severed and an estimated 4,000 gallons of lube oil was released which was contained by the secondary containment. The secondary containment was a concrete berm that surrounded the lube oil reservoir and the area beneath the STG. However, the water used by the Hayward Fire Department (HFD) to extinguish the STG fire collected in the secondary containment. Eventually the volume of water caused the secondary containment to overflow. The lube oil mixed with water made its way out to the stormwater retention pond of the site.
10 On the night of the incident, the HFD’s initial response was to contain the lube oil on site. They deployed booms in the stormwater retention pond and in one of the drainage canals from where the lube oil mixed with water was coming. The day after the initial release of the lube oil mixed with water the RCEC hired Environmental Logistics to do the required remediation work. The RCEC also hired a third-party biologist to survey the stormwater retention pond and the channel that feeds out to the San Francisco Bay. The biologist confirmed that no lube oil had made it off site.
The CEC staff surveyed the extent of the lube oil spill by starting at the outflow of the stormwater retention pond. The CEC was able to conclude that there was no immediate danger of lube oil leaving the site since the retention pond had sufficient capacity. The CEC staff inspected the surrounding area rocks, plants, and water and found no evidence of a lube oil spill in the outflow. However, there was evidence that lube oil entered the retention pond from the drainage canal behind the cooling tower of the power plant. The CEC staff also inspected the remediation work that was being conducted by Environmental Logistics.
Follow-Up Site Inspection
Staff returned to the RCEC site August 3, 2021, to examine the locations where metal pieces from the STG had landed after the incident. Representatives from the CPUC and the HFD also accompanied the CEC staff on this inspection. The CEC staff visited the Hayward Pollution Water Control Facility (HPWCF) and the Hayward Navigation Center to investigate where metal pieces from the STG had landed.
The CEC staff met with the HPWCF plant manager to examine the metal pieces thrown onto the site and to inspect any damage to the HPWCF facility. CEC staff learned from the plant manager that:
Some large metal pieces, weighing between 10 to 50 lbs., were found in the HPWCF drying beds southwest of RCEC. The pieces consisted of LP turbine blade parts and a large part of the LP turbine casing (Figure 2.2).
Some smaller metal pieces were found within the HPWCF facility to the east of RCEC that consisted mostly of copper (Figure 2.2).
Water treatment plant personnel sheltered in place during the fire with no injuries reported.
No structural damage occurred to the HPWCF facility.
11 Figure 2.2: Debris Field
Steam turbine generator debris field
Credit: California Energy Commission
The CEC staff then inspected the Hayward Navigation Center complex and met with the Housing Care Coordinator for Bay Area Community Services, the organization that operates the Hayward Navigation Center. Using data from the inspections at the HPWCF facility and the Hayward Navigation Center complex, the CEC staff created a map of the metal pieces found (Figure 2.2). Most of the metal pieces were found to the west of RCEC. These steel metal pieces ranged up to 50 lbs. Most of the small copper pieces of metal were found toward the east side of the power plant. The locations of the metal pieces released during the overspeed event are consistent with having been thrown from a rotating shaft.
The outlier of the debris field is the 12-pound piece of the LP turbine blade root. The blade root was discovered in the Hayward Navigation Center complex (Figure 2.2). The Hayward Navigation Center complex has multiple trailers on site serving people experiencing homelessness. The trailer used for meal preparation and eating was damaged when the 12- 12 pound metal piece traveled 1,200 feet, penetrated through the trailer’s roof, and landed on the floor (Figure 2.3). No individuals were in the trailer at the time since it was after operating hours. There was no damage to the floor of the trailer.
Figure 2.3: RCEC Metal Piece
The metal piece traveled approximately 1,200 feet and weighed 12 lbs.
Credit: Hayward Navigation Center Staff
Fire Department Response
The HFD was the first responder to the RCEC STG incident and requested back-up from the Alameda County Fire Department and the Fremont Fire Department. The CEC staff interviewed the HFD battalion chief who was the incident commander on the night of the STG incident. At the gate, RCEC personnel informed the battalion chief that 45 hydrogen cylinders could be exposed to the onsite fire. The battalion chief established fire engine teams in various locations around the site. The main emphasis was to contain the fire at the STG deck structure. Some of the fire engine teams were dispatched to help contain the lube oil/fire water that had escaped from secondary containment. CEC staff corroborated these events by reviewing the incident reports and radio recordings from the three responding fire departments.
During the fire department’s response, four fire fighters suffered injuries. There was concern from the HFD that the fire fighters could have been exposed to toxic air contaminants.
Because of this concern, the JAIT required that the RCEC conduct a separate analysis to determine if there was toxic exposure to the fire fighters. Jensen Hughes, a third-party consultant, analyzed the combustion byproducts from the incident fire. Their analysis determined that the fire did not produce environmental toxins that are untypical of an 13 industrial fire. Therefore, there is no reason to believe that the fire fighters were exposed to any unexpected environmental toxins by responding to the incident.
During one of the interviews with the HFD battalion chief, the CEC staff learned that there was a possible accident involving a vehicle hitting debris in the roadway on State Highway 92 near the toll plaza the night of the incident. The CEC staff reached out to the local California Highway Patrol (CHP) office for more information on the possible accident. The Hayward CHP office had no record of an accident that night matching that description in its jurisdiction.
However, the Hayward CHP office mentioned that it could have occurred in the Redwood City CHP’s jurisdiction. The CEC staff followed up in person October 19, 2021, at the Redwood City CHP office to ask if they had any records of a possible accident on the night of the RCEC incident. Redwood City CHP personnel reviewed the records and confirmed that there were no accidents matching the description the night of the incident. Therefore, the CEC staff concluded that there is no evidence of an accident on State Highway 92 from the STG debris from RCEC.
Compliance Inspection
The CEC maintains a compliance monitoring and enforcement program to ensure that permitted thermal power plants are constructed, operated, and decommissioned in accordance with the associated conditions of certification and all applicable laws, ordinances, regulations, and standards. The CEC’s post-certification compliance monitoring and enforcement authority can be found in Public Resources Code sections 25532 to 25534.2 and Title 20, California Code of Regulations, sections 1751 to 1770, as well as in conditions of certification within facility licenses. Physical compliance inspections are one tool that the CEC uses to maintain the compliance monitoring and enforcement program. The RCEC compliance inspection conducted October 19, 2021, focused on the areas of security, worker safety, hazardous materials management, and fire protection systems and maintenance. The CEC staff requested and reviewed documentation from RCEC related to worker safety, hazardous materials management, and fire protection systems and maintenance.
The compliance inspection included visual observation of the fire protection systems, site security, hazardous materials management, chemical storage, STG, water treatment area and associated bulk chemicals, and the zero-liquid-discharge (ZLD) system of the facility. The ZLD system had several lock-out tag-out (LOTO) tags on the electrical supply and control panels to prevent the ZLD system from activating. The CEC staff witnessed LOTO’s on several major systems throughout the plant. Material and equipment were also stored in various locations around the facility to prepare for the repair of the STG. The CEC staff observed that the plant appeared to be acceptably maintained.
Root Cause Analysis
The JAIT requested additional information regarding operations prior to the STG incident (Appendix A). Documentation related to the control system for the facility was reviewed along with piping and instrumentation diagrams for the STG and HRSG. 14 The root cause analysis was released to the CEC and the CPUC on November 24, 2021. The JAIT reviewed the root cause analysis and confirmed it by reviewing various reports, data, and documentation that the JAIT requested throughout the investigation. The JAIT determined that the root cause analysis was silent or did not go far enough in some areas, including maintenance, control room operator interface and training, and inadequate water induction protection. This determination required the JAIT to conduct a “gap” analysis to fill in the areas where the root cause analysis was lacking and address several unanswered “why” questions.
RCEC was informed of the upcoming audit on January 13, 2022. The JAIT conducted the audit of RCEC from February 7 to February 9, 2022, in support of the gap analysis. 15 CHAPTER 3:
Technical Analysis
Incident Causation
As RCEC’s night shift operating crew was going through its scheduled shutdown process on the evening of May 27, 2021, the facility suffered a serious incident that resulted in an explosion of the STG and subsequent fire involving released lubrication oil. Before the incident, the RCEC had been operating in a 1x1 configuration, with Combustion Turbine 2 (CT-2) and the associated HRSG-2 in operation. CT-2 was producing electricity, and HRSG-2 was producing steam to drive the STG. While the RCEC was operating in this 1x1 configuration, Combustion Turbine 1 (CT-1) and the associated HRSG-1 were offline and not operating. The RCEC had been in this operational configuration for about two days. Unknown to the power plant’s control room operator and onsite crew, the on-line HRSG-1 cold reheat isolation valve (CRH-1), designed to prevent steam being produced by the operating HRSG-2 from entering the reheat section of the offline HRSG-1, failed to close completely and allowed steam to leak past it. The CRH-1 valve’s actuator erroneously signaled to the control room operator that the valve had closed completely.
Over the two days operating in that 1x1 configuration, the steam produced in HRSG-2 continued to leak past the CRH-1 valve into the offline and much cooler HRSG-1 causing the steam to condense into water. Enough water condensed in the offline HRSG-1 to collect a substantial quantity of water in the section of the offline HRSG-1 known as the reheater (Figure 3.1). Because of the large quantity and high temperature of the condensing superheated steam, the condensed water remained near its boiling point and at an elevated pressure.
16 FAILED TO
COMPLETELY
CLOSE
"Cold Reheat"
Steam Lines CRH
(Return from
HP turb ine)
HRSG #1
"Hot Reheat "
Steam Lines HRH
SEIZED IN
OPEN POSITION
! "Closed D rains" !
J Hea der I
i CRV-2 ►
One -way
Stop/C heck
Valves
IP Turbine
Figure 3.1: Russell City HRSG Reheater Schematic
Schematic of RCEC’s HRSG reheater path to the IP section of the steam turbine Credit: California Energy Commission
As is normal during a shutdown, the steam pressure being supplied by HRSG-2 began a gradual, but steady, decline. At a point during the shutdown, the pressure being supplied by the HRSG-2 to the STG, dropped to a lower level than the pressure in HRSG-1, which was being maintained at its saturation (i.e., boiling) temperature. This situation allowed the accumulated water in the HRSG-1 to enter the IP section of the STG supply piping and flow past the hot reheat stop/check valve (HRH-1) and control valve (CRV-2) of the STG. The introduction of the cooler condensed water to the CRV-2 valve, located at the entrance to the IP section of the STG, caused it to seize in the open position.
As water flowed into the IP section of the STG, the power output of the STG began to fluctuate, then dropped suddenly to below zero output. At this instant, the STG control system initiated “motoring” of the generator, which is a term used when the STG is connected to the grid but is absorbing rather than putting out power to the grid. When motoring, the STG uses grid power to maintain the required 3600-rpm rotational speed. After about 30 seconds of motoring, the STG had yet to recover outputting electrical power, the automatic controls ceased the motoring, and the controls opened the breakers to disconnect the STG’s generator from the grid. Over the next few seconds, as the water cleared from the STG, with the CRV-2 valve of the STG still seized in the open position allowing steam to continue flowing, and with 17 no generator load to slow the rotation, the STG began to spin freely. From this point on, the STG increased rotational speed until it was torn apart by centrifugal forces.
Gap Audit Conclusions
A review of SIA’s root cause analysis provided an understanding of how the conclusions in the root cause analysis were determined. The JAIT determined that a broader scope of investigation was needed. The JAIT conducted interviews and technical discussions with RCEC staff and reviewed technical documents and operational logs onsite at the RCEC for three days from February 7 through February 9, 2022.
Following the onsite investigations and review of collected data, the JAIT concluded that the root cause of the event included three essential causal factors:
1. Steam leakage past the not-completely closed CRH-1 steam isolation valve allowed water to condense and accumulate in the out-of-service HRSG-1 reheater section.
2. Accumulated water was not detected and drained from the HRSG-1.
3. The HRH-1 stop/check valve in the steam line between the HRSG-1 reheat section and the STG had not been closed into a blocked position which allowed the water to be drawn into the IP section of the STG as described above in “Incident Causation.” The JAIT also determined that the key contributing conditions associated with the factors were:
1. Maintenance failures were associated with critical components.
2. Critical control system alarm points at RCEC are not “aggregated” into one alarm point system (annunciator), and operators are expected to monitor several systems during critical events, leading to a loss of situational awareness so that they could not react to the fact that a substantial amount of water had accumulated in HRSG-1.
3. The operations staff failed to identify the manually operated stop/check valve as a potentially critical blocking valve should water collection occur in the offline reheat steam piping.
To address the identified causal factors and key contributing conditions, the JAIT identified a set of corrective actions for implementation at the RCEC that the JAIT determined were necessary to ensure that the risk of a similar water induction incident occurring in the future is eliminated to the degree feasible by deploying redundant systems of prevention and detection.
Equipment Maintenance and Monitoring Program
The failure of the CRH-1 valve to close properly was identified as one of the causal factors of the incident by both SIA and the JAIT. SIA’s root cause analysis found that its actuator/gearbox assembly was degraded with severely worn internals of the gearbox. SIA’s root cause analysis concluded that the damage to the gearbox was caused by a heavily damaged gear box worm shaft roller bearing. The bearing components had been trapped within the worm and quarter gear further damaging the gearbox. This damage increased the gearbox backlash and resulted in reduced valve stroke.
18 The JAIT also observed evidence of inadequate lubrication and water intrusion into the gearbox. The CRH-1 valve stem packing showed lack of lubrication and extreme wear of its bushings. The resulting surface galling (pitting wear) would have required higher-than-normal actuation forces. Broken and chipped quarter-gear teeth in the gearbox were evidence of the gearbox operating under higher than designed-for loads, consistent with higher-than-normal valve torque required to compensate for the worn valve stem bushings. Maintenance records for the valve actuator reviewed by the JAIT showed that the actuation torque had been increased multiple times, and the actuator required replacement during the previous period, spanning more than a year.
Regular service inspections and lubrication would have likely detected or prevented the extreme wear that contributed to the failure of the valve to close. The JAIT recommends that the RCEC implement a regular preventative maintenance plan for this critical component. The preventative maintenance program will include the valve, gearbox, and actuator assemblies including the frequency of inspections, scope of services, and lubrication requirements. The JAIT will review the new preventative maintenance program to be developed by the RCEC.
Control Room Operator Interface and Training
The Mark VI control system of the STG was not fully integrated into the overall distributed control system (DCS) for the power plant. Therefore, the RCEC control room operators were responsible for monitoring the outputs from both the Mark VI control system and the DCS.
With multiple control systems operating in parallel, there is potential for operator confusion from nuisance alarms when the DCS system does not have all the power plant alarm points in a single system with levels of priority established for all of them. When alarm points are consolidated into one system, such as the DCS at the RCEC, the alarms can be organized into priority levels depending on importance and urgency.
The JAIT recommends that the Mark VI high priority alarm points be integrated into the DCS.
The reasoning is that the RCEC operators “can only see one thing at a time,” and in critical events the operator should have to view only one area for high-priority alarms which can be acted upon without confusion or delay. Furthermore, the integration of the Mark VI into the DCS allows for the application of a smart alarm logic system which can assist operators with nuisance alarms. Alarms in a repair state or nuisance depending on the order of importance can be suppressed so operators do not miss critical alarm notifications.
An additional complication for the control room operators identified from the gap audit was the lack of a unified clock for the control systems. The RCEC uses a DCS that controls the power plant that has a clock that stamps alarm times for various errors and conditions. The Mark VI also has its own clock that stamps STG alarms with times, but the times were not in agreement with there being a one-hour and 56-minute difference between the two control systems’ clocks. With alarms and alerts being sent to the operator simultaneously by the two control systems, but having discrepant timestamps almost two hours apart, it is difficult to confirm their order of arrival and timeliness, both during and after critical events. The JAIT recommends that the RCEC consolidate the time and date stamp for the DCS and Mark VI control systems so that they remain synchronized.
19 The JAIT also identified a need for additional training for the control room operators and developed a corrective action to require that the RCEC implement new training procedures for water induction events. Over the two days of 1x1 operation preceding the event, the control room operator was unable to identify the accumulation of water or the higher-than-expected pressure in the offline HRSG-1. This lack of knowledge demonstrated a lack of operator training for water induction events. Therefore, the procedure will be targeted toward assuring that operators have the proper authority and are trained to immediately act when certain indications suggest the possibility of water induction either before or during an event. The damage to the STG could have been avoided had the operators been provided situational awareness and adequate training enabling them to take appropriate and timely action.
For completeness, the JAIT reviewed the trip alarm sequencing that occurred during the incident to confirm that the STG was motoring for almost 30 seconds before the steam turbine’s generator primary protective relay device tripped the STG from the electrical grid. No evidence of noncompliance was found. Known as the “86-relay systems,” their design, installation, and operations are governed by the North American Electric Reliability Corporation (NERC) regulations and the Institute of Electrical and Electronics Engineers (IEEE) design codes. The JAIT reviewed and verified that these advanced diagnostic systems were configured, calibrated, and operated as designed per the relevant industry codes and standard guidelines.
Inadequate Water Induction Protection
At the time that the RCEC submitted construction design plans to the CEC’s Chief Building Official for approval, which established the applicable editions of building and engineering codes,1 RCEC was required to comply with all applicable California Building Codes and engineering laws, ordinances, regulations, and standards. The primary standard that addressed water induction at the time of the RCEC’s construction was the 2006 version of the American Society of Mechanical Engineers (ASME) TDP-1 titled TDP-1-2006, Recommended Practices for the Prevention of Water Damage to Steam Turbines Used for Electric Power Generation: Fossil-Fueled Plants. This standard was considered voluntary guidance containing suggestions for steam power plant design features that “should” be considered and addressed.
However, Calpine’s Application for Certification for RCEC indicated that the ASME TDP-1 was “applicable to the mechanical aspects of the power facility” and represented that ASME TDP-1 1 RCEC Condition of Certification GEN-1 states:
The project owner shall design, construct and inspect the project in accordance with the 2001 California Building Code (CBC) and all other applicable engineering LORS in effect at the time initial design plans are submitted to the CBO for review and approval. (The CBC in effect is that edition that has been adopted by the California Building Standards Commission and published at least 180 days previously.) All transmission facilities (lines, switchyards, switching stations, and substations) are handled in Conditions of Certification in the Transmission System Engineering section of this document.
In the event that the initial engineering designs are submitted to the CBO when a successor to the 2001 CBC is in effect, the 2001 CBC provisions identified herein shall be replaced with the applicable successor provisions. Where, in any specific case, different sections of the code specify different materials, methods of construction, or other requirements, the most restrictive shall govern.
Where there is a conflict between a general requirement and a specific requirement, the specific requirement shall govern. (Emphasis added.) 20 was among the codes and standards that would be “used in the design and construction of mechanical engineering systems for the Russell City Energy Center (RCEC).”2 The basic philosophy of ASME TDP-1 is to prevent water damage to steam turbines by providing three layers of protection wherever practical. The first layer of protection would be to prevent the intrusion of steam condensate into piping locations where it could be reasonably expected for the steam condensate to be driven into the operating steam turbine’s inlets. Should the first layer fail, the second layer would be to detect and drain any substantial collections of water as they occur so that the steam condensate would not pose any subsequent risk to the STG. If the first and second layers fail, the third layer would prevent the release of any collected steam condensate to the STG by providing positive isolation via a blocking valve. Thus, by requiring three critical layers of protection, the possibility of a water induction event of this kind is reduced to a level of being extremely unlikely.
Typically, power plants’ design elements are selected through contract negotiations between the owner and the major construction company and suppliers. Hence, the level of adoption of the recommended practices, in this case ASME TDP-1, was left to the owner’s discretion. In the case of RCEC’s design, there was not complete adoption of ASME TDP-1 into the design and construction with respect to its protection against steam turbine water induction events.
Some practices that are recommended to be automated under ASME TDP-1 were left at the discretion of the operators to fulfill through manual procedures.
At the time of the incident, RCEC did not have adequate protection from all three ASME TDP-1 intended layers of protection. Interviews with Calpine staff verified that they did not trust the reliability of the temperature and pressure sensor network of the HRSGs to detect water accumulation when they were offline. Hence, the alarms coming from the offline HRSG (and reheater section) were not acted upon. SIA’s root cause analysis also considered the water detection to be unreliable in the offline HRSG. Thus, there was no effective water detection (second layer of protection) that would have enabled the operators to open HRSG drains to dispose of accumulated condensed water.
Also, the manually operated stop/check valve (HRH-1) at the end of the offline reheater section of HRSG-1 was left in an “unblocked” configuration. This meant the HRH-1 valve operated as a one-way “check” valve, capable only of preventing steam from entering the reheater section from the STG direction. It was not capable of preventing the exiting of condensed water from the reheater to the STG. Therefore, the HRH-1 valve did not act as a blocking valve, meaning there was no effective capture of condensed water (third layer of protection).
With no functioning second or third layers of water induction prevention, RCEC was dependent solely on the CRH-1 valve (first layer of protection) to prevent any water accumulation from occurring in the first place. This allowed for a potential single point failure for a water induction event. In the operational period leading up to the incident, the valve leaked steam 2 Russell City Energy Center AFC, Vol. II, at p. 10C-1. 21 past it, allowing substantial water accumulation in the HRSG-1 reheater. Without the functional protection from the second and third layers of protection, the accumulated water went undetected, undrained, and uncaptured, making its way into the steam turbine. A design goal of TDP-1’s recommendations is to avoid potential “single point failures” that can lead to water induction. Each of the three causal factors identified in Gap Audit Conclusions, can be associated with one of the three critical layers of protection described above.
The JAIT proposes as a corrective action that the RCEC perform a conformance analysis of the existing power plant against ASME TDP-1-2013 to determine what additional modifications should be made to reduce the possibility of a future water induction event. The JAIT review of the conformance analysis will ensure that practical modifications would be identified and implemented. In addition, the CEC’s Delegate Chief Building Official (DCBO) would ensure that these required modifications are implemented to required industry codes and standards.
Therefore, the identified changes from the conformance analysis will bring the practices of the RCEC in alignment, to the extent feasible, with the current version of ASME TDP-1-2013, Prevention of Water Damage to Steam Turbines Used for Electric Power Generation: Fossil- Fuel Plants.
SIA’s RCA provided a list of restoration recommendations based on their analysis of the incident. Their restoration recommendations included:
Implement controls logic to utilize existing HRSG reheated system drains to discharge water from the HRSG harps when offline.
Implement controls logic to utilize existing HRSG reheated system drains to alleviate undesirable pressure within the HRSG reheater system when offline. Re-configure the CRH stop valve to close based on its actuator torque value. Convert the HRH stop/check valve from manually operated to electrically actuated including the implementation of controls logic to positively isolate the offline HRH piping and HRSG Reheat.
In addition to the JAIT’s corrective actions, the JAIT agrees that SIA’s restoration recommendations are appropriate and should be implemented as corrective actions prior to RCEC resuming commercial operations. The first restoration recommendation would allow any accumulated water in the offline HRSG to be detected and drained in a timely manner. The second restoration recommendation would prevent the buildup of excess pressure while the HRSG is offline. The third restoration recommendation would improve the reliability of the CRH-1 valve in providing positive isolation and would reduce the likelihood of damage to its actuator and gearbox by preventing excess closing forces. The fourth restoration recommendation would prevent the HRH stop/check valves from remaining unblocked when the HRSG is offline, thus preventing any collected water from being drawn to the operating steam turbine. All these changes are consistent with assuring the three levels of protection discussed earlier as necessary for conformance with ASME TDP-1.
The SIA restoration recommendations will also require changes to the maintenance and operating procedures of the RCEC. Therefore, the JAIT has developed a corrective action to require that Calpine develop the necessary revised procedures needed for the implementation 22 of the SIA restoration recommendations and provide them for CEC review. This will ensure that the RCEC has correctly implemented the appropriate procedural updates based on the SIA restoration recommendations and that the CEC provides oversight appropriate to its licensing authority.
The JAIT examined other potential paths for water to get to the STG. One area of concern was the steam attemperators, devices that control the steam temperature. Steam attemperators and mixers spray high pressure feedwater or steam mixed with the feedwater into the main steam line and control final steam temperature to the turbine. Malfunctioning or leaking attemperators and mixers are known to be a potential source for water induction and are addressed in the 2013 version of the American Society of Mechanical Engineers (ASME) standard: ASME TDP-1, Prevention of Water Damage to Steam Turbines Used for Electric Power Generation: Fossil-Fuel Plants. Annual inspection procedures are recommended. The JAIT recommends that RCEC create an annual preventative maintenance program for the steam attemperators that will be reviewed and approved by the JAIT. This maintenance program would ensure that the steam attemperators are operating correctly and will further reduce the probability of water induction in the future.
Public Safety
The JAIT concluded that the STG overspeed and subsequent explosion was due to a water induction event. Water induction events are considered by the industry to be a low probability event with a potential high impact, including to the surrounding community, as confirmed by this STG failure. The JAIT recommends that the RCEC look at other systems on their facility that could present a similar low probability/high risk of impact on the surrounding community.
Candidate systems for review include ammonia storage, fuel gas systems, hydrogen storage, and battery energy storage systems. The CEC has tasked the DCBO to review these systems for code compliance and produce a report detailing its findings. Any deficiencies would be corrected by the RCEC. This will ensure that the four identified systems do not present significant risks.
23 ■
CHAPTER 4:
Conclusions
Corrective Actions
In response to the RCEC’s May 2021 incident, the focus of this report has been to highlight the JAIT’s investigative activities, provide an understanding of what occurred, and determine the appropriate corrective actions necessary to allow RCEC to return to operation safely.
After reviewing SIA’s root cause analysis, the JAIT conducted an audit to address some perceived gaps. This gap audit was necessary to determine whether there were contributing factors to the event that were outside the scope of, or not addressed in, SIA’s root cause analysis. After completing the gap audit, the JAIT determined that there were three overall casual factors to the water induction event: (1) deficiencies in maintenance for some critical equipment, such as the CRH-1 valve assembly, (2) deficiencies in control room operator interface and training, such as the inability to detect and respond to water accumulation in the offline HRSG, and (3) inadequate protection from water induction, such as reliance on a single valve to prevent accumulation in the offline HRSG. Corrective actions were developed to address these three casual factors.
The corrective actions for the equipment maintenance and monitoring program at the RCEC include:
For each HRSG, implement a preventative maintenance and monitoring program for the cold reheat (CRH-1) valve, gearbox and actuator assemblies that includes frequency of inspections, services, and lubrication for review and approval. For each HRSG, implement an annual preventative maintenance program for the steam attemperators and mixers for review and approval.
Revise operations procedures needed to accommodate implementation of SIA’s restoration recommendations for review and approval.
The corrective actions for the control room operator interface and training include:
Synchronize the internal clocks that generate the time and date stamps for alerts and alarms for both the Mark VI and the distributed control system. Review and evaluate the alarm and trip points of RCEC’s programmable logic controllers making them more sensitive to alarm settings, where appropriate.
Consolidate the alarms generated by the DCS and Mark VI control systems into a single control system to reduce the need for operations staff to monitor multiple systems simultaneously.
Reduce the occurrence of nuisance/false alarms by providing “smart alarm” logic in the consolidated DCS and Mark VI control systems and provide an updated operator training that includes water induction events along with evidence of its completion.
The corrective actions for inadequate water induction protection include:
24 Provide an ASME TDP-1-2013 conformance analysis for the RCEC. Provide the list of design modifications that are being implemented at RCEC based on the ASME TDP-1-2013 conformance analysis along with evidence of their completion. Implement the SIA restoration recommendations along with evidence of their completion.
These corrective actions contain a mix of operations and maintenance changes, improvements to operator notification systems to improve situational awareness, and upgrades to hardware and control system integration. These corrective actions would not expand the facility, change the performance of the facility, or require any changes to existing conditions of certification.
These corrective actions are required to be implemented prior to the facility restarting combined-cycle operations. Verification that the corrective actions have been completed will be achieved through review of documentation provided by RCEC and by onsite inspection by the DCBO and JAIT staff or retained consultants.
In addition to the required corrective actions for the water induction event, the JAIT also conducted an audit of other aspects of the power plant. This included tasking the CEC’s DCBO with reviewing several systems on site that could have potential for offsite consequences to the surrounding community and providing a report that will specify whether the systems are code compliant and will detail any deficiencies needing correction. Such inspections are periodically conducted to help ensure that the normal operation of the plant remains in compliance with applicable regulations and industry standards. Collectively, these nine corrective actions will address the three causal factors that were identified by the JAIT during the investigation. Completion of the above corrective actions by the RCEC would prevent, to the degree feasible, any future turbine overspeed events due to water induction by deploying redundant systems of prevention and detection.
25 APPENDIX A:
Joint State Agency Investigation Team Document Requests Table A.1 details the requested additional information for the events surrounding the STG incident.
Table A.1: Document Request Timeline
Document
Request Title Due Date Documents requested
DR-
RC20210528-01
Monday,
June 7, 2021
Operator Logbooks from May 25, 2021 through May 30, 2021 Steam Turbine (ST) OEM Manuals for Lube Oil Bearing Seals Lube Oil Analysis Past Three (3) Years
Work Orders for all ST Bearing Seals Past Three (3) Years Failure and Root Cause Analysis of Failure and Fire (When available) DR-
RC20210617-02
Wednesday,
July 7, 2021
Digital Control System (DCS) Logs from 05/26/2021 at 00:01 hrs. to 05/28/2021 at 24:00 hrs.
DCS Instrument Calibration Records; most recent.
Overspeed Trip Tests; past three (3) years.
Plant Operators Training Records; past two (2) years.
Plant Organization Chart.
OSHA 300 Reports; past five (5) years.
Current Air Permits.
RATA Testing; past five (5) Years.
Shutdown Checklist.
DR-
RC20210617-03
Friday,
July 30, 2021
Steam Turbine rotational speed records of any type from 05/26/2021: 00:01 hrs. through 05/27/2021: 24:00 hrs.
Work Orders for all DCS Alarms from 05/25/2021: 00:01 hrs. through 05/27/2021: 21:47 hrs.
Status
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
A-1
DR-
RC20210617-04
Wednesday,
August 11,
2021
Steam Turbine rotational speed records of any type from 05/26/2021: 00:01 hrs. through 05/28/2021: 24:00 hrs.
Received
Steam Turbine P&ID’s with all current and intended modifications to the steam system.
Received
Operator Training Procedures Received
Operator Training Curriculum Received
Operator Qualifications Received
Operator Job Description Received
Calpine Technical Training Information Received
DR-
RC20210617-05
Friday,
August 27,
2021
Completed Responses to the attached “Russell City Operator Questionnaire” Received
DR-
RC20210617-06
Friday,
September 17,
2021
An unredacted copy of the full and final Root Cause Analysis of the incident DR-
RC20211019-07
Tuesday,
October 26,
2021
An unredacted copy of the full and final Root Cause Analysis of the incident The Department of Toxic Substance Control hazardous waste compliance report approved by the Certified Unified Program Agency (CUPA) involved and corrective Delayed
Received
DR-
RC20211026-08
Tuesday,
November 2,
2021
actions taken this year for all hazardous materials (Hazmat) accumulation storage areas (seven violations were indicated), fire suppression water clean-up, and other Hazmat waste at the Russell City Energy Center (RCEC).
Referred to
City of
Hayward
All RCEC Hazmat Manifests for the current year. Received Any photo evidence to substantiate EPA compliance.
Referred to
City of
Hayward
DR-
RC20211112-09
City of Hayward
Completed April
14, 2022
The Department of Toxic Substance Control hazardous waste compliance approved by the Certified Unified Program Agency (CUPA) involved and corrective actions taken this year (2021) for all hazardous materials (Hazmat) accumulation storage areas.
Open
A-2
Any photo evidence to substantiate EPA compliance.
DR-
RC20211026-10
Tuesday,
December 7,
2021
OEM Manuals for the HRSG Cold Reheat Stop Valve (CRHSV#1) All recent Preventative Maintenance Work Orders for the CRHSV#1prior to the overspeed event.
DR-
RC20211026-11
Friday,
December 17,
2021
OEM Gear/Actuator Installation, Operating and Maintenance Instructions for the HRSG Cold Reheat Stop Valve (CRHSV#1).
Received
Any post event Toxic Substance and Human Exposure evaluations performed All attachments to and appendices and referenced photos in the RCA. Received Power plant reconfiguration/startup checklist (starting in 1X1 mode, for changing from 2X1 operation to 1X1 operation, or from 1X1 to 2X1 operation) Received
Item 2 above, (completed) for the final configuration change prior to the incident Received Shut down checklist (completed) for the incident Received All manuals, presentations, and other documents regarding operator/employee trainings in effect at the time of the incident.
Received
Training status of personnel performing the startup/operation/shutdown leading to the incident
Received
"Additional operating data" referenced on pg. 14 of the RCA Received Extended operating data of startups and shutdowns of HRSG#1 and HRSG#2 (extending 2 hours or more after startups, and beginning 2 hours or more before shutdowns)
Received
A simplified schematic representation (similar to the figures shown in the presentation on November 30, 2021) of reheat loop/IP turbine including piping, drains, valves, sensors with labels as used in the RCA
Received
The presentation that was given on November 30, 2021 . Received Glossary of acronyms used in RCA Received
All earlier versions of RCA, or any portion thereof, including but not limited to the first version of the RCA summary.
Received
Prior risk assessment done for the 1x1 operation configuration (e.g. FMEA, fault- tree, or other).
Received
DR-
RC20211026-12
Open
Received
Received
Received
Tuesday,
January 12,
2021
A-3
Report of evaluation and test results of the valve (HRSG #1 CRH stop valve); Please provide the maintenance records for the last two years for HRSG #1 and HRSG #2 CRH stop valves.
Received
Manufacturer’s HRSG #1 CRH stop valve specs and assembly drawings. Received Report of investigation and test results of the IP-stop valve (IV#2 & RSV #2) Received IP-stop valve (IV#2 & RSV #2) manufacturer’s specs and assembly drawings. Received The assessment report that determined the viability of reusing or repairing the HP/IP turbine post incident.
Received
SIA and Calpine operator interviews for the personnel on site during the incident, including any transcripts, notes, or other recordings, including audio or video, of the interviews.
Received
The borescope inspection report of the horizontal HRH pipe sections of the CRVs and the inlet of the CRVs.
Received
A clarification of the statement in Section 5.11 of the RCA, “STG line breakers opening prior to the closure of IV #2 and RSV #2 based on delay logic within the protection system.”
Received
Any other reports generated by SIA concerning the facility or the incident, including but not limited to the recommendations made by SIA or any report regarding recommendations.
Received
Contracts between Calpine and SIA relevant to the RCA, including but not limited to the second contract for recommendations.
Received
All documents and information regarding the facility’s alarm design and/or protocols, including documents and information regarding the alarm priority levels.
Received
Agreements with PG&E (including power purchase agreements) that govern operation of the facility.
Received
Any communications from or with PG&E (written or emailed) regarding this facility, including notes from calls or oral communications with PG&E, on May 27, 2021, or during the 10 days prior and after May 27, 2021.
Received
A list of all documents SIA reviewed during preparation of the RCA and recommendation report.
Received
A-4
DR-
RC20210527-13
Monday,
January 17,
2022
All CAL-OSHA 300 reports for the current year (2021)
An Analysis or Testing of the Toxins released (airborne or otherwise) from the Incident
Evaluation of the Health and Physical Impact of the Toxins Released Received
Received
Proof of Notification of all exposed or affected people and personnel DR-
RC20220121-14
Friday,
January 28,
2022
P&ID Symbol Legend sheet
P&ID 25483-000-V1A-MBPR-00011
P&ID’s 103200-PID-002 through 103200-PID-012
P&ID 25483-000-M6-AB-00002
P&ID 25483-000-M6-AB-00005
P&ID 25483-000-M6-AB-00006
P&ID M6-BM-00001
P&ID M6-AE-00001
All post incident reports or debriefs from all operators that who were onsite during the incident.
Nooter-Eriksen HRSG drawings showing sectionals and side views with all drains and Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
Received
sizes.
OEM cold start procedure from Bechtel or others.
Operating procedures for total plant shutdown.
Operating history for changing from 2x1 to 1x1 operation for the past 3 years.
OEM recommendations operations changes from 2x1 to 1x1 operation.
Schematic of HRSG drainage valves on the steam pendant.
ALL DCS Alarms from 05/22/2021; 00:01 hrs. through 05/28/2021; 24:00 hrs.
Received
Received
Received
Received
Received
Received
Received
DR-RC20220201- Monday,
15 February 7, 2022
Pursuant to section 11.2 of General Order 167-B, the California Public Utilities Commission’s (CPUC) Electric Safety and Reliability Branch (ESRB), in cooperation with the California Energy Commission’s (CEC) Siting, Transmission and Environmental Protection Division, requests that Calpine have all staff that were onsite during the May 27, 2021, incident be available for interviews one day during the joint agencies’ on-site investigation taking place on February 7-11, 2022. Each staff member should be available at least one day during the on-site investigation including but not limited to Mr.
Warren Mushatt and the control room operator who was on the shift just prior Mr.
Supervisor
and Operator
are no longer
employed /
available
A-5
Warren Mushatt. Please confirm the availability of each staff member on or before February 4, 2022.
DR-RC20220103-
16 Sunday,
February 13,
2022
SUPERCEEDED BY DR-RC20220802-17
Logic Scope for SIA Corrective Action Recommendations Received Logic for Generator Lockout Protective Relays Received
Received
Received
Received
Daily Rounds Sheets from 05/22/21: 00:01 through 05/28/22: 23:59 Received DCS Logs for a similar change in operation, 2X1 to HRSG 2: 1X1 Operation Received DR-RC202200223-
18 Tuesday, March
1, 2022
Russell Energy Center Steam Turbine Generator (STG) Generator Protection Relay 86 (A&B) device alarm and trip history for the May 27, 2021, incident. There are two redundant devices, a Beckwith M-3425 86 Relay and an SEL Model 300G. Therefore, information from both devices should be included.
Open
Bechtel logic drawings and instrumentation calibration history for the Beckwith and SEL relays previously noted showing how these devices were designed to work and what options were selected when they were installed.
Open
Provide photographs or material inventory logs of the type of oil/grease used as lubrication for the CRH stop valve actuator assembly. Open DR-RC20220802-
17 Friday,
February 11,
2022
Larger and Clearer Graphs of graph 5.2 & 5.10 from the SIA RCA Unit Trip criteria for Vibration Parameters (i.e. graph 5-2) List of Procedural Changes as a result of all Corrective Actions Source: California Energy Commission
A-6