Control of Temporary Modifications

Content

• Introduction
• Notice Summary
• Applicability
• Summary of Events
• Significance of Events
• Corrective Actions
• Hazard Mitigation
• References

Introduction

This notice is one in a series of publications issued by the Office of Nuclear and Facility Safety to share nuclear safety information throughout the Department of Energy complex. For more information, contact Dick Trevillian, Office of Operating Experience Analysis and Feedback, Office of Nuclear and Facility Safety, U.S. Department of Energy, Washington, DC 20585, telephone (301) 903-3074. No specific action or responses are required solely as a result of this notice.

Safety Notices are distributed to U.S. Department of Energy Program Offices, Field Offices, and contractors who have responsibility for the operation and maintenance of nuclear and related facilities, and to other organizations involved in nuclear safety. Written requests to be added to or deleted from the distribution of Safety Notices should be sent to: BR Richard L. Trevillian, EH-33, Room E-460 GTN, U.S. Department of Energy, Washington, DC 20585.

The ESH Office of Information Management maintains a file of Safety Notices and supporting information. Copies can be obtained by contacting the Office of Information Management at (301) 903-0449 or by writing to the Office of Information Management, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

Notice Summary

This notice presents lessons learned about the necessity of controlling temporary modifications, specifically with respect to installation and operability testing. Inadequate design and review of temporary modifications has resulted in damage to both temporary and permanent equipment and forced suspension of facility operations. This notice describes five events related to temporary modifications (including one event at River Bend Nuclear Plant), provides generic information on potential problem areas, and discusses recommended corrective action.

Applicability

The information contained in this notice applies to all DOE organizations when plant modifications to equipment and facilities are made. The Office of Nuclear and Facility Safety (NFS) advises facility operators to identify potential safety hazards associated with temporary modifications and to be aware of conditions that may lead to such hazards.

No specific action or responses are required solely as a result of this notice.

Summary of Events

Reverse Rotation of Emergency Diesel Generator

This event involved problems with a temporary modification to an emergency diesel generator. Several errors occurred during maintenance on generator EGEN1 at Rocky Flats in Building 707.¹ Maintenance personnel attempted to connect a Temporary Diesel Generator (TDG) to the emergency electrical distribution system. The intent of the temporary modification was to provide an emergency generator to respond to power outages in exactly the same way as EGEN1, while it was undergoing cooling system modifications. Work was performed in accordance with an approved work package and involved connecting the TDG to the EGEN1 bus. However, facility personnel failed to disconnect EGEN1 from the bus when they connected the TDG, resulting in a parallel arrangement of the two generators that was a violation of both the work package and the station lockout/tagout procedure and a significant hazard to personnel. Later, maintenance personnel attempted a load test that caused EGEN1 to be motorized and, due to incorrect phasing of the TDG, rotate backwards. The emergency swithgear breaker closed and immediately tripped open, terminating the load test.

To correct the deficiencies found in the initial test, facility personnel issued a field change to the work package to include electrical isolation of EGEN1, inspection of the switchgear emergency breaker, and reverification of the TDG electrical phase rotation. Electricians performed the phase rotation check which again resulted in the switchgear emergency breaker closing and immediately tripping open. Because of this failure, management suspended all building activities except those necessary to maintain safety.

After another revision of the work package, load testing was reinitiated several days later. During this test, the emergency breaker successfully connected to the emergency bus, and the TDG loaded for about an hour. Upon completion of the test, however, the emergency breaker failed to disconnect from the emergency bus when the TDG was shut down, resulting in damage to the surge protection circuit of the TDG.²

The contractor and the Office of Nuclear and Facility Safety conducted simultaneous reviews of the event to determine the sequence of events, root cause(s), and contributing cause(s). They identified the following conditions.

1. Involved personnel believed that the schedule for installation had to be met and this perception may have contributed to a lack of a thorough walkdown prior to operability testing.

2. The system engineering group did not fully understand the operations logic of the EDG 1 or TDG, and the design engineering group was not involved in preparation of the work package or troubleshooting activities until after the final retest.

3. The work package did not include separate steps or independent verification for completion of critical tasks or require documentation of results of intermediate tests (e.g., phase rotation checks).

4. The work package contained illegible and unclear wiring and logic diagrams, and the drawings directed performance of critical tasks that were not contained within the work package.

5. The work package did not include calibration instructions for TDG instrumentation and did not identify a safety-related power source for the TDG battery system.

6. The TDG operability test was the same as the EDG 1 operability test with only nomenclature changes; it did not address the subtleties of operation or provide instructions on how to reset the mechanical trip mechanism; it did not include "response-not-obtained" actions.

7. There were numerous instances of failure to follow procedures during equipment Isolation, modification, inspection, and operability testing.

8. Troubleshooters addressed only the most obvious problems and did not provide adequate corrective actions; complete walkdowns of the numerous wiring changes were not conducted until after the event.

9. Previous indications of potentially defective equipment were not reviewed for generic applicability; and earlier, the electrical breaker manufacturer examined the skid-mounted EDG 1 output breaker and noted that the mechanical linkage lubricant was dry, but no effort was made to check the status of other breakers in the emergency electrical system.

10. Abnormal diesel generator status indications were noted but not investigated prior to testing the TDG.

Uninterruptible Power Supply Damage Caused by Cooling Water System Valve Rupture

The second event occurred during post-modification testing of a temporary modification on the river water system and the service raw water system at the Savannah River K- Reactor.³ Facility personnel originally installed the temporary modification to provide cooling water to building air conditioner chillers using polyvinylchloride (PVC) piping. After initial leak testing, they uncovered an air leak, tipped the test, and repaired the leak by replacing a section of piping. They then conducted a flow test to qualify the line as a fire protection water source. Initial flow was inadequate, so they opened a supply valve on the river water system. At the same time, a 30-in. PVC gate valve used to connect the river water and service raw waters systems ruptured, causing a large amount of water to spill into the emergency pump room. Facility personnel isolated the service raw water system and roped off the spill area because of radiological concerns. During clean-up, water entered the unit and an uninterruptible power supply was damaged.

During a critique of the event, investigators identified the following conditions.

1. Workers improperly installed a pipe-to-elbow joint in a manner that did not meet procedure requirements, resulting in a weak joint that failed under pressure.

2. A pre-installation hydrostatic test that would have revealed piping weaknesses was not performed.

3. The modification did not address use of the temporary line as a fire protection water source.

4. Facility personnel did not analyze the modification for hydraulic forces caused by using the temporary line as a fire protection source.

5. The temporary modification package did not specify testing requirements or system pressure instrumentation.

6. There was no post-modification test procedure.

7. The temporary modification procedure did not require documented approval when the scope of the temporary modification changed.

Loss of Stack Ventilation Damper Control and Personnel Evacuation

The third event occurred on February 3, 1992, at the Pinallas Plant when a technician bumped into a nitrogen regulator valve that was part of a temporary modification related to ventilation damper control, causing it to close.⁴ The incident occurred during maintenance on the tritium monitoring system. The valve closure interrupted nitrogen supply to the discharge dampers for the east stack fans, allowing the dampers to close. Exhaust air flow for the tube exhaust and tritium recovery system was lost. Facility personnel restored the nitrogen supply within ten minutes of the event. There were no adverse effects as a result of the closed valve, except in one area where air flow reversed, causing positive pressure, which resulted in personnel evacuation and suspension of ongoing tube loading activities.

Investigators subsequently determined that in preparation for the planned sitewide electrical outage, the control air system for the stack fan dampers was modified. The existing configuration utilized compressed air for the dampers, but the outage would shut down the air compressors, thereby cutting off control air to the dampers. Therefore, facility personnel decided to use nitrogen in place of the control air. The modification included Installation of a temporary pressure regulator valve in the tritium recovery system monitoring room. However, after the outage, the control air for the dampers was not restored to its original configuration; and the pressure regulator valve remained in the limited work space, unprotected from equipment and personnel movement. The pressure regulator valve was not labeled as to function or purpose, and personnel working in the area were not aware of its importance.

Deficiencies contributing to this event included the following.

1. Plant procedures did not address temporary modifications to high-priority systems (e.g., design, approval, tagging requirements).

2. Facility personnel did not perform a formal safety review for the nitrogen line, intended as a temporary measure to control the tack fan dampers, which should have defined a need for control of the critical valve(s).

Unexpected Reactor Scram Signal

The fourth event occurred on August 16, 1991, at the Savannah River K-Reactor when a Very High Temperature scram signal was received in the control room.⁵ The signal originated from testing jack panel thermocouples. Investigators found that the safety computer was not bypassed as required during the jack panel test; and, as a result, a false cram signal was introduced into the computer. The scram signal was thought to have been blocked by a temporary modification; however, the work package did not specify proper control of the safety computer bypass switches. In later critique, investigators determined that the incident was caused by a deficient temporary modification procedure, which did not provide adequate instructions for initial bypass of the safety computer and also did not provide positive control to ensure that the safety computer remained bypassed. This event caused no adverse impact on environment, safety, or health because the reactor was shut down.

Breach of Primary Radiological Containment

A fifth event occurred at the River Bend commercial nuclear power plant on April 2, 1992, when personnel discovered that the containment building was breached during modification of the standby service water piping.⁶ The piping inside containment was cut while the outboard containment isolation valve was open. Core alterations were performed while this condition existed, resulting in a Technical Specification violation.

Investigators determined that error by both the releasing Senior Reactor Operator (SRO) and the Tagging Official was the root cause of the incident. The SRO did not adequately question personnel performing the temporary piping modification as to work details. The SRO also failed to list at the associated maintenance work order on the tracking list for Limiting Condition for Operation (LCO) for containment integrity during fuel handling, which was necessary to complete the communication and notify operators of the containment breach when they attempted to establish containment integrity for refueling. In addition, the Tagging Official did not adequately question workers who requested that the outboard isolation valve be opened to verify that the pipe was drained. The Tagging Official also misunderstood the location of the work and therefore failed to verify closure of the isolation valve after draining was verified.

Investigators identified the following contributing factors.

1. Operations personnel believed that a more extensive review of work packages was performed by outage management than was actually the case.

2. Operations personnel placed heavy reliance on a service water modification status board in the control room, but the board was not updated.

3. Facility personnel did not identify the potential for a containment integrity conflict during development and planning of the modification request and modification work order.

4. The applicable modification procedure did not include provisions for a post design-review checklist specifically addressing the operational impact of modifications.

Significance of Events

These events are significant because inadequate design, installation, use, and restoration of temporary modifications can result in damaged equipment and personnel injury or undetected degradation of equipment and systems essential to safety and mitigation of accidents. These events also emphasize the importance of worker compliance with procedures along with the need to stop work when procedure or work package faults become evident. Furthermore, troubleshooters should address all system anomalies and not just the most obvious deficiencies. Lessons learned from these events are also applicable to permanent modifications.

Corrective Actions

Involved personnel performed immediate corrective action and followup. They reviewed equipment and process failures and undertook remedial efforts that included restoration of damaged equipment and initiation of additional controls for modification and testing, as well as improvements in training programs.

Hazard Mitigation

Actions that can be taken to minimize damage resulting from inadequate temporary modifications include the following.^7,8,9

1. Provide increased control over the modification design process.

   • Require that all modifications, including significant temporary modifications, be prepared by a dedicated design organization.

   • Assign overall responsibility for modifications to one individual and an alternate to ensure continuity of thought and implementation.

   • Require independent technical and safety review of all modifications that include consideration of the integrated plant response to the modifications and the synergistic effect of the modifications.

   • Evaluate temporary modifications for unreviewed safety question determinations in accordance with the requirements of DOE 5480.21, Unreviewed Safety Questions.

   • Require a complete walkdown of proposed modifications as part of the modification development process.

   • Provide operators with instructions for operating temporarily modified equipment and guidance for periodic monitoring of operation through surveillance testing or calibration.

   • Use legible drawings and include all directions in the modification package rather than on drawings.

   • Require independent verification and sign-off for critical steps in work packages and procedures.

   • Provide specific criteria for quality control inspection points during implementation of modifications.

   • Provide explicit actions to be followed and identify individuals to be notified if field conditions differ from anticipated conditions in modification packages.

   • Establish criteria for the maximum number of changes that can occur prior to requiring a complete work package revision.

   • Require thorough reviews of changes to approved packages in order to ascertain the full impact of change.

   • Evaluate tasks associated with the modification to ascertain if they can be accomplished within the time constraints identified.

2. Provide increased control over the modification design process.

   • Require that all modifications, including significant temporary modifications, be prepared by a dedicated design organization.

   • Assign overall responsibility for modifications to one individual and an alternate to ensure continuity of thought and implementation.

   • Require independent technical and safety review of all modifications that include consideration of the integrated plant response to the modifications and the synergistic effect of the modifications.

   • Evaluate temporary modifications for unreviewed safety question determinations in accordance with the requirements of DOE 5480.21, Unreviewed Safety Questions.

   • Require a complete walkdown of proposed modifications as part of the modification development process.

   • Provide operators with instructions for operating temporarily modified equipment and guidance for periodic monitoring of operation through surveillance testing or calibration.

   • Use legible drawings and include all directions in the modification package rather than on drawings.

   • Require independent verification and sign-off for critical steps in work packages and procedures.

   • Provide specific criteria for quality control inspection points during implementation of modifications.

   • Provide explicit actions to be followed and identify individuals to be notified if field conditions differ from anticipated conditions in modification packages.

   • Establish criteria for the maximum number of changes that can occur prior to requiring a complete work package revision.

   • Require thorough reviews of changes to approved packages in order to ascertain the full impact of change.

   • Evaluate tasks associated with the modification to ascertain if they can be accomplished within the time constraints identified.

3. Provide increased control over the modification installation process.

   • Assign responsibility for implementation to one individual or to a dedicated team with turn-over requirements specified for those installations that require more than one shift to complete.

   • Ensure that pre-job briefings address anticipated conditions and individuals to contact in the event of problems, address procedural compliance and safety, and are provided for oncoming shifts and overtime workers.

   • Reverify lockout/tagouts associated with the modification at the start of a work day and prior to testing.

   • Ensure that communication between supporting organizations (including quality control, operations, maintenance, and radiological protection) are adequate. At a minimum, these communications should include notification of schedule changes.

   • Ensure that personnel training adequately addressees the full scope of the modification and that both carts and operations personnel received training prior to implementation of the modifications.

4. Provide increased control over the modification testing process.

   • Ensure that tests utilize nomenclature and locations specific to the associated equipment.

   • Ensure that tests are reviewed for hardware/process compatibility by the design modification organization and operations. Implement training on the testing procedure, including a review of contingency actions and specific elaboration on the differences in operational or control characteristics.

   • Conduct testing only after review and walkdowns by the lead individuals responsible for development and installation of the design change package including valve and breaker positions of interfacing systems.

   • Require documentation of intermediate test results and calibrated test equipment for future reference.

   • Test systems or components after removal of the modification installation ensuring that post- modification tests adequately verify the functionality of affected portions of the system and that quantitative acceptance criteria are established and met.

5. Strengthen the root cause analysis process.

   • Provide root cause analysis training to engineering staff, operations supervisors, and plant managers to ensure that the concepts are factored into every facet of every modification and operational process.

   • Emphasize the correction of all causes, not just the most apparent.

   • Remove time constraints from the root cause determination process.

   • Review results of equipment failures and root cause analyses for similar applications.

References

1. DOE Occurrence Report FRO--EGGR-PUFAB-1992-0061. 13 April 1992. "Temporary Diesel Generator Fails Load Test, Creating a Limiting Condition for Operations (LCO) Out-of-Tolerance Condition."

2. DOE Occurrence Report FRO--EGGR-PUBAV-1992-0066. 13 April 1992. "Failure of Load Test with Damage to the Temporary Diesel Generator."

3. DOE Occurrence Report SR--WSRC-REACK-1992-0214. 18 December 1992. "Rupture of Temporary Modification Piping and Subsequent Loss of the CWGM."

4. DOE Occurrence Report ALO-PI-GEND-PINELLAS-1992- 0006. 5 January 1993. "Stack Fan Failure."

5. DOE Occurrence Report SR--WSRC-REACK-1991-1110. 6 June 1992. "Safety Computer #1 VHT Scram (DPSOL 8463) Inadvertent."

6. Licensee Event Report 92-008-01, River Bend Station accession no. 9209140008. 1 September 1992. "Containment Integrity Not Maintained During Fuel Handling." Nuclear Regulatory Commission (NRC) Public Document Room. Washington, D.C.

7. NRC Information Notice 89-81, 6 December 1989. "Inadequate Control of Temporary Modifications to Safety- Related Systems". NRC Public Document Room. Washington, D.C.

8. NRC Information Notice 91-17. 11 March 1991. "Fire Safety of Temporary Installations or Services." NRC Public Document Room. Washington, D.C.

9. Institute of Nuclear Power Operations Good Practice TS-412 (INPO 85-016), Rev. 2 April 1992. "Temporary Modification Control." Institute of Nuclear Power Operations. Atlanta, Georgia.