Independent Verification and Self-Checking

Content

• Introduction
• Notice Summary
• Independent Verification
• Self-Checking
• Applicability
• Summary of Events
• Significance of Events
• Corrective Actions
• Hazard Mitigation
• References

Introduction

This notice is one in a series of publications issued by the Office of Nuclear and Facility Safety to share nuclear safety information throughout the Department of Energy complex. For more information, contact Dick Trevillian, Office of Operating Experience Analysis and Feedback, Office of Nuclear and Facility Safety, U.S. Department of Energy, Washington, DC 20585, telephone (301) 903-3074. No specific action or responses are required solely as a result of this notice.

Safety Notices are distributed to U.S. Department of Energy Program Offices, Field Offices, and contractors who have responsibility for the operation and maintenance of nuclear and related facilities, and to other organizations involved in nuclear safety. Written requests to be added to or deleted from the distribution of Safety Notices should be sent to: BR Richard L. Trevillian, EH-33, Room E-460 GTN, U.S. Department of Energy, Washington, DC 20585.

The ESH Office of Information Management maintains a file of Safety Notices and supporting information. Copies can be obtained by contacting the Office of Information Management at (301) 903-0449 or by writing to the Office of Information Management, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

Notice Summary

This notice is a review of lessons learned from events showing the importance of independent verification and self-checking for increasing safety of operations at Department of Energy (DOE) facilities. Failure to perform these work practices has resulted in personnel injury, shutdown of facility operations, spread of contamination, and equipment damage. These events were reported in the Occurrence Reporting and Processing System (ORPS) and the Operating Experience Weekly Summary.

Independent Verification

Independent verification is the practice of checking a given task for conformance to established criteria by a qualified person other than the one who performed the task. This practice is necessary because no matter how proficient a worker is, he can make mistakes, and it is unlikely that two workers will independently make the same mistake. Independent verification is normally separated by distance and time to insulate the verifier from the worker's performance.

Another form of verification is called concurrent dual verification, which requires a worker and a verifier to independently verify the component and then concur on the action. This practice is useful when operation and verification can not be separated by time and distance. For example, an incorrect installation of jumpers, could actuate safety systems or shut down critical equipment.

Self-Checking

Closely related to the practice of independent verification is self-checking, a risk management tool designed to reduce human error. By teaching workers to focus their attention on the details of the task at hand self-checking becomes an ingrained work practice to positively identify the correct unit, train, or component and to review the intended action and expected response before performing the task. This tool is widely used in the commercial nuclear industry.

The Hanford site uses a technique called STAR, which is an acronym for stop, think, act, and review. The technique requires a worker to (1) stop before performing the task in order to eliminate distractions and identify the correct component; (2) think about the task, the expected response, and the actions required if that response does not occur; (3) act by reconfirming the correct component and performing the intended function; and then (4) ureview/u by comparing the actual response to the expected response.

Applicability

This notice is applicable to operations and maintenance personnel at DOE-owned or -operated facilities where the conduct of operations is important. The Office of Nuclear and Facility Safety (NFS) encourages operators of these facilities to become familiar with the practices of independent verification and self- checking and to recognize the consequences of not applying them. No specific action or response is required as a result of this notice. NFS recommends processing this information in accordance with the guidance in DOE-STD- 7501-95, "Development of DOE Lessons Learned Programs."

Summary of Events

Independent Verification

The following events were selected to illustrate how independent verification could have prevented the problems.

Lockout/Tagout Errors¹

On November 11, 1994, an In-Tank Precipitation Facility manager at the Savannah River Site reported that operators had locked out the wrong valve in a gang valve. Two operators were assigned to implement a lockout/tagout order for a vent valve in a gang valve used to transfer high-level radioactive waste to other tanks or to evaporators for processing. Incorrect valve lineups and tagouts can result in equipment damage, spread of contamination, or personnel injury.

Both operators used the valve label to identify the vent isolation valve; however, the valve label was incorrect and they were not able to verify that the valve was the correct one because insulation covered the piping. Procedures required operators to verify valve locations by tracing the attached pipe to a well-defined system component such as a tank, heat exchanger, or another valve. Without this verification, one operator tagged and locked the flush isolation valve and the other operator incorrectly verified the flush valve as the vent valve. When the operators noticed the discrepancy, the facility manager stopped construction activities associated with the lockout until the discrepancy was corrected. Facility personnel corrected the labeling error, unlocked the flush valve, and locked the vent valve.

On November 12, at the Savannah River Site, engineers performing a walkdown of a new steam system following steam system repairs at the Receiving Basin for Offsite Fuels noticed that a steam isolation valve was locked in the wrong position. The operator who installed the tag did not place the valve in the correct position and a second operator incorrectly verified the steam valve to be closed. The steam system repair was performed with a reduced safety margin. If the system had been in normal operation and an upstream block valve failed, personnel injury could have occurred. Investigators determined that the operator who performed the lockout/tagout and the verifying operator were trained in techniques for independent verification and valve manipulation.

In both events, investigators determined that the direct cause was personnel error. The operators did not follow requirements for independent verification and conduct-of-operations. These events illustrate two key lessons: first, the importance of attention to detail when performing independent verifications and, second, the importance of positive equipment identification. Attention to detail is especially important when tasks may affect facility safety; and, if personnel cannot positively identify equipment, they should stop until identification is made.

Improper Review of System Configuration²

On September 22, 1992, maintenance workers at the Hanford Fast Flux Test Facility were installing a temporary pressure (vacuum) gage to monitor and control cell pressure while permanent gages were calibrated. The task was authorized by an approved work package. Workers were unaware that the temporary gage had been installed downstream of an isolation valve that was tagged closed. Operators responded to an indicated high (negative) pressure in the cell by opening a bleed valve to reduce cell vacuum. The operating pressure in the cell was + 2 inches water gage through a feed and bleed system. When the pressure indication on the temporary gage did not change in response to the bleed, operators traced the system and found the closed and tagged isolation valve.

The shift manager reviewed the work package and authorized workers to remove the tag and open the valve. With the gage operational, facility personnel determined that cell pressure had been reduced to minus 16 inches water gage. The lower operating limit for cell pressure is minus 7 inches water gage. This pressure transient had the potential to induce stress in the steel cell liner beyond its design limit. Facility personnel returned the cell pressure to normal operating range. If operators had independently verified the path of the vacuum line from the cell to the isolation valve before beginning the task, they would have seen that the temporary gage was not properly located to sense cell pressure.

Self-Checking

The following events were selected to illustrate how self-checking could have prevented these events.

Accidental Pump Start³

On September 16, 1992, at Lawrence-Berkeley Laboratory, a maintenance technician was sprayed with water as he and an electrician were replacing the motor on a lift pump. The pump was one of two used to transfer liquid waste to a neutralizing tank. Both pumps were mounted on top of a tank in a concrete pit. While the men were working on the first pump, an Environment, Safety, and Health representative asked for a sample from the tank. The technician disconnected the pipe union above the second pump to take the sample.

In the meantime, the electrician had been checking the overload heaters on the first pump and he returned to verify that the motor on the pump was rotating properly. He asked the technician to watch the motor rotation when he started the first pump. However he started the second pump instead and the liquid in the pipe sprayed through the disconnected joint onto the technician. The electrician pulled a disconnect handle and stopped the spray. He called medical personnel who found that the technician had no chemical burns. Fortunately, operators had drained the tank and flushed it with water before the incident.

If the electrician had verified that the pump was the correct one before he started it, the event would not have occurred.

Operation of Wrong Circuit Breaker⁴

On September 29, 1992, ventilation in a controlled radiological area at the Hanford Plutonium Finishing Plant was lost when an electrician opened a supply breaker. He was preparing to restore power to a panel when he noticed people near the panel; and, as a safety precaution, he left the circuit breaker and asked them to stand clear of the panel while he energized it. Upon returning to the breaker location, the electrician placed his hand on the wrong breaker and operated it to the open rather than the closed position, causing loss of power to the ventilation fans. The backup ventilation exhaust turbine started as designed and maintained a negative pressure in the system. Everyone was evacuated from the affected areas until radiological technicians surveyed the areas and restored normal ventilation.

Electrician Contacted Live Line⁵

On October 16, 1992, an electrician received serious burns when he contacted a live 12 kilo- volt line in a switch box assembly at the Stanford Linear Accelerator Facility. Although an approved work permit was correctly executed and the required lockout/tagout was completed, the line remained energized because the fuses the electrician intended to remove were on the hot side, of the power supply disconnect. A sign had been placed on the switch panel calling attention to the fact that power flowed upward through the switch, that is, through the fuses and then through the disconnect switch. The lockout/tagout did not protect the electrician as intended.

When the electrician attempted to remove the fuses from the switch box, he believed he was providing an additional measure of safety with a back-up open circuit. If he had read the sign on the switch panel before pulling the fuses, he would have known that the fuses were part of the energized circuit.

This incident highlights the importance of constant vigilance and use of personnel protection equipment when working with electrical devices. Workers should be trained to test circuits rather than assuming they are de- energized. Safety and operating procedures should be reviewed to eliminate ambiguities and misinterpretations of instructions.

Maintenance on Wrong Equipment⁶

On November 11, 1992, Paducah Gaseous Diffusion Plant maintenance mechanics discovered that they had performed maintenance on the wrong equipment. The mechanics were preparing to replace seals in two compressor stages inside a U-shaped area containing a number of process cells. The work permits clearly identified the cell and stages to be worked on. The mechanics placed their tools and equipment near a cell in the U-shaped area opposite the cell with the identified stage and left the area. Health physics personnel arrived after the mechanics left and set up a radiation work boundary around the cell where the tools and equipment were placed. The mechanics returned later and proceeded to work in the cell that health physics personnel had marked off. After removing one of the seals, the mechanics noticed that the identification number did not match the number on the seal maintenance record card and realized they were working on the wrong compressor stage.

This incident occurred because the mechanics failed to check the cell number against the number identified in the work permit before starting the job. Investigators also raised questions about the way procedures prescribe coordination between health physics and maintenance personnel. Health physics personnel should have verified the cell against the work permit before setting up the boundary.

A facility manager's review of the occurrence indicated that the incident was the result of poor communications and a lack of attention to detail. Sufficient information was provided in the procedure for each person involved in the task to identify the exact work location. Training sessions were conducted to emphasize the practice of self-checking.

Significance of Events

These events are significant because of dangers involved in the operation of complex facilities where hazardous materials are handled. Even with automated controls, hands-on maintenance of the physical plant and the software must be performed. Training, experience, and dedication of operators and maintenance personnel are mandatory. Nevertheless, workers are vulnerable to distraction and complacency, as well as emotional and physical stresses that can affect both judgment and performance. In most of the events, the work controls (work package, permit, briefings) were not sufficient to prevent error. The events illustrate the need for independent verification and self-checking to provide the last defensive barrier to error. These work practices greatly increase safe operation and protect the health and safety of operating personnel and the public.

Corrective Actions

Both DOE and the Nuclear Regulatory Commission (NRC) have tracked occurrences in their areas of responsibility. Evaluation of these events generated recommendations and programs aimed at reducing the frequency of personnel errors. Training programs can be revised to emphasize independent verification and self-checking techniques. The commercial nuclear power industry has also supported the philosophy of independent verification with the development and publication of an example program. In June, 1993, DOE issued a standard for good practices of independent verification.⁷ This standard should be considered when planning or reviewing independent verification programs at DOE facilities. Chapter 10 of DOE 5480.19 provides administrative guidance for independent verification.⁸

Hazard Mitigation

Emphasis on independent verification and self- checking by management and supervisory personnel can reduce conduct-of-operations errors. Success of the program depends on the realization that all personnel, regardless of training, educational level, and responsibility, are vulnerable to lapses of memory or loss of concentration. Such effects may be more intense during periods of high activity when personnel can be overwhelmed with information, or during periods when there is little activity to stimulate the mind, or much repetition. Emotional stress can also degrade an operator's or supervisor's concentration and judgment and affect performance of routine tasks. Training personnel to self-check planned actions and consider the consequences of each action before execution can be a large factor in reducing operator and supervisor errors. When coupled with verification of planned actions by an independent, qualified individual, the likelihood of personnel errors is significantly reduced.

References

1. Operating Experience Weekly Summary 94-46, "Faulty Independent Verification Practices Lead to Lockout/Tagout Errors," ORPS Report SR--WSRC-ITP- 1994-0034 and SR--WSRC-RBOF- 1994-0018, November 14, 1994. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

2. Operating Experience Weekly Summary 92-23, "Failure to Properly Review System Configuration," ORPS Report RL--WHC-FFTF-1992-0015, October 1, 1992. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

3. Operating Experience Weekly Summary 92-25, "Accidental Pump Start," ORPS Report SAN--LBL-OPERATIONS-1992- 0008, October 15, 1992. EH Information Center, U.S. Department of Energy, EH- 72/Suite 100, CXXI/3, Washington, DC 20585.

4. DOE Final Occurrence Report RL-- WHC-PFP-1992-0031, "The Wrong Breaker Was Actuated, Due To Inadvertent Deviation From A Procedure, Resulting In The Loss Of Normal Ventilation," January 12, 1993. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

5. Operating Experience Weekly Summary 92-26, "Electrician Contacted Live 12 KV Line," ORPS Report SAN--SU-SLAC- 1992-0028, October 22, 1992. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

6. Operating Experience Weekly Summary 92-30, "Maintenance Performed on Wrong Equipment Due to Personnel Error," ORPS Report ORO--MMES- PGDPGENPLT-1992-0002, November 19, 1992. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

7. DOE-STD-1036-93, Guide to Good Practices for Independent Verification, June 1993. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

8. DOE 5480.19, Conduct of Operations Requirements for DOE Facilities, July 9, 1990. EH Information Center, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.