Decision Analysis Techniques

Content

• Introduction
• Notice Summary
• Applicability
• Methods
• Examples
• Final Evaluation
• References

NOTE: This safety notice contains extensive charts, tables and figures, which cannot be formatted for this electronic version. Please refer to the hard copy of this Safety Notice.

Introduction

This notice is one in a series of publications issued by the Office of Nuclear and Facility Safety to share nuclear safety information throughout the Department of Energy complex. For more information, contact Dick Trevillian, Office of Operating Experience Analysis and Feedback, Office of Nuclear and Facility Safety, U.S. Department of Energy, Washington, DC 20585, telephone (301) 903-3074. No specific action or responses are required solely as a result of this notice.

Safety Notices are distributed to U.S. Department of Energy Program Offices, Field Offices, and contractors who have responsibility for the operation and maintenance of nuclear and related facilities, and to other organizations involved in nuclear safety. Written requests to be added to or deleted from the distribution of Safety Notices should be sent to: BR Richard L. Trevillian, EH-33, Room E-460 GTN, U.S. Department of Energy, Washington, DC 20585.

The ESH Office of Information Management maintains a file of Safety Notices and supporting information. Copies can be obtained by contacting the Office of Information Management at (301) 903-0449 or by writing to the Office of Information Management, U.S. Department of Energy, EH-72/Suite 100, CXXI/3, Washington, DC 20585.

Notice Summary

Office of Nuclear and Facility Safety (NFS) engineers have reviewed some methods of analyzing the effect of management decisions on safety and included discussions of two techniques in this notice. One technique is to use statistics to compare safety at different facilities. The second is a method of analyzing individual occurrences to determine their safety significance.

Applicability

This notice is applicable to the conduct of operations at facilities owned or operated by the Department of Energy (DOE). NFS advises operators of these facilities to become familiar with the techniques of decision analysis and apply them to improve facility safety. No specific action or response is required as a result of this notice. NFS recommends processing this notice in accordance with DOE-STD- 7501-95, "Development of DOE Lessons Learned Programs."

Methods

The statistical methods discussed in this notice can be used with data gathered from a variety of sources. However, the primary data source available to DOE users is the Occurrence Reporting and Processing System (ORPS). ORPS has information in both coded and text fields. By using narrative searches on the text fields, the set of occurrence reports can be narrowed to only those applicable in a particular case. Distribution reports can then show the number of times each possible code occurs in the nature of occurrence, direct cause, contributing cause, and root cause fields. However, caution must be used in extracting and using this data. Care must be taken in the initial narrative search to include all possible names for a type of event.

The possibility of including ORPS reports in the sample that are the opposite to that intended must be evaluated. For instance, when searching on the word "dose," a report that states: "no personnel received a measurable dose" would be included, but it should be removed upon review. Also, codes entered in many of the coded fields are subjective, and not every person entering ORPS reports would categorize a given event in the same way. Also, some users may not use multiple codes for fields where they are allowed, such as nature of occurrence. These types of problems may make facilities that are actually similar appear to be different.

The use of statistics to compare dissimilar facilities requires a method of normalizing the data. Normalization is the process of putting data from dissimilar sources on the same basis. For instance, even if two facilities are equally safe, the number of occurrences depends on the amount of activity at the site. The best way to normalize data is to divide the number of occurrences of a particular type by the amount of work done that could result in such an occurrence. However, data on the amount of work in different categories is usually unavailable, and other normalization techniques must be used. The following bases for normalization of occurrence data are listed in order of decreasing utility.

• hours worked

• labor cost

• total cost

• number of people at site

• total occurrences

In many cases, the only way to normalize the data is to compare the number of occurrences in several different categories at several different facilities to the totals for each category and facility using the chi-squared test¹.

Chi-squared results can be used to determine the statistical significance of differences in the distribution of occurrences at various facilities. The chi-squared test can be used to determine whether two samples could have been taken from the same population. Each sample is characterized by the number of events or items falling into each of several "bins." Chi- squared measures how close the fractions in each bin are in different samples. For example, chi-squared can be used to determine whether two samples of marbles were taken from one well-mixed barrel or from two different barrels by comparing the number of marbles of each color found in each sample. Because the chi-squared test does not depend on a particular distribution, it is applicable to all types of data.

The chi-squared test may be used with data from the ORPS applying the codes for nature of occurrence, direct cause, contributing cause, and root cause. Comparisons may be made of two or more similar facilities, of part of a facility to the entire facility, or at the same facility during different time periods. Facilities may be compared in individual categories using the cumulative binomial distribution¹. Low values indicate that the facility has less than the expected number of occurrences, while high values indicate excessive occurrences. The first two cases discussed in "Examples" are (1) a comparison of three similar facilities and (2) a comparison of part of a facility to the whole facility.

If normalization by the number of hours worked, cost, or number of people at the site is possible, the beta distribution¹ may be used to compare the facilities. Analysis of normalized data is discussed in "Examples".

In all cases, the final result of the analysis is the probability that the null hypothesis is true. The null hypothesis is that the facilities are similar in nature. The smaller the probability, the more confidence there is in the result. A probability of 0.05 (95 percent confidence level) is commonly used to report statistical significance. This means that, on the average, one out of every twenty statistical tests conducted on data from the same population will appear to be significant at the 95 percent confidence level.

NOTE
Significance does not imply certainty. Also, significance testing is only good for rejecting a hypothesis, not accepting one.

Statistical tests for individual categories are similar, but they are two-tailed tests in which statistical significance of both larger-than-expected and smaller-than-expected numbers of occurrences are considered. In order to have 95 percent confidence, probability limits of 0.025 and 0.975 must be set.

Details for using these statistical methods is not given here, but they are covered in any introductory text on statistical methods or data analysis.

Methods are also available for assessing the significance of individual occurrences. One such method is to assess the number and quality of barriers to injury or significant release. If the preliminary assessment indicates that both the frequency and consequence associated with the event are at or above the medium level, a more complete assessment of the significance is necessary. The frequency of the major consequence is estimated using event tree methodology, and the severity of the consequence is estimated using an appropriate physical model, the nature of which will depend on the type of event.

Several techniques that are valuable in determining the frequencies of initiating events and the probabilities of branches on event trees are not discussed in this notice. These include Bayesian methods, common-cause analysis, component reliability estimation, human error estimation, and system importance. These topics will be addressed in future notices.

Examples

Example 1

Direct cause codes of hoisting and rigging occurrences at three facilities are compared to determine if the differences among them are statistically significant.

Table 1 contains data from the ORPS database for occurrences at the three facilities. The first two columns contain the direct cause code and description. The next three columns contain the number of occurrences per direct cause code at each facility plus facility totals. The sixth column shows the total for each direct cause code and the grand total in the bottom row. The next three columns show the expected values if all the facilities were similar. The underlying assumption is that, if the facilities are similar, the proportion of occurrences characterized with a given direct cause code will be about the same at each facility.

Table 2 shows the chi-squared test of the data in Table 1. The number of degrees of freedom, shown in the first column, is the number of values in the table that can be chosen arbitrarily while leaving the row and column totals unchanged. The first row shows the chi-squared value in the second column for a comparison confidence level shown in the third column. The comparison chi-squared value is evaluated at the comparison significance level and number of degrees of freedom. The second row shows the computed chi-squared value in the second column. The associated confidence level, computed from this value and the number of degrees of freedom, is in the third column. The computed chi-squared value indicates that the results are in the expected range with the significance less than the 95 percent level.

Table 3 illustrates the use of the cumulative binomial distribution to estimate the significance of each individual observation. One individual result indicates significantly more procedure problems (code 2) than expected at facility A. However, about 5 percent of the results are expected to be significant at this level, so the facility A result may be a statistical anomaly.

Example 2

Direct cause codes from one facility are compared to those from the entire DOE complex to determine if the facility is representative of the whole complex.

Table 4 contains data from the ORPS database for all occurrences at the selected facility and the DOE complex. The first two columns contain the direct cause codes and occurrence descriptions. The next two columns contain the number of occurrences in the complex and at the facility with totals in the bottom row. The next column shows the expected values if the facility is representative of the complex. The fifth column shows the contribution of each row to the total chi-squared value. The last column uses the cumulative binomial distribution to estimate the significance of each individual observation.

Table 5 shows the chi-squared test results of the data in Table 4. The number of degrees of freedom is shown in the first column. The second row shows the chi-squared value in the second column for a comparison confidence level shown in the third column. The third row shows the computed chi- squared value in the second column. The associated confidence level is in the third column.

The chi-squared value for this case has a significance of 100 percent, which means that the facility is not at all representative of the complex. From the last column of Table 4, the facility has many more occurrences than expected in the equipment/material problems category (code 1), but many fewer occurrences than expected in the procedure problem (code 2), design problem (code 4), management problem (code 6), and external phenomenon (code 7) categories.

Example 3

Direct cause codes from three facilities were compared to determine whether the differences among them are statistically significant. The number of workers at each facility was used to normalize the data.

Table 6 contains data from the ORPS database for hoisting and rigging occurrences at three different facilities. The first two columns contain direct cause codes and descriptions. The next three columns contain the number of occurrences in each direct cause category at each facility. The total number of employees at each facility is shown in the bottom row. Total occurrences for each direct cause code and the grand total of employees are shown in the last column.

Table 7 shows the expected values and the chi-squared computation for each direct cause code. Because the total number of events at each facility is not used to normalize the data, the number of degrees of freedom is the number of values in the table that can be chosen arbitrarily, leaving the row totals unchanged.

Table 8 shows the chi-squared test results of the data in Table 7. The comparison chi-squared value is evaluated at the comparison significance level and number of degrees of freedom. The chi-squared value indicates that the results are in the expected range with the significance less than the 95 percent level.

In table 9, occurrence rates of each direct cause category at each facility are shown as the number of occurrences per 100,000 employees. This is a standard way of reporting accident statistics² . The significance of each category is determined by using the beta distribution (sometimes known as the incomplete beta function).

Overall, the three facilities are similar at the 95-percent significance level. Two of the results are significantly higher than expected: equipment/material problems (code 1) at facility C and procedure problems (code 2) at facility A. One result is significantly lower: equipment/material problems (code 1) at facility A.

Example 4

An event tree is used to determine the safety significance of a glovebox fire on November 22, 1994, at Los Alamos National Laboratory in the Plutonium Processing Facility.^3,4

Workers placed a can with exterior surface corrosion in a glovebox, and cotton rags in the can were removed and laid out to dry. The rags began to smolder, filling the glovebox with smoke and prompting an employee to actuate a fire alarm. The rags caught fire and burned to completion without damaging any other material or equipment. There was no spread of radioactivity as a result of the event.

The first step in the analysis was a semi-quantitative ranking of the operating event based on two factors⁵: (1) the estimated maximum consequence of a similar event and (2) the conditional probability of that consequence. The classification grid for this event is shown in figure 1.

The fire was caused by enough Pu-238 to generate about 10 watts of heat (which ignited a rag weighing 113 grams) and generated about 2 megajoules of heat. The quantity of Pu-238 required to generate 10 watts is radiologically significant; however, the fraction that could be inhaled by a worker was small, so the analysts judged the consequence to be medium. According to the classification scheme, events classified as "medium/medium" are subject to further analysis.

The conditional probability was based on the degree of residual protection measured by the number of barriers remaining against fatality, injury, or other undesirable result. In this case, there were three barriers: (1) detection of the fire and intervention to contain it, (2) glovebox containment, and (3) negative pressure in the glovebox provided by its ventilation system. Considering the barriers, the analysts characterized the conditional probability of a significant release as medium.

The event tree in figure 2 was used to estimate the frequency of release, given that a glovebox fire had occurred. The conditional probability of a significant release was estimated to be 2.5 E 10-3.

The inhalation dose was estimated to be 9.6 rem using: where

• D = the dose in rem,
• I = the Pu-238 inventory in the rags (18 g),
• ARF = the airborne release fraction⁶ (6E10-3),
• GRF = the glovebox release fraction (0.1),
• RF = the respirable fraction of the particles (0.01),
• SA =the specific activity of Pu-238 (17.2 Ci/g),
• BR = the breathing rate (3.5E10-4 m3/s),
• T = the duration of the worker's exposure (10 s),
• CEDE = the committed effective dose equivalent of Pu-238 (460 rem/mCi),
• V = the free volume of the room (311 m³).

The assumption that the particles were evenly distributed in the room is used in the model. Variations in the distribution of the particles might double this dose to 19.2 rem in the worst case. Combining the conditional probability with the worst-case dose gives an average dose of 19.2 rem ( 0.0025 or 0.045 rem/event.

The fatal cancer risk coefficient for the worker was taken to be 4 E 10-4 fatality/rem⁷.

The analysts computed the fatality risk as 2 E 10-5 fatality/event. Figure 3 illustrates the fatality risk from the event as compared to the OSHA threshold for significant risk to workers for exposure to benzene (1 E 10-3 fatality) and the average lifetime accident fatality risk in other U. S. industries (4 E 10-3 fatality).

The event could have been prevented by placing the rags in an inert glovebox, and future glovebox fires can be avoided by a procedure change. Inert gloveboxes are already available at the facility; thus, the cost of avoiding similar events is low (Figure 4). The benefit of using inert gloveboxes is larger than the cost, so they should be used to store rags that may be contaminated with Pu-238.

Risk-based analyses such as these may be a valuable decision analysis method when used with other information and techniques available to managers. Users of the event-tree method should include all available information to establish the events and the split fractions (the numerical values on the branches) in the event tree.

Plant personnel with knowledge of the equipment and procedures are particularly important sources for necessary information. Users should also examine common-cause relationships between events. For instance, inadequate maintenance can affect both the glovebox containment and the ventilation function. Finally, users should be aware of uncertainties associated with resulting values from these analyses. If the uncertainties are too large, an uncertainty analysis should be performed to quantify the magnitudes of the uncertainties.

Final Evaluation

Procedures described in the examples provide complementary methods for assessing the significance of operational events. Statistical methods provide a way of determining types of occurrences that may require further attention at particular facilities. Event tree methodology provides a way to determine the significance of a particular event by considering the possibility that a similar event could have more severe consequences, evaluating the probability and severity of these consequences, and comparing the resulting risk to known hazards.

References

1. Abramowitz, Milton, and Stegun, Irene, Handbook of Mathematical Functions, National Bureau of Standards, Applied Mathematics Series 55, pp. 940-945, 960, June 1964.

2. Statistical Abstracts of the United States 1994, U. S. Department of Commerce, pp. 96-100.

3. Operating Experience Weekly Summary 94-48, "Thermal Decomposition of Plutonium-238 at Los Alamos National Laboratory."

4. DOE Occurrence Report ALO-LA-LANL-TA55-1994-0037, "Burning Rags in Glovebox," November 22, 1994.

5. Neogy, P. Brookhaven National Laboratory, Private Communication to A. R. Marchese, March 20, 1995, DOE (301) 903-2712.

6. DOE/DP-70056-H1, U. S. DOE Defense Programs Safety Survey Report, Volume 1, November, 1993.

7. Annals of the ICRP, "1990 Recommendations of the International Commission of Radiological Protection," Volume 21, nos. 1 - 3, 1991.