September 10, 2007
The Honorable Thomas P. D'Agostino
National Nuclear Security Administration
Dear Mr. D'Agostino:
The Defense Nuclear Facilities Safety Board (Board) has completed a review of nuclear criticality safety (NCS) implementation, conduct of operations, and configuration management in the Plutonium Facility vault at Los Alamos National Laboratory. The enclosed report prepared by the Board's staff provides a detailed discussion of the results of this review. This letter focuses on one of the issues identified during the review that deals with the improper utilization of the laboratory's Materials Accountability and Safeguards System (MASS) software to ensure compliance with criticality safety limits.
The laboratory uses MASS to track material movements and special nuclear material storage throughout the Plutonium Facility, including movements into and out of the vault. MASS often provides the sole source of information about a given item, such as mass and isotopic composition. Despite the laboratory's contention that MASS is merely an operator aid and is not relied upon to determine the safety of material movements, discussion with operators and review of procedures revealed that MASS fulfills critical safety-related functions, such as determining compliance with NCS limits. MASS was initially conceived as a security-related accountability program and accordingly was not designed to perform a criticality safety-related function. As a result, its criticality safety aspects were not subjected to the rigorous quality assurance requirements and configuration management that its current safety-related usage would require.
MASS was developed by the laboratory in the 1970s in a programming language that is difficult to maintain and has not seen a significant software upgrade since its introduction. Consequently, the laboratory has long-standing plans to transition MASS to a modern computing platform and programming language. Even though the need for a MASS update was identified more than 10 years ago, upgrade efforts have been unsuccessful. The laboratory recently revived the effort to modernize and upgrade MASS, but progress is not yet evident. In addition, it is unclear whether the current concept for a new MASS acknowledges its safety-related function and therefore identifies the appropriate software quality assurance and configuration management requirements.
Therefore, pursuant to
Therefore, pursuant to42 U.S.C. § 2286b(d), the Board requests a report within 90 days of receipt of this letter outlining the overall strategy and key milestones for the MASS upgrade. Included in this strategy should be:
c: Mr. Glenn S. Podonsky
Mr. Donald L. Winchell, Jr.
Mr. Mark B. Whitaker, Jr.
DEFENSE NUCLEAR FACILITIES SAFETY BOARD
Staff Issue Report
August 6, 2007
MEMORANDUM FOR: J. Kent Fortenberry
FROM: C. Goff and E. Elliott
SUBJECT: Materials Accountability and Safeguards System Software at
This report documents a review by the staff of the Defense Nuclear Facilities Safety Board (Board) of nuclear criticality safety (NCS) implementation in the Plutonium Facility (PF-4) vault at Los Alamos National Laboratory. The review included a site visit on June 5-7, 2007, as well as a follow-up teleconference held on July 19, 2007. Members of the Board's staff B. Broderick, R. T. Davis, E. Elliott, C. Goff, C. Keilers, M. Moury, and J. Plaue participated in the review.
Materials Accountability and Safeguards System Software. The laboratory tracks material movements and special nuclear material (SNM) storage throughout PF-4 using the software program referred to as the Materials Accountability and Safeguards System (MASS). MASS often provides the sole source of information about a given item, including mass, isotopic composition, material form, and storage location. MASS is also used to generate the material label that is affixed to the outside of each SNM container. Before SNM is physically transferred to a new location in the facility, the transfer must be completed electronically in MASS by the operators. In some cases, the criticality limits for a given location (e.g., glovebox or vault) are loaded into MASS.
Reliance on MASS for Safety - According to laboratory management, MASS is regarded as an operator aid that provides information for criticality control purposes, but is not relied upon to perform a criticality safety function. This belief is consistent with the description of MASS in the administrative procedure that governs NCS implementation in PF-4 (TA55-AP-522) which states that MASS is "a useful administrative tool to assist nuclear material handlers in managing fissile material inventories within approved limits." The NCS group also informed the Board's staff that operators use MASS only to obtain data; compliance with NCS limits is to be determined by hand calculation while referencing the approved values contained in the NCS posting and associated Criticality Safety Limit Approval (CSLA).
The assertion that MASS is only an operator aid is not consistent with the laboratory's implementing procedures. In some situations, MASS automatically provides both NCS limit information and performs the calculation to verify compliance with the limits. In TA55-AP- 522-the procedure that describes MASS as an "administrative tool'-there are procedural steps detailing the movement of items into a glovebox that state, "Before physically moving the fissile material, perform the MASS transaction to check that the amount and form of the material being transferred is within the criticality safety limit for the destination glovebox." Hand calculations are not mentioned in the procedure or the associated training. Additionally, a Technical Area-55 Notice (TA55-Notice-011) sent to all employees on October 3 1, 2006, instructs operators to execute the electronic transfer in MASS before physically moving the material into the location, with the express purpose of preventing an overmass condition. The notice uses the following lessons-learned statement to explain the new requirement: "The practice was to physically move items, then enter the information into MASS. This means that computer material accountability checks were not performed until the item was in the new location. This permitted human error, which can result in exceedance of criticality safety limits.” In both the procedure and the notice, the laboratory is instructing operators to use MASS to comply with NCS limits.
After speaking with vault operators, it was clear to the Board's staff that MASS was also being relied upon in the vault for compliance with criticality safety limits. For example, some vault storage locations have NCS limits that are based on the total mass and allow multiple SNM containers. MASS keeps a running total of the mass in each storage location, and will output a warning message that informs the operators if an NCS limit will be exceeded when they electronically process an SNM transfer. According to the vault operator procedure (NMT4-WI- 101) if MASS generates a warning that a criticality limit will be exceeded, the operators are instructed to consult the CSLA to check the NCS limit, after MASS has alerted them. The operators also consult a handwritten Vault Availability Book (the Blue Book), which is used primarily to ensure that the item will physically fit into the storage location. The Blue Book also provides a form to indicate the location's running mass total, which could serve as a second check of the total mass; however, it does not contain the NCS limits. In this case, MASS is relied upon to prevent an "overmass" condition, which means that MASS is performing a safety function exceeding that of an "operator aid."
Safety Software Requirements-MASS was originally developed in the 1970s as a security-related accountability program, and thus it lacks the configuration management control and software quality assurance pedigree of a safety-related system. Department of Energy Order 414.1 C, Quality Assurance, requires that software performing safety functions meet certain requirements, including appropriate configuration management of key data, verification and validation of software operation, and documentation. Currently, MASS does not meet many of these requirements. As an example, changes to NCS limit data in MASS that should require configuration management control are handled informally via email and are not verified or audited by the NCS group.
Planned Improvements-As mentioned previously, MASS was developed more than 20 years ago, and was written in a programming language that is difficult to maintain; it has not undergone a software upgrade since its inception. The original author of the code, now retired, is the only person capable of editing the source code and is still relied upon to make improvements and correct problems, which included three instances of system inoperability during the past year.
Because of the age of the code and the limited ability to update it, the laboratory began efforts to transition MASS to a modern computing platform and programming language more than 10 years ago; however, these efforts were unsuccessful. A March 2006 report by the laboratory's Audits and Assessments office found that the MASS upgrade project lacked "full management support, project leadership, defined scope, prepared schedule and budget, and quality assurance plan for a successful functional and timely implementation." This finding was repeated from two previous assessments, one performed in July 2004 and the other in March 2002.
Recently, the laboratory revived the effort to modernize and upgrade MASS. In particular, laboratory management formally: (1) consolidated responsibility for the system, which historically has been a significant hindrance to progress and (2) committed to providing the necessary resources for the upgrade. During the teleconference with the Board's staff on July 19, 2007, the laboratory further indicated that early planning had begun for a renewed effort that is expected to be mature in early fall 2007. No project manager had been assigned to the project, and substantive planning incorporating lessons learned from previous upgrade attempts was not evident. It was unclear whether the concept for a new MASS acknowledges its safety-related function and therefore identifies the appropriate software quality assurance and configuration management requirements. The effort to upgrade and modernize MASS would appear to be an excellent opportunity to implement these requirements formally. A representative from the Los Alamos Site Office (LASO) expressed the expectation that the managerial rigor of the MASS update effort would be commensurate with that of other major design and construction projects, but did not explicitly acknowledge MASS as safety-related software.
Observations on Vault Operations. The following observations made by the Board's staff have been informally communicated to the laboratory for its use.
NCS Involvement in Operational Changes-The staff was briefed on findings from an internal assessment by the PF-4 Deputy Facilities Operations Director. This assessment identified that the NCS group was not always involved when an operation underwent a change. This lack of NCS involvement in the change process has resulted in ongoing operations that are not in compliance with NCS postings. The NCS group is required by ISD 130- 1 .O, Nuclear Criticality Safety Program Manual, to evaluate new or revised operations involving significant quantities of fissionable material. However, this requirement is weakly implemented at the present time, a situation that the improved Unreviewed Safety Question process should rectify by requiring NCS concurrence on proposed changes to fissile material operations. In the interim, the NCS group issued a letter stating that NCS approval is needed for any changes to fissionable material operations.
Adequacy of Vault Operator Staffing-Vault operations require two trained vault operators. Currently, there are only two trained vault operators, with a third in training who is approximately 1 year away from full qualification. Because of the limited number of qualified personnel, the vault is typically accessible to users only during the morning and must close if one of the two vault operators is absent from work. Overall, the laboratory has not developed adequate staffing plans to safely support the expected increase in programmatic mission, as well as ongoing nuclear material stabilization and repackaging activities, including actions to implement the Board's Recommendations 94- 1, Improved Schedule for Remediation in the Defense Nuclear Facilities Complex, and 2000- 1, Prioritization for Stabilizing Nuclear Materials.
Priority of NCS Reevaluation for the Vault - Based on earlier walkdowns conducted as part of the NCS Program Improvement Plan (PIP), vault operations were categorized as a medium-risk activity, and NCS evaluations governing the vault were deemed to have only "administrative deficiencies." The Board's staff examined the vault documentation, which consists of a number of convoluted NCS evaluations that rely heavily on expert judgment. In some cases, the evaluations contradict one another. In at least one case, controls identified as necessary in the evaluations (i.e., use of neutron-absorbing material) are not being implemented appropriately. These deficiencies in the NCS evaluations for the vault are unlikely to be addressed in the near term, because the NCS group is currently focused on evaluating and documenting other operations deemed to be of higher risk during NCS PIP walkdowns. Additionally, bringing the vault NCS evaluations into compliance with DOE Standard 3007, Guidelines for Preparing Criticality Safety Evaluations at Department of Energy Nonreactor Nuclear Facilities, will require substantial staff resources. According to the current schedule, the vault evaluations will not be rewritten until the third quarter of fiscal year 2009. The Board's staff believes the existing vault evaluations contain technical deficiencies and that their rewriting should be accorded higher priority, especially given the large number of vault movements and the fact that the vault is nearing capacity.
Weaknesses in the Conduct of Operations - The staff observed a movement of SNM into the vault. Overall, the vault staff and material handlers appeared to be highly knowledgeable; however, the Board's staff did observe weaknesses in the conduct of operations. For example, the vault operators informed the staff that the local copy of the procedure that governs the use of MASS was 2 years out of date and did not accurately reflect some of their current practices. While an updated procedure was available in the facility's document control system, it was not being utilized. The Board's staff emphasized the need for the correct version of the procedure to be present in the workspace. Subsequently, the local (and out-of-date) copy of the procedure was replaced.
of Federal NCS Oversight. Currently, LASO has one individual performing NCS oversight,
with augmentation and monitoring provided by personnel from the National Nuclear
Security Administration (NNSA)