October 24, 2011

Gentlemen:

Reference: Central and Eastern United States Seismic Source Characterization for Nuclear Facilities Project: Participatory Peer Review Panel Final Report

Introduction

This letter constitutes the final report of the PPRP1 (“the Panel”) for the Central and Eastern United States Seismic Source Characterization for Nuclear Facilities Project (the “CEUS-SSC Project” or “the Project”). The eight Panel members (Jon P. Ake, Walter J. Arabasz, William J. Hinze, Annie M. Kammerer, Jeffrey K. Kimball, Donald P. Moore, Mark D. Petersen, J. Carl Stepp) participated in the Project in a manner fully consistent with the SSHAC Guidance.2 The Panel was actively engaged in all phases and activities of the Project’s implementation, including final development of the Project Plan and planning of the evaluation and integration activities, which are the core of the SSHAC assessment process.

The Panel's involvement, described more fully later in this letter, also included review of analyses performed by the Project to support the evaluation and integration processes, review of interim evaluation and integration products, and review of the interim draft project report and the final project report. Additionally, panel members participated in specific analyses as resource experts, and panel members were observers in or participated as resource experts in eight of the eleven Technical Integrator Team (TI Team) working meetings held to implement the integration phase of the assessment process.  We want to express our appreciation for the opportunity to participate in the CEUS-SSC Project in this way.  

In the remainder of this letter we provide our observations and conclusions on key elements of the project implementation process, and we summarize our reviews of the draft and final project reports. As we explain in our comments, assurance that the center, body, and range of the technically-defensible interpretations ("CBR of the TDI")3 have been properly represented in the CEUS-SSC Model fundamentally comes from implementing the structure and rigor of the SSHAC Guidance itself.  We are aware that the SSHAC Guidance is accepted by the Nuclear Regulatory Commission and the Department of Energy for developing seismic hazard models that provide reasonable assurance, consistent with the seismic safety decision-making practices of these agencies, of compliance with their seismic safety policies and regulatory requirements.  For these reasons, we describe aspects of the SSHAC Guidance to provide context for our observations and conclusions.

Project Plan: Conformity to the SSHAC Assessment Process 

The SSHAC Guidance recognizes that observed data, available methods, models, and interpretations all contain uncertainties.  These uncertainties lead to alternative scientific analyses and interpretations. In other words, experts in the broad technical community do not hold a single interpretation.  Accepting this scientific situation, the SSHAC assessment process is designed to engage the scientific community in an orderly assessment of relevant data, methods, models, and interpretations that constitute current scientific knowledge as the basis for development of a seismic hazard model that represents the CBR of the TDI.  

The assessment process is carried out by means of two main activities: evaluation and integration.4 In implementation, the evaluation activities are structured to inform the integration activities. The evaluations are carried out by means of workshops in which the TI Team engages proponents of alternative interpretations that represent the range of relevant current community knowledge. Resource experts in the various relevant data sets are also engaged. The workshops have the dual purposes of, first, evaluating the degree to which alternative interpretations are supported by observed data and, second, defining uncertainties in the degree to which the interpretations are defensible, given the observed data.  Integration is carried out by individual evaluator experts or evaluator expert teams (Level 4 process) or by a Technical Integrator (TI) Team (Level 3 process) who, informed by the evaluation activities, characterize the range of defensible alternative interpretations in an integrated hazard model and assess the scientific uncertainty distribution. Based on our review of the Project Plan and our subsequent discussions with the Project Team, we concurred that the Plan conformed with the SSHAC Guidance, incorporating lessons learned from fourteen years experience using the Guidance, and that the planned implementation was structured to properly carry out the SSHAC assessment process for development of the CEUS-SSC Model.

SSHAC Level 3 Assessment Process 

The SSHAC Guidance describes implementation processes for four levels of assessment depending on the scientific complexity of the assessment and the intended use of the assessed hazard model.  For an assessment such as the regional SSC model for the Central and Eastern United States, which will be used at many sites for making safety and licensing decisions for nuclear facilities, the SSHAC Guidance recommends using an assessment Level 3 or Level 4.  

There are process differences between a Level 3 and Level 4 implementation, but the objective is the same: to obtain from multiple proponent experts information that supports an informed assessment of the range of existent relevant interpretations and associated uncertainties that together represent current community knowledge and to perform an informed assessment of the CBR of the TDI. We understand that within the SSHAC assessment process "technically defensible" means that observed data are sufficient to support evaluation of the interpretation and the corresponding uncertainty.

In a Level 4 assessment process a TI Team facilitates the assessment, identifying and engaging proponent and resource experts, performing supporting analyses, and conducting knowledge evaluation workshops and assessment integration working meetings.  Multiple experts or teams of experts perform as evaluators of the range of existent interpretations and as integrators of the hazard model.  The individual evaluator experts or evaluator expert teams take ownership of their individual or team assessments.  In a Level 3 assessment all of these activities are consolidated under a single TI Team consisting of a TI Lead, multiple evaluator experts representing the scope of required scientific expertise, and experienced data and hazard analysts.  

As we noted earlier in this report, assurance that the CBR of the TDI is properly represented in a hazard model comes from rigorously implementing the SSHAC assessment process itself.  We note that an important lesson learned from multiple implementations of the SSHAC Guidance over the past fourteen years is that the Level 3 and Level 4 assessment processes provide comparably high assurance that the relevant scientific knowledge and the community uncertainty distribution are properly assessed and represented in the hazard model.  The Level 3 assessment is significantly more integrated and cohesive and is more efficient to implement.  These considerations led us to endorse use of the Level 3 assessment for implementation of the CEUS-SSC Project in our Workshop No. 1 review letter.  During the course of the Project we observed that the higher level of cohesiveness inherent in the Level 3 assessment process leads to significantly improved communication, facilitating the experts' performance of their technical work.

Overall Project Organization

A complex project with multiple sponsors such as the CEUS-SSC Project cannot be successful unless it is well organized and energetically managed so that the various participants understand the interconnectedness of their activities and perform their technical work as a cohesive group.  In this regard the adopted project management structure allowed the Project Manager to provide integrated overall project leadership, manage the database development activities, and effectively maintain communication with the PPRP and project sponsors while allowing TI Team lead to concentrate on the structural and technical activities of the assessment as the Project unfolded.  We conclude that the project organization was effective overall and particularly so with regard to facilitating the TI Team's implementation of the assessment process.

Implementing the SSHAC Level 3 Assessment Process

Irrespective of the level of implementation, evaluation and integration are the main activities of a SSHAC assessment.  The evaluation activities aim to identify and evaluate all relevant available data, models, methods, and scientific interpretations as well as uncertainties associated with each of them.  The integration activities, informed by the evaluations, aim to represent the CBR of the TDI in a fully integrated SSC model.   

Evaluation

Consistent with the SSHAC Guidance the evaluation phase of the CEUS-SSC project accomplished a comprehensive evaluation of the data, models, methods, and scientific interpretations existent in the larger technical community that are relevant to the SSC model.  In significant part the process was carried out in three structured workshops, each focusing on accomplishing a specific step in the evaluation process.  

The first workshop (WS-1) focused on evaluations of relevant geological, geophysical, and seismological data sets (including data quality and uncertainties) and on identification of hazard-significant data and hazard-significant SSC assessment issues.  It became clear that a number of issues relating to the earthquake catalog, the paleoliquefaction data set, the potential-field geophysical data, updating procedures for assessing maximum earthquake magnitude, and development of procedures for assessing earthquake recurrence would require focused analyses.  These analyses were appropriately carried out within the TI Team working interactively with appropriate resource experts recognized by the larger scientific and technical community. 

WS-2 focused on evaluations of the range of alternative scientific interpretations, methods, and models within the larger scientific community and on corresponding uncertainties. WS-3 focused on evaluations of hazard feedback derived at seven representative test locations using a preliminary CEUS-SSC model. Specifically, the workshop focused on the identification of the key issues of most significance to completing the SSC model assessment. 

Experience has shown that evaluations to gain understanding of the quality of various data sets and uncertainties associated with them are essential for fully informing an SSC assessment.  We observed that in WS-1 resource experts for the various data sets did a high-quality job of describing the data sets and giving their perspective about the data quality and associated uncertainties. We conclude that the understanding of data quality and uncertainties gained in WS-1 together with continued interactions between the TI Team and data resource experts significantly informed the TI Team's evaluations.  The TI Team's evaluations of the data quality and uncertainties are well documented in the innovative "Data Summary Tables" and "Data Evaluation Tables" included in the Project Report.  Importantly, the TI Team continued to effectively engage data resource experts in productive analyses of potential-field geophysical data, the earthquake catalog, development of the paleoearthquake data set (including an integrated assessment of the paleoliquefaction data in order to extend the earthquake catalog), the development of methods for assessing maximum earthquakes, and the development of earthquake recurrence analyses.  All of these focused analyses strongly informed the assessment process. Moreover, documentation of the analyses resulted in stand-alone products of the Project that will serve future users of the CEUS-SSC Model.

The compilation and evaluation of potentially relevant methods, models, and alternative scientific interpretations representing the community knowledge and corresponding uncertainties must be considered the core process activity of any SSHAC assessment.  This step was largely carried out in WS-2.  Success in defining the community knowledge depends on fully engaging proponent experts representing the range of methods, models, and interpretations existent at the time.  Full engagement means that the proponent experts completely and clearly describe their interpretations and the data that support them and provide their individual evaluations of corresponding uncertainties. We observed that the actions taken by the Project and TI Team to explain the workshop goals and to guide participants toward meeting those goals was very productive. We conclude that the workshop was highly successful in meeting the stated goals and that it fully met the expectation of the SSHAC Guidance with respect to evaluating the range of alternative scientific interpretations. The discussions during the workshop and between the TI Team and Panel following the workshop evolved the "SSC Framework" concept, which provided transparent criteria that framed the TI Team's systematic identification and assessment of seismic sources throughout the CEUS. 

Feedback from hazard calculations and sensitivity analyses is an important step in a SSHAC assessment to understand the importance of elements of the model and inform the final assessments.  For development of a regional SSC model to be used for site-specific probabilistic seismic hazard analyses (PSHAs) at many geographically distributed sites, feedback based on the preliminary model is particularly important.  Following WS-2 a preliminary SSC model termed "the SSC sensitivity model," was developed and used for hazard sensitivity calculations that were evaluated in WS-3.  While the SSC sensitivity model was clearly preliminary, the evaluation of sensitivity results that took place in WS-3 provided important feedback for completing analyses and for supporting the TI Team's development of the preliminary CEUS-SSC model.  The Panel was able to review the preliminary model and provide feedback in a subsequent project briefing meeting on March 24, 2010.

Together the three workshops provided the TI Team interactions with the appropriate range of resource and proponent experts. These experts were carefully identified to present, discuss, and debate the data, models, and methods that together form the basis for assuring that the CBR of the TDI have been properly represented in the hazard model.  Experts representing academia, government, and private industry participated.  The TI Team also reached out to a wide range of experts as they developed the database and performed the integration activities to develop the SSC model.  The Panel participated throughout this process, and is satisfied that the TI Team fully engaged appropriate experts to accomplish the goals of a SSHAC Guidance.

Integration

Consistent with the SSHAC Guidance, integration is the process of assessing the CBR of the TDI and representing the assessment in the SSC model.  Informed by the evaluation process, the integration process includes representation of the range of defensible methods, models, and interpretations of the larger technical community together with new models and methods developed by analyses during the evaluation and integration process.

For the CEUS-SSC Project, development of the earthquake catalog, methods for assessing and representing maximum earthquake magnitudes, and methods for earthquake recurrence assessment continued during the integration process.  The Panel reviewed all the analyses at various stages of development and provided comments and recommendations. The TI Team performed the integration process by means of eleven working meetings.  Members of the Panel participated in most of these working meetings as observers or resource experts.  The full Panel participated in the discussions during both feedback meetings and provided formal comments and recommendations following the meetings.  We observed that the integration process was thorough and that it acceptably complied with the SSHAC Guidance.  Based on our participation and observations we conclude that the integrated CEUS-SSC Model appropriately represents the center, body, and range of current methods, models and technically defensible interpretations.

PPRP Engagement

Consistent with the SSHAC Guidance, the Panel was fully engaged in peer-review interactions with the TI Team and the Project Manager of the CEUS-SSC Project throughout the entire project period-from development of the Project Plan in early to mid 2008 through production of the Final Project Report in mid to late 2011.5 The Panel provided both written and oral peer-review comments on both technical and process aspects at many stages of the Project's evolution. Key PPRP activities, leading up to this final report, have included:

• Review of the Project Plan.

• Formulation of a PPRP implementation plan, specifically for the CEUS-SSC Project, to ensure adherence to the general guidance provided by SSHAC and NUREG-1563 for the scope and goals of a PPRP review.

• Involvement in each of the three Project workshops, including advising in the planning stage; participating collectively as a review panel during the workshop (and individually as resource experts when requested by the TI Team), providing timely comments on technical and process issues; and submitting a written report of the Panel's observations and recommendations following each workshop.

• Development and implementation of a process, together with the TI Team, to document the resolution of recommendations made in PPRP formal communications.

• Participation as observers (and occasionally as resource experts when requested by the TI Team) in eight of the TI Team's 11 working meetings.

• Peer-review and written comments, including several informal reports, on the TI Team's intermediate work products, particularly early versions of the CEUS-SSC Model.

• Direct interaction with the TI Team and Project Manager in more than 20 teleconferences and four face-to-face briefings-in addition to the three workshops and eight working meetings of the TI Team noted above.

• Extensive, critical peer-review of the Project's 2010 Draft Report and 2011 Final Report. 

The Panel, collectively and individually, fully understood the SSHAC Guidance for a structured participatory peer review and the requirements for a Level 3 assessment process; had full and frequent access to information and interacted extensively with the TI Team and Project Manager throughout the entire project; provided peer-review comments at numerous stages; and, as documented within the Final Project Report, was fully engaged to meet its peer-review obligations in an effective way.

Project Report

The SSHAC Guidance makes clear that adequate documentation of process and results is crucial for their understanding and use by others in the technical community, by later analysis teams, and by the project sponsors. The Panel understood what was needed to conform to the SSHAC requirements, and it was committed to ensuring that the documentation of technical details associated with the CEUS-SSC Model in the Project Report was clear and complete.  The Panel was equally committed to ensuring the transparency of process aspects of the project, both in implementation and in description in the Project Report.

The Panel provided lengthy compilations of review comments (see Appendix I of the Project Report) for both the 2010 Draft Report and the 2011 Final Report.  These included hundreds of comments, categorized as general, specific, relating to clarity and completeness, or editorial.  The massive amount of detail provided by the TI Team in the Project Report and the intensiveness of the Panel's review comments both reflect great diligence and a mutual understanding by the TI Team and the PPRP of the thoroughness and high quality of documentation expected in the Project Report. See Review Comments

The Project Manager and the TI Lead provided review criteria to the Panel for both the draft and final versions of the Project Report. The criteria for reviewing the Draft Report6 covered the range of technical and process issues consistent with requirements of the SSHAC Guidance, including draft implementation guidance (see footnote #4).  Key criteria, among others, include sufficiency of explanatory detail; adequate consideration of the full range of data, models, and methods-and the views of the larger technical community; adequate justification of the data evaluation process, logic-tree weights, and other technical decisions; proper treatment of uncertainties; and conformance to a SSHAC Level 3 assessment process.  To be clear, the PPRP is charged with judging the adequacy of the documented justification for the CEUS-SSC Model and its associated logic-tree weights.  The TI Team "owns" the Model and logic-tree weights.

Criteria for reviewing the Final Report focused on reaching closure to comments made on the Draft Report and ensuring that no substantive issues remained unresolved.  To that end, among its many review comments on the Final Report the Panel identified "mandatory" comments, which the TI Team was required to address in the final version of the Project Report.

The Panel made thorough, extensive efforts in its documented reviews of the 2010 Draft Report and the 2011 Final Reports (as well as in many related interactions with the TI Team) to ensure a high-quality Project Report that fully meets SSHAC requirements for clear, complete, and transparent documentation of all aspects of the CEUS-SSC Project. We are pleased to confirm that implementation of the CEUS-SSC Project fully conformed with the SSHAC Guidance and that the resulting CEUS-SSC Model properly meets the SSHAC goal of representing the center, body, and range of technically-defensible interpretations.

This concludes our Final Report for the CEUS-SSC project.

Copy:

Lawrence A. Salomone
Kevin J. Coppersmith
Brent Gutierrez

¹ Participatory Peer Review Panel

² Budnitz, R. J., G. Apostolakis, D. M. Boore, L. S. Cluff, K. L. Coppersmith, C. A. Cornell, and P. A. Morris, 1997. Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and the Use of Experts (known as the “Senior Seismic Hazard Analysis Committee Report,” or the “SSHAC Guidance”). NUREG/CR-6372, U. S. Nuclear Regulatory Commission. TIC; 235076. Washington, DC.

³ See Section 2.1 in the CEUS-SSC Final Report for discussion of concepts relating to the center, body, and range of the "technically-defensible interpretations" vs. the center, body, and range of the "informed technical community."

⁴ For an excellent discussion of this two-stage process, see Practical Implementation Guidelines for SSHAC Level 3 and 4 Hazard Studies, USNRC NUREG-XXXX, Draft for Review, Office of Nuclear Regulatory Research, May 2011.  

⁵ See CEUS-SSC Final Report: Section 2.5, Table 2.2-1, and Appendix I

⁶ See PPRP report dated October 4, 2010, in Appendix I of CEUS-SSC Final Report