This chapter describes the SSHAC Level 3 assessment process, how it was implemented to assess the CEUS-SSC model, and how that implementation was accomplished in compliance with the SSHAC guidance. 

The “SSHAC assessment process,” which differs only slightly for Level 3 and 4 studies, is a technical process accepted in the NRC’s seismic regulatory guidance (Regulatory Guide 1.208) for reasonably ensuring that uncertainties in data and scientific knowledge have been properly represented in seismic design ground motions consistent with the requirements of the seismic regulation 10 CFR Part 100.23. Therefore, the goal of the SSHAC assessment process is the proper and complete representation of knowledge and uncertainties in the SSC and GMC inputs to the PSHA (or similar hazard analysis). This reasonable representation of knowledge and uncertainties is referred to in the SSHAC guidance as “the center, the body, and the range of the informed technical community.” The SSHAC assessment process, if properly implemented, provides high levels of confidence that the SSHAC goal has been met. Therefore, the way it is conducted is important and subject to “process” as well as “technical” peer view. A key responsibility of the Participatory Peer Review Panel (PPRP) is to ensure that the SSHAC assessment process has been properly implemented.

SSHAC developed guidance for four “study levels” of implementing an assessment that depend on the degree of uncertainty and contention involved and on the intended use of the seismic hazard model. SSHAC recommended that a Level 3 or Level 4 assessment process be used for complex assessments, the products of which have high public importance and attract public scrutiny, such as regional seismic hazard models intended to be used over a sustained time period as base-case models for site-specific PSHAs. Such models require the highest level of assurance that the community uncertainty distribution has been properly represented. For the CEUS-SSC Project, the decision to use a SSHAC Level 3 assessment process was based on experience with implementing the SSHAC guidance, which has shown that a properly executed Level 3 assessment process can provide a level of assurance of meeting the SSHAC goals comparable to that of Level 4, which is more costly and time-consuming to implement (selection discussed in the Project Plan and Section 1.2.1).

Discussion of the SSHAC process in this chapter comes from four sources:

1. The SSHAC document itself (Budnitz et al., 1997).
2. A summary of workshops conducted to identify lessons learned from the implementation of SSHAC in actual projects (Hanks et al., 2009).
3. A summary of the ongoing efforts of the NRC to develop more specific SSHAC guidelines (Coppersmith et al., 2010).
4. Draft NRC guidance for the implementation of SSHAC Level 3 and 4 projects (NRC, 2011).

This chapter begins with a discussion of the fundamental SSHAC goals and activities that make up a SSHAC assessment process. This is followed by a discussion of the SSHAC Level 3 assessment process implemented by the CEUS-SSC Project, including the roles of key participants, project organization, key activities, and the PPRP. The final section summarizes how the CEUS-SSC assessment process compares with the process prescribed in the SSHAC guidelines.

2.1 Goals and Activities of a SSHAC Assessment Process

Any PSHA requires that both knowledge and uncertainties be assessed and incorporated into the analysis. The SSHAC guidance expresses that fundamental goal in this way:

Regardless of the scale of the PSHA study, the goal remains the same: to represent the center, the body, and the range of technical interpretations that the larger informed technical community would have if they were to conduct the study. (Budnitz et al., 1997, p. 21)

An important part of the definition is the term “informed,” which is defined by SSHAC:

Regardless of the level of the study, the goal in the various approaches is the same: to provide a representation of the informed scientific community’s view of the important components and issues and, finally, the seismic hazard. (“Informed” in this sense assumes, hypothetically perhaps, that the community of experts were provided with the same data and level of interaction as that of the evaluators). (Budnitz et al., 1997, p. 26)

Thus there are two aspects of what constitutes an “informed” member of the technical community; the individual (1) is assumed to have knowledge of the project-specific and other relevant data, and (2) is assumed to have gone through the same interactive process that the evaluator experts have gone through in the project. Such an interactive process involves multiple workshops, structured interactions with proponents of alternative viewpoints to reveal the technical basis for various hypotheses, and feedback cycles to understand the implications of all technical assessments and associated uncertainties. The Technical Integration (TI) Team carries the responsibility of representing the center, body, and range of the views of the informed technical community.

“The center, the body, and the range” is taken to mean an appropriate representation of knowledge and uncertainty in the important components to a hazard assessment and is referred to by the SSHAC as “the community distribution.” A proper representation of the community distribution as defined in SSHAC appropriately meets the requirements of the NRC’s seismic regulation, 10 CFR 100.23.

After a review of multiple SSHAC projects and lessons learned, the NRC (2011) revisited the terminology associated with the SSHAC goals and proposed alternative wording:

The key statement in the SSHAC guidelines, that encapsulates the ethos of the SSHAC approach, is as follows: “Regardless of the scale of the PSHA, study the goal remains the same: to represent the center, the body, and the range that the larger informed technical community would have if they were to conduct the study” (NUREG/CR-6372). For brevity, the “center, body, and range of the informed technical community” is denoted CBR of the ITC. A key word in the concept is “informed,” which is specifically defined in the SSHAC guidelines to mean an expert who has full access to the complete database developed for a project, and has fully participated in the interactive SSHAC process. In other words, the selected experts who participate in the PSHA study must endeavor to represent “the larger informed technical community” by assuming the hypothetical case where the others in the larger technical community become “informed” through participation in the same process. The SSHAC guidelines recognize that this is a hypothetical exercise, but the goal would be to ensure that a broad range of views are considered. In practice, however, the term “informed” is often either ignored or misinterpreted as simply meaning expert in the field of interest. Thus, the process of capturing or representing the CBR of the ITC has been viewed by some as a process of somehow conducting a poll or surveying the larger community for their opinions.

In the spirit of maintaining the fundamental SSHAC objective and clarifying the concept with terms that reflect actual practice, an alternative statement of the fundamental objective of the SSHAC process is presented in this report. This alternate description explains that the objective of the SSHAC guidance is actually achieved through a two-stage process of evaluation followed by integration. Therefore, consistent with the original intent of the SSHAC guidance, we recast the goals of the SSHAC process in terms of the two main activities (i.e., evaluation and integration) by the following statement:

The fundamental goal of a SSHAC process is to carry out properly and document completely the activities of evaluation and integration, defined as:

Evaluation: The consideration of the complete set of data, models, and methods proposed by the larger technical community that are relevant to the hazard analysis.

Integration: Representing the center, body, and range of technically defensible interpretations in light of the evaluation process (i.e., informed by the assessment of existing data, models, and methods).

In light of these definitions, we propose that it is clearer to refer to the CBR of the “technically defensible interpretations” (TDI), instead of CBR of the ITC. However, it is important to emphasize that the careful evaluation of the larger technical community’s viewpoints remains a vital part of the SSHAC process. We simply have removed the term “informed” because of its specialized definition in the original SSHAC guidelines. Similarly, we propose to replace the term “community distribution” that is used frequently in the original SSHAC guidelines to describe the outcome from a SSHAC assessment process with the term “integrated distribution.” This is to remove any perception that the final assessments and models were arrived at through a mere poll of the community.” (NRC, 2011)

As discussed extensively in the SSHAC report (Budnitz et al., 1997) and affirmed in NRC (2011), a SSHAC assessment process consists of two important sequential activities, which, for a Level 3 assessment, are conducted by the TI Team under the leadership of the TI Lead: evaluation and integration. Each activity is discussed below related to the particular CEUS-SSC model-development activities that are entailed.

2.1.1 Evaluation

The TI Team evaluates relevant data, models, and methods that pertain to SSC inputs to a hazard analysis. The activities associated with evaluation are as follows:

• Identify hazard-significant issues. Hazard calculations and sensitivity analyses are performed at the beginning of the project to help steer the data compilation toward hazard-significant issues. After a preliminary model is developed, hazard calculations are done again to evaluate hazard sensitivity for feedback.
  

• Identify and compile project-specific data in a database. The project database is a fundamental tool with which the TI Team makes its evaluations. Workshop #1 helps in the data identification process as resource experts are assembled, and contacts with the larger technical community are made outside the workshop throughout the evaluation process.

• Collect new data. If resources allow, new data may be gathered that address particular SSC
  issues.
  

• Conduct and document the data evaluation process. A comprehensive review of pertinent data is conducted to identify their relevance to SSC (Data Summary tables) and to evaluate the data relative to their use in the SSC model (Data Evaluation tables). The data evaluation process continues throughout the integration process.
  

• Evaluate alternative data, models, and methods that exist within the technical community.
  The purpose of this evaluation is to gain a clear understanding of the data, models, and methods that have been proposed in the community, including their technical bases, strengths, weaknesses, and uncertainties. Facilitated discussions among proponent and resource experts can help with this in Workshop #2 and other communications outside the workshop. It is important to focus the discussions on the specific issues of importance to SSC.

2.1.2 Integration

Integration is model-building by the TI Team to arrive at a defensible expression of knowledge and uncertainty in inputs to SSC. This includes the full expression of the model elements (logic tree branches), their relative weights, and the range of credible uncertainties. The activities associated with integration are as follows:

• Understand expert assessment issues: The TI Team must understand the tools and issues associated with quantifying epistemic uncertainty and aleatory variability. It must also understand-and counter-common expert assessment issues (anchoring, availability, and other cognitive biases).
  

• Develop SSC models: The TI Team must identify technically defensible conceptual models and parameter values and include them in the SSC logic tree. Weights are assigned that reflect the degree of support for the models and parameter values in the available data and current technical understanding. The TI Team can develop new and innovative models to explain the available data, and it can develop new methods for analyzing the data and building the models, as long as the methods are consistent with the goal of expressing knowledge and uncertainties about the key issues. This activity is done at least twice: once for a preliminary SSC model and again for the final SSC model. For the CEUS-SSC Project, model-building occurred four times: for the SSC Sensitivity model, SSC Preliminary model, Draft SSC model, and Final SSC model.

• Perform hazard sensitivity calculations and collect feedback: These are run based on the preliminary SSC model developed by the TI Team in order to identify the most significant model elements and the importance of the uncertainties to the hazard results. These are provided as feedback and were discussed at Workshop #3. Hazard calculations and sensitivity analyses are also conducted, based on the final SSC model, to provide additional understanding of the model components and associated uncertainties.

• Document the bases for the assessments: The TI Team is responsible for documenting activities so that the reader can understand the basis for the model elements and the expressions of uncertainty made (weights on tree branches, parameter distributions, etc.). New data gathered for the project, along with new models and methods developed by the TI Team, impose the burden of high levels of documentation in order for reviewers of the report to understand their technical basis and application.

As noted previously, the evaluation and integration process is sequential during the SSHAC assessment process. During the evaluation phase, the applicable data are compiled and evaluated and the views of the technical community-expressed by proponent and resource experts-are duly considered. During the integration phase, models are developed by the TI Team as part of the evaluation process. Integration does not entail a poll or vote of the views of the larger technical community. It is model-building by the TI Team that is informed by its careful evaluation of all applicable data, its knowledge of the views of the community on certain issues, its discussions in workshops, and its direct communication with members of the community. The TI Team constructs an integrated model, usually expressed as a logic tree, which reflects knowledge and uncertainties in models and parameter values.

2.2 Roles of CEUS-SSC Project Participants

The roles that various participants play in a SSHAC assessment process are important and are defined specifically in the SSHAC guidelines (Budnitz et al., 1997). The CEUS-SSC Project was conducted in accordance with SSHAC guidelines for Level 3 projects, which explicitly define the roles of project participants who contribute to a PSHA project. Beginning with the review of the Project Plan at the Kick-Off Meeting on May 8, 2008, all project participants were informed of their expected roles before their participation, and they were reminded of their roles at the beginning of each workshop, at working meetings, and at other opportunities throughout the project. Table 2.2-1 identifies the meetings that were conducted during the course of the project, including the participants and meeting dates. SSHAC descriptions of the Project Sponsor; Project Manager; Technical Integrator; resource, proponent, and evaluator experts; and participatory peer reviewers are described below. NRC (2011, Section 3.6) provides additional discussion of these roles and responsibilities in a review of projects conducted using the SSHAC guidelines. Organization for the CEUS-SSC Project, including its structure and lines of communication within that structure, is explained in Section 2.3.

The Project Sponsor is the entity that provides financial support for a project and “owns” the results of the study in the sense of property ownership. The CEUS-SSC Project has three sponsors: the NRC, DOE, and EPRI. The Project Manager is defined as the individual responsible for maintaining project scope, budgets, and schedules and coordinating communications among the project participants and the Project Sponsor(s). The Technical Integrator is defined as a single entity-e.g., an individual or team-that is responsible for conducting the evaluation and integration processes. As discussed in Section 2.1, a Level 3 assessment includes evaluation and integration by a TI Team under the technical leadership of the TI Lead.

Three types of experts having distinctive roles are identified in a SSHAC assessment process: resource experts, proponent experts, and evaluator experts. A resource expert is a technical expert with specialized knowledge of a particular data set, model, or method of importance to the hazard analysis. The expertise may be in the form of site-specific experience, or knowledge of particular methodologies or procedures. A number of resource experts participated in Workshop #1 and summarized their data sets. In addition, a number of resource experts were contacted outside the workshop environment to provide their data and expertise (Table 2.2-2). A proponent expert is an expert who advocates a particular hypothesis or technical position. At Workshop #2, several proponent experts presented their tectonic hypotheses to the TI Team and debated the merits of their models. The workshop also provided the opportunity for the TI Team to question the proponent experts regarding the technical support and uncertainties associated with their models. An evaluator expert is an expert who can evaluate the relative credibility of multiple alternative hypotheses to explain a given set of observations. Each evaluator expert uses professional judgment to quantify uncertainties, based on review and evaluation of all potential hypotheses and available data. An evaluator may challenge a proponent’s position and question the technical basis for conclusions as a means of gaining insight into the uncertainties.

The members of the CEUS-SSC TI Team were charged with fulfilling the roles of evaluator experts. At the outset of their participation on the project, the Team members were instructed in working meetings, and later reminded at workshops, that their role as evaluator experts would entail an objective evaluation and integration process, as described in Section 2.1 of this report. The need for removal of a member who would not assume the proper evaluator expert role was described, as was the process that would be followed by the TI Lead to carry out the removal, should it be necessary. The TI Lead is responsible for ensuring that all TI Team members know their roles as evaluators and that they maintain those roles throughout the course of the project.

Peer review is considered a key aspect of the Level 3 assessment process. This is to ensure that the process followed is adequate, uncertainties are properly considered and incorporated into the analysis, and the results provide a reasonable representation of the diversity of views of the technical community. Technical peer review is the review of the earth sciences aspects of a study, including a review to ensure that all applicable technical hypotheses have been considered. A review of how the study is structured and executed is referred to as a process peer review. Two different methods for peer review are described in SSHAC. Participatory peer review is defined as an ongoing or continuous process that provides the peer reviewers with full and frequent access throughout the entire project, in contrast to a late-stage peer review that occurs when a project has almost been completed. The principal benefit of a participatory peer review is that if problems are discovered, the opportunity exists for a mid-course correction without the need for work to be substantially redone at the end. SSHAC strongly recommends the use of a participatory peer review for both technical and process reviews for projects in which a Level 3 approach is used. Accordingly, a participatory peer review process was used on the CEUS-SSC Project.

2.3 CEUS-SSC Project Organization

The project organization is shown on Figure 2.3-1, and the functions are summarized below. 

Project Sponsors: The CEUS-SSC Project was jointly sponsored by the DOE, NRC, and utilities and vendors under the auspices of the EPRI Advanced Nuclear Technology (ANT) program, Action Plan Committee (APC). The joint sponsorship of the study by both public and private sector representatives is unique for regional seismic hazard assessments in the United States. It signifies the recognition by the multiple parties that they have common needs-a fully defensible seismic source model that can be used for nuclear facility sites throughout the CEUS-and common goals of seismic hazard inputs that are stable and long-lived. Sponsor representatives were present at all workshops and key project meetings.

Project Management: Project management responsibilities were divided between those related to contract management; technical communication with sponsors, the TI Lead, and the PPRP; and those related to scope, budget, and schedule. EPRI assumed responsibility for contract management and provided the fundamental interface for contracts. These responsibilities included contracting with sponsors and CEUS-SSC Project participants, providing support for workshops, and establishing requirements for the project report and website. The Project Manager was responsible for developing the project plan; communicating with the sponsors, TI Lead, and the PPRP; and developing project tools for maintaining project scope, budgets, and schedules. The lines of communication for the project are shown on Figure 2.3-2. Jeffrey F. Hamel, EPRI ANT Program, communicated directly with the CEUS-SSC Project Manager, Lawrence A. Salomone. Mr. Salomone, who established the industry-government partnership for the CEUS-SSC study, was the principal interface with the TI Lead, the PPRP, and the project sponsor representatives (Figure 2.3-2). He assisted EPRI management in establishing and maintaining project budgets and schedules and preparing status reports, and he had primary responsibility for the delivery of all technical products. He was the principal spokesperson to the outside community, which included the DOE, NRC, USGS, and industry. 

Participatory Peer Review Panel (PPRP): Members of the PPRP were responsible for reviewing both the technical and process aspects of the CEUS-SSC Project. They were observers at the majority of the technical meetings held during the course of the project (see Table 2.2-1). They attended all project workshops and provided feedback and written comments after each workshop. They attended 8 of the 11 TI Team working meetings to observe the process and progress of the project. They also attended three PPRP briefings to review in depth the technical assessments being made by the TI Team at key points during the study. A fourth PPRP briefing, the closure briefing, was held to bring closure to the entire project review process. Throughout the project, the PPRP provided verbal and written comments that assisted the TI Team in carrying out its assessments. PPRP responsibilities included reviewing and providing written comments on the Project Plan and reviewing both the Draft Project Report and the Final Project Report developed by the TI Team.

Technical Integration (TI) Team: The TI Team, led by Kevin J. Coppersmith, had primary responsibility for developing and documenting the technical basis for all project assessments and products, as described in Section 2.4. The 12-member TI Team was responsible for implementing the SSHAC Level 3 methodology throughout the project, including all key assessment steps of evaluation and integration. Such steps include working with the Project Manager to develop the Project Plan, developing the project database, conducting three workshops, facilitating the requisite expert interactions, conducting 11 working meetings, communicating with the PPRP, and documenting all process and technical aspects of the study in a project report. Members of the TI Team and the Project Manager wrote the project report.

Database Manager: The Database Manager was responsible for retrieving and compiling applicable data for use in developing the SSC model. These data sets were provided in the formats appropriate for use in the TI Team’s deliberations. The Database Manager provided support for resolving copyright issues, working meetings, workshops, and PPRP briefings, as needed.

Technical Support: The TI Team was assisted in a number of areas by several individuals. The technical support team provided support to the hazard calculations, interpretations of the paleoliquefaction database, compilation of the geophysical databases, assistance with graphics and GIS, and development of workshop summaries and the project report.

Resource Experts, Proponent Experts, Specialty Contractors, and Other Project Participants: A large number of representatives of the larger technical community participated in the project as resource experts, proponent experts, and specialty contractors. Steps were taken by the TI Team, as supported by the PPRP, to ensure that the participation of resource experts and proponent experts in Workshops #1 and #2 was appropriate and complete in order to be representative of the range of current scientific community interpretations, for which awareness and knowledge were required. The PPRP reviewed the list of resource experts and proponents selected for Workshop 1 and Workshop 2, respectively. Specialty contractors were engaged on the project to provide certain technical products, including geophysical maps, stress interpretations, and guidance for the assessment of paleoliquefaction. Personnel from the USGS played an extended role in this project to ensure that all supportable interpretations of the scientific community were fully identified, evaluated, and represented in the SSC model. Several USGS personnel provided detailed review and feedback on specific issues (e.g., the earthquake catalog, Mmax, and methods), which were considered in the assessment of the SSC model by the TI Team.

Technical knowledge and experience on specific topics of discussion were provided at the workshops by resource experts and proponent experts. The workshops provided an important opportunity for the TI Team evaluators to gain knowledge regarding specific databases in Workshop #1 and to question and challenge the findings of the proponent experts in Workshop #2. Table 2.2-2 provides a list of the resource experts who gave presentations at Workshop #1 and the proponent experts who participated in Workshop #2. Throughout the project, a number of technical experts provided their insights, data, and viewpoints at the request of members of the TI Team. These individuals are listed in Table 2.2-2. Their participation was invaluable in keeping the TI Team abreast of current data, models, and methods and for providing a basis for assessing the technical bases and uncertainties associated with recent and ongoing studies in the technical community.

2.4 Key Tasks and Activities

As outlined at the beginning of the project in the Project Plan, the CEUS-SSC Project was structured around a set of tasks and activities that would fulfill the requirements of a SSHAC Level 3 project (Coppersmith et al., 2010). The key tasks and activities that define the CEUS-SSC Project are described in this section. The components of a typical SSHAC Level 3 or 4 project and their interactions are illustrated on Figure 2.4-1.

2.4.1 Database Development

A fundamental resource developed as part of the CEUS-SSC Project is the project database. The database mainly provides information for the use of the TI Team in its evaluation and integration processes. Most of the database consists of publications from the professional literature, maps, and similar documents. To respond to project needs, many of the maps in the database were entered into a GIS format. All documents were entered in a format that allowed them to be displayed, geo-registered, and superposed for the consideration of the TI Team in working meetings. A summary of the project database is given in Appendix A, and a description of the data is provided in the metadata files, which provide a means of searching by data type.

Although the major data compilation effort occurred early in the project, the project database continued to be developed throughout the course of the project. Identification of the data sets that populate the database began at project initiation, based on the SSC experience of the TI Team members. More data sets were identified in Workshop #1 (Significant Issues and Data). Resource experts who participated in the workshop presented their own specific data and, after the workshop, they provided lists of recommended references for consideration by the project. Throughout the course of the project, members of the TI Team communicated with a large number of researchers in the technical community and continued to identify data that were in the process of being developed and could be included in the project database. To supplement the existing data, certain new data were compiled for use by the project, including gravity maps, magnetic anomaly maps, an update to the U.S. stress map, and a compilation of paleoliquefaction data. In addition, a new earthquake catalog was developed using existing catalogs and new magnitude conversions and other updates (see Chapter 3). No new data were collected (e.g., field geologic investigations, geophysical surveys) as part of the CEUS-SSC Project.

The database is considered a deliverable of the project, and it has been placed in a format that will allow it to be used by researchers in the future via a dedicated website www.ceus-ssc.com .

2.4.2 Identification of Significant Issues

SSC for purposes of a PSHA is a specialized activity, and the technical issues within SSC that are important to seismic hazard are a subset of the larger range of issues that define seismologic, geologic, and tectonic interpretations. To provide a focus on the data, models, and methods of greatest importance to the hazard, sensitivity calculations were conducted. The experience by the TI Team gained from past seismic hazard analyses was also considered in the identification of hazard-significant issues. Workshop #1 was partially devoted to the identification and discussion of the technical SSC issues of greatest significance to a PSHA conducted for purposes of the design and review of nuclear facilities. The goal in that discussion was to focus on the data that would be most useful in defining the SSC model at the annual frequencies of interest (e.g., 10–3 to 10–7/yr) for nuclear facilities. For example, data were identified that would be important to constraining maximum magnitudes, paleoseismic recurrence estimates, and the recurrence rate of larger-magnitude earthquakes. Likewise, information was identified that could be used to quantify the uncertainties in these assessments, such as the characteristics of global SCR earthquakes, age estimates of paleoseismic earthquakes, and uncertainties in earthquake catalog completeness as a function of location, time, and magnitude.

Also, as discussed in Chapter 4, throughout the project an effort was made to keep the project “hazard-informed” in the sense that highest priority would be given to the issues having the most significance to the hazard results. The goal was not to eliminate issues but to ensure that those issues of highest significance were adequately addressed. This is especially important in a regional study of this kind that includes extensive earth sciences data sets developed for a variety of purposes by numerous researchers.

2.4.3 Workshop #1-Key Issues and Available Data

The goals of Workshop #1 were as follows:

• Introduce the participants in the project to the goals, expectations, and schedule for the project.

• Identify the key issues that would need to be addressed in the course of the SSC.

• Review the available data, including data quality.

• Identify the path forward for the project.

The workshop began with a description of the importance of the CEUS-SSC Project to groups involved with the nuclear industry, including utilities, regulators, and oversight groups. By assembling a single team of experts to develop a new and stable CEUS-SSC, the science for seismic hazard assessment would be advanced, plus there would be cost and schedule-related benefits for existing and planned nuclear facilities. An explanation of SSHAC assessment process goals, study levels, and responsibilities was provided. This included a discussion of the roles of the TI Team members as evaluator experts. The Team was reminded, as they were in all subsequent workshops and working meetings, that they were expected to be objective evaluators of the available data, models, and methods. They were also reminded that the SSHAC assessment process would entail both evaluation and integration, as defined in the SSHAC guidelines.

Three questions are involved in defining SSC: Where will future earthquakes occur? How large will they be? and How frequently will they happen? The scientific assessments needed for SSC were described, including quantification of uncertainties, and examples from projects conducted in the United States and other countries.

The first session of the workshop focused on a review of the technical issues of importance to the CEUS-SSC Project in the context of preparing a PSHA. The sensitivity of seismic hazard results to input parameter choices and associated uncertainties was discussed for several localities, including the New Madrid and Charleston regions. Defining and properly considering sensitivities is an important goal of the project. It was noted that for seismic sources having potentially large earthquake magnitudes (e.g., New Madrid and Charleston regions), assessments of parameters such as characteristic magnitude distributions and source zone locations are particularly important because these sources could potentially affect local and distant sites.

The next session of the workshop focused on data that are available and that may be useful in addressing the key issues discussed in the previous session. The structure of the database being developed for the project was described. Additional analysis was planned for some data sets to make them more useful for the project. After hearing a review of the data documentation process and a brief description of the data sets that had been compiled, workshop participants considered possible gaps in the available data sets.

The bulk of the workshop focused on data and information that could potentially be used for SSC in the CEUS. Data presentations were made by resource experts who had been involved in the development of pertinent databases. Before the workshop, the TI Team reviewed the data being compiled in the database and identified resource experts to participate in the workshop (Table 2.2-2). The list of resource experts was reviewed by the PPRP to ensure that a broad spectrum of experts from the scientific community were identified. Although it is not possible to allow the participation of all resource experts in the technical community, the TI Team identified a representative group of participants from the spectrum of disciplines that are important to seismic source characterization. Those resource experts who were not already participants in the workshop were contacted by TI Team members to discuss their data and gain access to it. Members of the community who provided their data and interpretations are listed in Table 2.2-2.

At the workshop, the resource experts had been asked to focus on data accessibility, formats, and applicability. While first-order interpretations of data were provided, discussions of alternative interpretations and models of the data were kept to a minimum for this workshop. First, presentations were given on gravity data, magnetic data, and a global seismic refraction catalog. Next, the complexities of the origins of earthquakes within stable continental regions were described, as were tectonic features of the Precambrian basement in the Midcontinent, in situ stress and earthquake focal mechanisms, strain fields in the Eastern United States, and paleoliquefaction at localities within the CEUS. The final session of the workshop was focused on the seismicity catalog to be compiled, including the primary sources of earthquake data and the plans to identify dependent events and to assess catalog completeness. Presentations were also made on the approaches used to develop the USGS catalog and selected regional catalogs, including work to identify historical earthquakes.

Workshop #1 was documented on a CD. The CD, which contained the agenda, presentations, a workshop summary, a list of participants, the PPRP letter report, and a photo album of participants, was distributed to all participants; the contents are posted on the EPRI website. The TI Lead and Project Manager hosted a half-day briefing for international observers and young professionals prior to the workshop to improve their understanding of the context of the workshop and its role in the SSHAC assessment process.

2.4.4 Workshop #2-Alternative Interpretations

The goals of Workshop 2 were as follows:

• Review the project SSHAC Level 3 methodology, ground rules, expert roles, and peer review processes.

• Provide an opportunity for the project (TI) team to understand proponent views on important technical issues.

• Discuss the range of alternative views and uncertainties within the larger technical community.

• Discuss the path forward for the project.

The goals of the workshop were accomplished by a series of presentations and discussions designed to provide the TI Team with information it would need to develop a preliminary SSC model. In the development of this model, the knowledge and uncertainties in available data, models, and methods must be taken into consideration. A series of workshop presentations were made by proponent experts who had been asked to provide their views on key technical issues posed in written questions from the TI Team that were provided to each proponent before the workshop. The proponents were also asked to include discussions of the uncertainties associated with their views. The workshop provided an important opportunity for the TI Team to gain a better understanding of the community’s views, to directly question the experts regarding the technical bases for their interpretations, and to debate alternative viewpoints regarding key SSC issues. In several cases, the proponent experts were encouraged to debate the pros and cons of their hypotheses among themselves in a facilitated format, thus allowing the TI Team to understand the key technical bases and uncertainties associated with the alternative models.

Before the workshop, the TI Team reviewed those data, models, and methods being proposed by the technical community having relevance to SSC in the CEUS. The team then identified members of the community who would provide a summary of their viewpoints during the course of the workshop. The list of proponent experts was reviewed by the PPRP to ensure that a broad spectrum of views in the scientific community were represented. Representatives from the USGS were also asked to provide their views on whether there were additional models or methods that should be represented. Those proponent experts are identified in Table 2.2-2. Although it is not possible to allow the participation of all proponent experts in the technical community, the TI Team identified a representative group of participants from across the spectrum of applicable data, models, and methods of importance to the CEUS seismic source characterization. Those proponent experts who were not already participants in the workshop were contacted by members of the TI Team to gain access to their published and unpublished interpretations, and an understanding of their viewpoints and the uncertainties in their interpretations. Members of the community who participated in providing their interpretations are identified in Table 2.2-2.

After the introductory session of the workshop, presentations were made on the following topics:

• Seismicity and seismic parameters, including maximum magnitudes, in selected areas of the CEUS (e.g., the Charlevoix and lower St. Lawrence Seaway regions). 

• Tectonic features throughout the CEUS, including neotectonic features in the Appalachian Piedmont, Ouachita sub-detachment structures, rifts in the Midcontinent, faults and folds in the Illinois Basin, and Quaternary deformation features in the New Madrid region.

• Paleoliquefaction evidence throughout the CEUS, including in the Mississippi Valley and the Wabash Valley, and methods for quantifying uncertainties in paleoliquefaction studies. 

• Alternative interpretations of the state of stress, strain, and earthquake hazards in the regions surrounding the epicenters of the large-magnitude New Madrid and Charleston earthquakes.

• Seismic sources in the Gulf of Mexico.

• Source model features of the 2008 USGS national hazard maps for the CEUS, focused on source characterization in the New Madrid and Charleston.

During the course of these presentations, facilitated discussions occurred, focused on implications to SSC for hazard analysis, including the conceptual models that would represent the range of interpretations and the degree of support of the models based on available data. Proponents provided all references related to their work after the workshop. 

Workshop #2 was documented on a CD. The CD, which contained the agenda, presentations, a workshop summary, a list of participants, the PPRP letter report, and a photo album of participants, was distributed to all participants; the contents are posted on the EPRI website. The TI Lead and Project Manager hosted a half-day briefing for international observers and young professionals prior to the workshop to improve their understanding of the context of the workshop and its role in the SSHAC assessment process.

2.4.5 Working Meetings

Although the workshops provided an opportunity for the TI Team to consider and discuss a variety of topics, much of the actual SSHAC assessment processes of evaluation and integration occurred at the working meetings that took place between and after the workshops. Eleven working meetings were held with the entire TI Team (Table 2.2-1), most meetings typically lasting two to three days, and many other subgroup meetings, webinars, and conference calls were held to discuss and resolve the numerous technical issues associated with the project assessments. Each working meeting was focused on one or more agenda items that required attention by the TI Team, including the following: 

• Identification of potential participants at workshops, including resource experts at Workshop #1 to discuss their data sets, and proponent experts at Workshop #2 to discuss their alternative models and methods.
  

• Development of a conceptual SSC framework and the associated master logic tree.
  

• Approaches to developing the Data Summary and Data Evaluation tables.
  

• Issues associated with the new earthquake catalog.
  

• Alternative approaches to Mmax assessment and updates to the Bayesian approach and their implications.
  

• Approaches to spatial smoothing.
  

• Defining and characterizing repeated large-magnitude earthquake (RLME) sources. 
  

• Use of the paleoseismic data to define the size and recurrence of RLME sources.
  

• Renewal vs. Poisson recurrence models.
  

• Alternatives to seismotectonic zones.
  

• Structure of logic trees, alternatives to include as logic tree branches, and weights on branches.
  

• Hazard feedback and recurrence sensitivity analyses.

Much of the planning for the workshops in terms of developing the agendas and identifying participants was conducted in the working meetings. This allowed the entire TI Team to consider the larger technical community and ensure that a representative cross section of experts was asked to participate in the workshops. Although the workshops provided a forum for interaction among the large number of resource and proponent experts who participated, additional contacts were made with members of the larger technical community outside the workshops (see Table 2.2-2); the working meetings provided the opportunity for all TI Team members to discuss the results of those additional communications. At the same time, the TI Team members devoted considerable effort to completing their Data Evaluation and Data Summary tables that document the data, models, and methods that were considered. These are provided in Appendices C and D.

The working meetings were typically held in a conference room environment, with the project database available at all times for projection and discussion. Working Meetings #10 and 11 were held using a conference-call/webinar format. One to three representatives from the PPRP attended 8 of the 11 working meetings in order to observe the deliberation and technical assessment processes (Table 2.2-1). Each working meeting ended with a set of actions for various members of the TI Team to pursue and to bring back to the entire team at the next meeting.

2.4.6 SSC Sensitivity Model Development

As discussed in Section 2.1, a SSHAC assessment process begins with evaluation of available data, models, and methods, followed by the integration process of model-building to incorporate knowledge and uncertainty. The integration process on the CEUS-SSC Project occurred in four stages beginning with development of the “SSC Sensitivity model” and associated hazard calculations, development of the “SSC Preliminary model” and associated hazard calculations, development of the “Draft SSC model” and associated hazard calculations, and development of the “Final SSC model”and associated hazard calculations. As a tool to assist the TI Team in the development of its SSC model, a “conceptual SSC framework” was developed (Chapter4) that provided a basis for documenting the data consideration and evaluation process, defining the key criteria for identifying seismic sources, and structuring the SSC model around a master logic tree. As the integration process ran through the four stages of SSC model development, the conceptual SSC framework provided a common structure for the TI Team.

A key part of a SSHAC Level 3 assessment process is the opportunity to receive and consider feedback about the implications of preliminary assessments. To do so, the SSC Sensitivity model was developed, which included a wide range of conceptual models and parameter values. Based on the conceptual SSC framework, a master logic tree was developed that describes the basic approaches and conceptual models for characterizing the spatial and temporal distribution of future seismicity. For example, the master logic tree for the SSC Sensitivity model included alternative approaches to the spatial characterization of seismicity, ranging from the smoothing of all past earthquakes to the identification of seismotectonic source zones. Paleoseismic data were included in the definition and characterization of RLME sources, and the background zones were defined based on observed seismicity. A key aspect of the SSC Sensitivity model was including models and parameter values that describe a wide range of uncertainty so that the feedback calculations could be carried out to show the relative importance of these assessments. The focus of the SSC Sensitivity model was not on the weights on the logic tree, but the range of branches on the logic tree in order to show their effect on the calculated hazard results and their potential contributions to uncertainty. These hazard calculations and sensitivity analyses were the subject of Workshop #3 Feedback.

2.4.7 Workshop #3-Feedback

The goals of Workshop 3 were as follows:

• Review the progress of the project in terms of meeting key milestones, such as development of the database and earthquake catalog.
  

• Review the SSHAC assessment process being followed.
  

• Discuss the seismicity catalog developed for the CEUS-SSC Project.
  

• Discuss the seismic source characteristics of the SSC Sensitivity model.
  

• Present feedback to the TI Team and staff in the form of SSC sensitivity analyses and hazard sensitivity analyses.
  

• Identify the key issues of most significance to the SSC models.
  

• Discuss the analyses being conducted related to hazard significance.
  

• Discuss the path forward for the CEUS-SSC Project.

These goals were accomplished by a series of presentations and discussions. Basic principles of the SSHAC assessment process and their implementation in the CEUS-SSC Project were described. A discussion was presented on the TI Team’s role in the evaluation process of evaluating the data, models, and methods of the larger technical community, and in the integration process of building models that represent current knowledge and uncertainties. Discussion was presented regarding the need to document the data, models, and methods that have been considered during the evaluation phase of the project. A case history was described that traced the CEUS-SSC Project documentation of an alternative model that postulates that the New Madrid seismic zone will not be seismically active in the future. Proponents for these models participated in Workshop #2, they were contacted and responded to requests for their current relevant data and interpretations, Data Summary and Data Evaluation tables documented that the TI Team has considered the data and proponent views, a representation of the model could be found in the SSC logic tree, and discussion of the proposed model would be documented in the project report in the description of seismic sources associated with the New Madrid seismic zone. It was concluded that there is clear documented evidence that the data and interpretations provided by the proponent experts were evaluated and documented appropriately by the TI Team.

Development of the CEUS-SSC earthquake catalog was described next, including compilation and merging existing catalogs, magnitude conversions to define all earthquakes by moment magnitude, declustering, and assessment of catalog completeness. A preliminary earthquake catalog was completed for use in preparing the hazard sensitivity analyses to be discussed in the workshop. Representatives from the USGS were present at the workshop and were specifically asked to provide their views on the various aspects of the earthquake catalog. Maximum magnitude distributions under development for the CEUS-SSC Project source zones were described. Representatives from the USGS also provided their views on the the maximum magnitude methodologies being used.

Hazard results for seven demonstration site locations were presented. The seismic sources contributing to the hazard, the various parameter estimation approaches used, and model sensitivity were discussed for each of the demonstration sites. Both RLME and regional source zones were described and the sensitivity results were compared. Based on these calculations and sensitivity analyses, a set of conclusions was drawn regarding the most important SSC issues that either contribute most to mean hazard or are important contributors to the uncertainty in the hazard. In addition to hazard sensitivity, calculated results were discussed pertaining to the SSC issues that contribute most to Mmax and earthquake recurrence. The outcome of these feedback studies was that the TI Team could set priorities for focusing on the SSC issues and uncertainties of most significance in developing the SSC Preliminary model.

Quantifying the precision of seismic hazard results in the CEUS was discussed in the next presentation. The purpose of the analysis described was to derive quantitative estimates of how seismic hazard results might change if studies were repeated by different researchers using the same basic information. This type of quantification gives an indication of how well the hazard is understood and how precise our calculated hazard values are. This can also provide information in the future on whether changes in hazard due to new findings should be considered significant. (A discussion of the hazard precision results and conclusions is given in Section 9.4.) The final workshop presentations focused on the path forward for the project, including how the project results will be used.

Workshop #3 was documented on a CD. The CD, which contained the agenda, presentations, a workshop summary, a list of participants, the PPRP letter report, and a photo album of participants, was distributed to each participant; the contents are posted on the EPRI website. The TI Lead and Project Manager hosted a half-day briefing for international observers and young professionals before the workshop to improve their understanding of the context of the workshop and its role in the SSHAC assessment process.

2.4.8 SSC Preliminary Model Development

After Workshop #3 and armed with the feedback information, the TI Team proceeded to develop the SSC Preliminary model. Unlike the SSC Sensitivity model, which contained a number of elements strictly for purposes of sensitivity analysis, the SSC Preliminary model included a logic tree that was intended to represent knowledge and uncertainties, or the center, body, and range of technically defensible interpretations (NRC, 2011). The ranges of branches on the SSC logic tree and their relative weights assigned to the branches were developed by the TI Team through extensive discussions of the available data, models, and methods. The integration process requires that the Team members objectively evaluate the available information and define the center, body, and range of technically defensible interpretations. To do so, the Team was encouraged to consider all methods for uncertainty treatment (e.g., logic trees, continuous probability distributions) and, if desired, build new models to capture current knowledge and uncertainty. For example, strong belief that the available paleoseismic data related to the existence and location of RLMEs are compelling led to spatial models that include these sources and that define the uncertainties in their geometry.

Alternative models regarding the spatial variations in Mmax and future earthquake characteristics led to alternative models of seismic sources (e.g., Mmax zones versus seismotectonic zones). Consideration of spatial variation of recurrence parameters led to the refinement of approaches to expressing spatial stationarity and the aleatory variability of future recurrence parameters through smoothing. The degree of support in the available data that the model elements hold were expressed as weights on alternative branches of the logic tree. Uncertainties in parameter values were expressed as probability distributions. The sequence of nodes within the logic tree expresses the dependencies of assessments from general conceptual models on the left to parameter distributions that define the models on the right.

2.4.9 Finalization and Review of SSC Draft and Final Model

After the SSC Preliminary model was developed, a second round of hazard calculations and sensitivity analyses was conducted to provide feedback to the TI Team. These analyses focused on the remaining issues of most importance to the model. The elements of the SSC Preliminary model were presented to the PPRP in a briefing as a means of keeping the PPRP informed of the TI Team’s deliberations. During the briefing, the PPRP provided its comments on the key elements of the SSC model and identified key issues that required resolution as the model-development process continued. Another version of the earthquake catalog was developed, after incorporating comments provided by the USGS and other outside experts, and working meetings were held to finalize the SSC Draft model. Discussions focused on the most important technical issues, the weights on alternative elements of the logic tree, and the final quantification of uncertainties. For example, the issue of alternative approaches to maximum magnitude assessments was debated and a series of meetings and conference calls were conducted in order to consider the implications and relative defensibility of alternative conceptual models governing the Mmax estimates. Consideration and discussion also centered on the most appropriate approaches to smoothing of recurrence parameters. The approach used (discussed in Section 5.3.2) allows for a number of assessments to be made (e.g., strength of the prior distribution on the a-, and b-values) and a number of sensitivity analyses were conducted in order to understand the implications of different aspects of the model, and to compare it to other smoothing approaches.

The SSC Draft model was completed and, as defined in the SSHAC guidelines, it is based on a systematic evaluation of the data, models, and methods proposed by the larger technical community and an integration process that provides the TI Team’s representation of the center, body, and range of technically defensible interpretations. The model was documented in a Hazard Input Document (HID) and Draft Project Report, which were issued to the PPRP and other groups for review (see Section 2.4.10). 

In anticipation of detailed reviews, the TI Team continued its refinement of the SSC Draft model after submittal of the Draft Report and while it was being reviewed. A key issue concerned the assessments of earthquake recurrence, which showed for many sources that the “predicted” recurrence rate averaged over the source based on the smoothing approach adopted was over predicting the rates of “observed” larger-magnitude earthquakes in the catalog. A number of exploratory analyses were conducted to shed light on the reason for this mismatch. Issues related to the earthquake catalog were considered, including the merging of multiple catalog sources, spatial variations in completeness, conversions of various magnitudes and intensity to moment magnitude, and declustering. At the same time, issues related to recurrence estimation were evaluated, including the use of various magnitudes in constraining the exponential recurrence distributions, the influence of the strength of the prior distribution on b-values, and the degree of spatial stationarity between the locations of large-magnitude earthquakes and future large-magnitude events.

As discussed in Section 2.5.2, the PPRP issued a comprehensive set of comments on the Draft SSC model and identified key issues that would require resolution in the development of the Final SSC model. Working Meeting #9 on February 7 and 8, 2011, was later conducted with observation by the full PPRP to review these last key issues associated with the earthquake catalog and the recurrence assessments. Working Meetings #10 and 11 were conducted using a webinar format to review the results of ongoing and exploratory work. A few remaining issues, many identified in the comments from the PPRP on the Draft Report, were also dealt with at this time, such as the bases for the weights given in the master logic tree and methods for assessing seismogenic crustal thickness.

The SSC Final model includes refinements that deal with the outstanding issues identified. For example, the refined model shows reasonable agreement between the recurrence rates for various seismic sources and the observed frequency of earthquakes. Likewise, the review of the catalog led to refinements of the spatial and temporal distribution of catalog completeness estimates. Also, conversions were refined to provide consistent estimates of intensity-to-mb-to-M and intensity-to-M. Accordingly, the refined conversions could be used for moment magnitude estimates for the entire catalog and, in turn, the observed frequency of observed earthquakes was recalculated for all seismic sources. Hazard calculations were conducted using the refined SSC Final model and associated sensitivity analyses were carried out. These hazard results and sensitivity analyses are included in this report.

The refinements to the SSC model, associated hazard results, and revisions made to the project report were reviewed and discussed with the PPRP in a briefing held on June 21 and 22, 2011. The briefing provided an opportunity for the PPRP members to gain a full and complete understanding of the process and technical aspects of the project and to provide oral comments on the SSC Final model. 

2.4.10 Documentation

The SSC Final model was documented in an HID (as was done earlier for the Draft SSC model and the Draft Project Report) to provide a basis for hazard calculations, and the Final Project Report was developed. The steps involved in this documentation are summarized below.

2.4.10.1 Development of the Hazard Input Document

Upon completion of the SSC Final model, the essential elements of the model were documented in the HID for the project (Appendix H). The HID is the key deliverable of the project that can be used for hazard calculations in the future. Specifically, this document is meant for the hazard analyst-providing clarity about the model to be implemented and obviating the need to distill the model from the full report. The HID helps ensure that implementation of the model, which can be challenging due to its size and complexity, is as intended.The technical assessments that constitute the SSC Final model are not justified or discussed in the HID. Rather, the HID includes the logic tree structure for all assessments, the associated branches and weights, and the output recurrence and Mmax distributions that are required for the PSHA. The technical justifications for the assessments in the HID are given in this project report.

2.4.10.2 Development of Earlier Draft Report

The Draft Project Report documented all the assessments made by the TI Team in 2010 and summarizes the methodology that was used to make the assessments. The Draft Report was developed by all members of the TI Team and the Project Manager. It summarized all the key process steps, discussed their consistency with a SSHAC Level 3 assessment process, provided a description of all key project deliverables, and provided a technical discussion and explanation for all elements of the SSC Final model. The appendices to the report provided project-specific documentation of key products such as the final HID, Data Evaluation and Data Summary tables, the project database including the earthquake catalog, and summaries of the workshops and project written communications. The goal of both the Draft and Final Project reports is to provide a self-contained complete description of all aspects of the project such that future readers of the report will understand the methodology, the technical elements of the SSC model, and the technical bases for all assessments.

2.4.10.3 Draft Report Review

Review of the Draft Report was conducted by the PPRP, sponsors, USGS, and other groups, and written review comments were provided to the TI Team for its consideration. The TI Team was instructed to give highest priority to the PPRP comments, but to consider all the reviewer comments in making revisions to the project report. All reviewer comments were considered by the TI Team, and the responses to reviewer comments were summarized in comment response tables. The goal of the report review process was to provide the PPRP and other stakeholders an opportunity to comment on the completeness, clarity, and consistency of the documentation of the SSC model. Consistent with its role within a SSHAC process, the PPRP provided its comments pertaining to both the documentation of the process followed in the project as well as the technical assessments included in the SSC model.

Review of the Draft Report by the sponsors was facilitated by briefings held with members of the TI Team and the NRC (on August 10, 2010); with the utilities (on November 4, 2010); and with the utilities, NRC, and DOE (held February 9–10, 2011). A Final Report was developed that reflects revisions made in light of reviewer comments as well as a description of all refinements made to the model by the TI Team after issuance of the Draft Report (described above in Section 2.4.9). The Final Report was issued to the PPRP and Sponsor reviewers in two installments, on June 16 and August 5, 2011, for their review and concurrence. To assist the review, a briefing was held with the PPRP to review all aspects of the SSC model and the report documentation (June 21–22, 2011).

2.4.10.4 Final Report Development

The fundamental bases for revisions to the Draft Report were the written comments provided by the PPRP and other reviewers. In addition, the TI Team recognized the need to refine certain elements of the SSC Draft model and improve the documentation of the process aspects and technical assessments made for the project. A systematic process was followed for responding to each of the reviewer comments to ensure that all comments were addressed. The Final Report was issued to the PPRP for its final review and concurrence, and a final PPRP closure briefing was held September 7-8, 2011.

2.5 Participatory Peer Review Panel

2.5.1 Roles and Responsibilities

SSHAC guidance specifies that if a PSHA project is to be successful, the crucial need for a strong peer review process cannot be overemphasized. The members of the PPRP met the SSHAC criteria that peer reviewers “must be ‘peers’ in the true sense: recognized experts on the subject matter under review” (Budnitz et al., 1997, p. 48). The purpose of peer review is to provide assurance of the following:

• A proper SSHAC Level 3 process has been followed.

• The diversity of views prevailing within the technical community has been considered.
  

• Knowledge and uncertainties have been properly quantified and incorporated into the analysis.
  

• Documentation is clear and complete.

The CEUS-SSC Project used a participatory peer review process, which involved continuous review throughout all phases of the project. As recommended by the SSHAC guidelines, the PPRP was responsible for reviewing both the technical and process aspects of the project. The peer reviewers interacted frequently with the TI Team, provided formal written comments at regular intervals, and reviewed and approved the project report. 

2.5.2 Reviews and Feedback

The purpose of a participatory peer review process, as opposed to a “late-stage” process, is to provide advice and recommendations during the course of the study and not just near the end. Such feedback is valuable to the TI Team and improves the focus and quality of the evaluation and integration processes. For example, early in the project, the PPRP reviewed the Project Plan and provided its views on the planned work activities. Also, members of the PPRP identified data sets that could be considered by the TI Team. PPRP review comments were instrumental in the TI Team’s developing a data documentation process-including Data Evaluation and Data Summary tables-that would benefit future users of the study. As another example, the PPRP provided its views on how feedback calculations should be considered by the TI Team and cautioned that the calculations should not limit the team’s approach to representing the full range of legitimate views within the technical community. These reviews and recommendations were invaluable in assisting the TI Team in enhancing the assessment process being followed. The technical reviews also greatly assisted the team in focusing on key technical issues and ensuring a complete evaluation of all applicable data, models, and methods.

To assist in the PPRP’s monitoring and review of the project, PPRP briefings were held with the TI Team on May 13, 2009; March 24, 2010; and June 21–22, 2011; as well as the final closure briefing September 7–8, 2011 (Table 2.2-1). These briefings served as opportunities for the PPRP to ask questions and gain clarification about the SSC Sensitivity model, SSC Preliminary model, SSC Draft model, and SSC Final model. They also provided the TI Team with feedback on the models and alerted the Team to the need to provide technical bases and documentation on the key technical assessments. In addition to the briefings, representatives from the PPRP were present as observers at 8 of the 11 working meetings of the TI Team. This provided the PPRP with additional perspective on the technical assessments being made by the TI Team.

In terms of written review of the project report, the PPRP provided an extensive set of comments on the Draft Report that addressed both technical issues and process issues. The TI Team members responded to all the comments and summarized their responses in Comment Response tables, which are included in Appendix I. After revision of the Draft Report in light of comments from the PPRP, as well as comments from the sponsors and other groups (i.e., Defense Nuclear Facilities Safety Board, USGS), a draft Final Report was issued. The PPRP then provided a detailed review of that document (see comments in Appendix I). The PPRP comments on the draft Final Report were defined as either “mandatory,” meaning that the review comments must be addressed by the TI Team in its final documentation of the project report, or “non­mandatory,” meaning that the comments are intended solely to help improve the Final Report. 

2.5.3 Fulfillment of SSHAe-Prescribed Scope of Review of Both Technical and Process Issues

The SSHAC guidelines highly recommend that a participatory peer review process be followed and that the peer review process for a Level 3 project be directed at both the technical and process aspects of the study (Budnitz et al., 1997, p. 50). The “technical” aspects include the TI Team’s evaluation process for considering the applicable data, models, and methods that exist within the larger technical community, and the integration process that represents the center, body, and range of technically defensible interpretations. The technical aspects require a high level of technical expertise on the part of the PPRP, while the process aspects require a knowledge and experience in the application of SSHAC assessment processes. “Process” aspects include carrying out all methodological steps, such as developing a project database, conducting workshops, developing feedback, encouraging technical interaction and debate, and documentation. The PPRP for the CEUS-SSC Project included the requisite expertise and experience to fulfill both aspects of its charge. Individual members of the panel are acknowledged experts in the technical fields related to SSC, and most members have had considerable project experience related to SSHAC studies or studies using similar methodologies (see Appendix G).

2.6 Consistency of CEUS-SSC Assessment Process with SSHAC Guidelines

The SSHAC Level 3 assessment process is a structured technical assessment process accepted in the NRC’s seismic regulatory guidance for ensuring that uncertainties in data and scientific knowledge have been properly represented in seismic design ground motions. The TI Team is responsible for meeting and documenting these goals, and peer reviewers are responsible for evaluating whether these goals have been met. As an accepted expert assessment process that includes participatory peer review of both the process and technical aspects, the SSHAC Level 3 assessment process, if conducted properly, provides confidence that the data, models, and methods of the larger technical community have been considered and that the center, body, and range of technically defensible interpretations have been represented.

This section compares the process followed in the CEUS-SSC Project with that prescribed in the SSHAC guidelines in order to draw conclusions about whether the SSHAC assessment process has been adequately followed. Chapter4, Methodology for Characterizing Seismic Sources, of the SSHAC report (Budnitz et al., 1997) is the applicable section for evaluating the CEUS-SSC assessment process. The SSHAC report devotes most of the methodology discussion to attributes of a Level 4 assessment process, with only minor attention given to the attributes of a Level 3 assessment process. Nevertheless, experience on projects conducted over the past 15 years has shown that the differences in the implementation of Levels 3 and 4 are small (mainly related to the assessment by individual evaluator experts rather than an evaluator team). Therefore, the methodology steps specified in the SSHAC guidelines for a Level 4 assessment for SSC (Budnitz et al., 1997, p. 70) can serve as the basis for comparison with the CEUS-SSC assessment process.

The following discussion will first present the methodology steps exactly as given in the SSHAC guidelines, and then provide a summary of how each step was addressed in the CEUS-SSC Project. Aspects of the steps that pertain to Level 4 projects are adjusted, as appropriate, to pertain to a Level 3 project.

The basic steps in the recommended methodology for SSC are given below in terms of the specific application to SSC.

 

1. Conduct careful expert selection. The process of expert selection should be based on a clear set of criteria aimed at capturing a full range of diversity of expert interpretations.

 

The members of the CEUS-SSC TI Team were selected based on their experience and technical expertise. As a group, the Team’s expertise spanned the wide range needed to conduct SSC, including the disciplines of geology, geophysics, tectonics, seismology, and hazard analysis. The team members are acknowledged experts in their respective fields and are thoroughly acquainted with the active researchers in those fields, as well as in the areas of important ongoing research. In addition to having disciplinary expertise, the TI Team had considerable experience in conducting an SSC for a PSHA. As a result, the team members understood the important issues for such an analysis as well as the current tools for uncertainty treatment.

In addition to the TI Team, a large number of representatives of the larger technical community participated in the project as resource experts, proponent experts, and specialty contractors. Steps were taken by the TI Team, as supported by the PPRP, to ensure that the participation of Resource Experts and Proponent Experts in Workshops #1 and #2 was appropriate and complete in order to be representative of the range of current scientific community interpretations. Specialty contractors were engaged on the project to provide certain technical products, such as geophysical maps, stress interpretations, and guidance for the assessment of paleoliquefaction. Personnel from the USGS played an extended role in this project to ensure that all supportable interpretations of the scientific community were fully identified, evaluated, and represented in the SSC model. Several USGS personnel provided detailed review and feedback on specific issues (e.g., the earthquake catalog, Mmax, and methods), which were considered in the assessment of the SSC model by the TI Team.

2. TFI role. The technical facilitator/integrator should play a strong role, running workshops and expert interactions, monitoring the behavior and participation of the experts, conducting calculations and sensitivity analyses, documenting the final results, and taking intellectual responsibility for the results of the project.

 

The SSHAC Level 3 equivalent of the TFI is the TI Lead, who has technical responsibility for the assessment, leads the TI Team, and works with the specialty contractors charged with certain activities (e.g., hazard calculations, database management, report production). For the CEUS-SSC Project, the TI Lead was responsible for organizing all workshops, working meetings, and PPRP briefings. The TI Lead also was responsible for establishing the SSHAC ground rules for all these interactions and for ensuring that all project participants understood and abided by their particular SSHAC-prescribed roles. Throughout the assessment process it was emphasized to all TI Team members that they would be required to assume intellectual ownership of all aspects of the SSC model. The TI Lead was responsible for organizing the report preparation process and for ensuring completeness and consistency in the contributions from the various team members.

3. Provide a uniform data base to all experts. SSC-related data sets, as defined by the experts themselves, should be provided to all of the experts in formats most useful to the experts.

 

From the outset of the CEUS-SSC Project, database development was a strong focus. An earthquake catalog was developed from a variety of sources, and considerable effort was associated with providing an estimate of moment magnitude for all earthquakes in the catalog. An accessible database was developed for use by the TI Team that was derived from both existing information and newly compiled data sets, including GIS-based components such as magnetic anomaly, gravity, and stress data. Workshop #1-Significant Issues and Available Data was devoted to the identification of available data that would be specifically applicable to addressing the key SSC issues. A number of resource experts with knowledge of applicable data sets made presentations on their data sets and provided a basis for the project data development process. The resource experts provided information related to the quality of the data, their formats, and the history of data development in key regions throughout the CEUS. In addition to the geophysical data sets, the project also sponsored the compilation and evaluation of paleoseismic data because of the importance of that data to the identification and characterization of seismic sources in the region.

As part of the Conceptual SSC Framework for the project (see Chapter 4), a process was instituted for documenting the data that were considered by the TI Team in its evaluations. First, in a “generic” table that is generally applicable to the entire study region, the applicable types of data were identified that have potential use in SSC. Then all data that were considered for particular sub regions or source zones were identified in Data Summary tables, which include a description of each data source’s relevance to SSC. Then, for those data that are used in the source characterization process, Data Evaluation tables for each seismic source were developed that provide an evaluation of the quality of the data and degree of reliance given to each data source in the source characterization process. The goal of these tables is to provide clear documentation of the data sets that were available to the TI Team at the time of its evaluations; the tables also provide an evaluation of the data relative to their specific use on the CEUS-SSC Project.

The Data Summary and Evaluation tables are viewed by both the TI Team and the PPRP as critical to the success of the project. This is the first project to rigorously and systematically document this information, and it is viewed by the PPRP as essential information to support the descriptions and discussion found in Chapters 6 and 7. Early in the project, the PPRP encouraged the TI Team to create a system that would more effectively document the data identification and data evaluation processes, and the TI Team developed the format for the Data Summary and Data Evaluation tables. It is expected that the structure of these tables will provide a valuable methodology step for future SSHAC Level 3 projects.

4. Conduct multiple expert interactions. Interaction among SSC experts is strongly recommended, through such vehicles as workshops, small working meetings, etc.

 

The heart of the SSHAC assessment process is technical expert interaction. These interactions allow the TI Team members to carry out their evaluation and integration processes, including identifying and evaluating data; witnessing the debate of alternative hypotheses by members of the technical community; challenging the views of proponent experts in order to understand the uncertainties; developing models that portray the knowledge and uncertainties in SSC model components; considering feedback related to preliminary assessments; and arriving at an integrated model that represents current knowledge and uncertainties. Considerable learning and reexamination of held views occurs and is encouraged during the course of these interactions. The CEUS-SSC Project took full advantage of this notion in conducting three topical workshops, 11 working meetings, four PPRP briefings, and three meetings to brief the international observers and young professionals attending the workshops. In addition, numerous subgroup meetings, conference calls, and webinar's were held among the TI Team members during the course of the development of the SSC Sensitivity model, the SSC Preliminary model, the SSC Draft model, and the Final SSC model. Each interaction was structured and facilitated to focus on the goals of the SSHAC assessment process, and participants were reminded of their roles and responsibilities within that context.

In addition to the interactions among the TI Team members, all members interacted extensively with members of the technical community through personal visits, e-mails, telephone calls, and attendance at professional society meetings. Such communication allowed team members to be apprised of current research and to gain an understanding of the uncertainties associated with available data, models, and methods. Team members were also participants in pertinent professional meetings, such as the NRC-sponsored workshop on methods for assessing maximum magnitudes held at the USGS in Golden, Colorado (Wheeler, 2009), the USGS workshop on the CEUS Earthquake Hazards Program held in Memphis, Tennessee, October 28–29, 2009, and the Seismological Society of America meeting in Memphis, Tennessee, April 13–15, 2011. The goal in these interactions was to gain an understanding of the current knowledge and uncertainties regarding the technical issues of significance to seismic sources in the CEUS.

5. Elicit SSC judgments from experts. Individual expert elicitations should be conducted through person-to-person interviews. Elicitations of expert teams are also acceptable.

 

The notion of individual expert elicitations is specific to a Level 4 assessment process and thus not directly applicable. However, the assessment and evaluation process that occurs within such expert interviews was carried out by the TI Team. Nine of the 11 working meetings (see Section 2.4.5) were multi-day meetings of the TI Team to review data and develop the SSC assessments. Each working meeting was structured around particular aspects of the ongoing evaluation and integration process.

One or more members of the PPRP participated as observers in 6 of the 9 multi-day working meetings and in 8 of the 11 total working meetings. Between working meetings, subgroups developed their interpretations of specific aspects of the model (e.g., geometries of particular zones, paleoseismic recurrence parameters) and their findings were brought to the entire TI Team for evaluation at the working meetings. It was emphasized throughout the assessment that all members of the TI Team would be expected to claim intellectual ownership of the integrated SSC model.

6. Conduct sensitivity analyses and submit feedback to experts. Following the elicitations, extensive sensitivity analyses should be conducted by the TFI and provided to the experts. They then should interact again as a group to review their interpretations.

 

A hallmark of a SSHAC Level 3 or 4 assessment process is the consideration of feedback by the expert evaluators and the use of that information to gain additional insights into the importance of various aspects of the models. Feedback is provided in terms of both implications to calculated seismic hazard results and implications to various components of the SSC model itself (e.g., earthquake recurrence rates). Three complete feedback cycles were conducted in the CEUS-SSC Project. In the first, an SSC Sensitivity model was developed that provided a complete expression of the knowledge and uncertainties regarding seismic source characteristics in the study area. The elements of the model and their uncertainties were quantified using a logic tree approach, thus allowing for hazard calculation and sensitivity studies that isolated the relative significance of various assessments in the model. The SSC Sensitivity model included some elements that were designed to illustrate the significance of various technical issues and their uncertainties. The feedback results from the SSC Sensitivity model were presented at Workshop #3-Feedback, where conclusions were drawn regarding the most important contributors to the mean hazard at seven demonstration sites throughout the study region. This information provided a basis for focusing the subsequent effort on the significant issues. For example, the magnitude and recurrence rate of RLME sources were shown to be quite important, particularly those like the Meers and Cheraw faults whose recurrence rates based on paleoseismic evidence is higher than rates based solely on observed historical seismicity.

After the first round of feedback, the SSC Preliminary model was developed. This model defined the source characteristics and their uncertainties, as expressed in the logic tree branches and associated weights. All the data evaluations and information gained from expert interactions were brought to bear in the development of this model. A second round of feedback was collected to help focus the effort further. This feedback was discussed in a briefing with the PPRP, and the key technical issues were identified. The discussion centered on the range of views on these issues that had been considered during the evaluation process and the way knowledge and uncertainties had been represented in the SSC Preliminary model. The feedback discussions led to a number of focused activities, including additional work in developing a uniform earthquake catalog, consistent and statistically appropriate treatment of the paleoseismic data, refinement of the approach to spatial smoothing of seismicity to allow for variable a-and b-values, new approaches to reanalyzing the available worldwide database of earthquakes within stable continental regions for purposes of estimating maximum earthquakes, and approaches to incorporating paleoearthquakes into the assessment of the largest observed events for purposes of Mmax assessment. The calculated feedback coming from exercising the SSC Preliminary model was indispensable, and the comments and insights provided by members of the PPRP were valuable as well. The feedback was used to develop the Draft SSC model, which was included in the Draft Project Report along with hazard calculations and sensitivity analyses designed to provide insight into the relative importance of various aspects of the model. The Draft SSC model was reviewed in the Draft Project Report by the PPRP, and a briefing was held to review all of its components. In light of this feedback, the Final SSC model was developed; the model is described in detail in this Final Project Report.

7. Finalize SSC interpretations and combine at hazard level. Integration/aggregation of SSC interpretations usually occurs at the hazard level. The TFI should create the proper conditions, through the application of 1 through 6 above, to combine the expert judgments using equal weights. Allowance should be made for cases where unequal weights are appropriate.

 

As discussed in Section 2.1, the SSHAC assessment process calls for two important activities: evaluation and integration. This methodology step is integration and, although the specific issues related to combining multiple-expert assessments are not applicable to a Level 3 process, the need for an integration step is applicable and vital. Integration is model-building and the proper representation of current knowledge and uncertainties. Throughout the project, the TI Team members fulfilled their roles as evaluators of available data, models, and methods by representing their knowledge and uncertainties in the SSC assessments. They further fulfilled their integrator roles by defining branches and weights on the logic tree that they believed would best represent the views of the larger technical community if it had a similar knowledge of the project databases and if it had gone through the same interactive process.

Speaking to a Level 4 assessment process, the SSHAC guidelines allow for the TFI to consider combining expert assessments using unequal weights. This allowance (which has never been applied in an actual SSHAC project) is provided in case the experts do not fulfill their roles as evaluators and integrators. In that case, the TFI is given the authority to adjust the weights on the component expert models in order to provide-in aggregate-an integrated model that properly represents the center, body, and range of technically defensible interpretations. It is likewise possible to imagine a similar situation within a Level 3 TI Team, whereby a team member does not play the role of an evaluator and integrator. In such a situation, the TI Lead would be responsible for reminding the person of his or her proper role and, if necessary, removing the person from the team. Because all team members are responsible for all aspects of the model, however, it is unlikely that any individual team member would have a significant effect on the integration process. In fact, no such problems presented themselves on the CEUS-SSC Project.

8. Peer review. An active or “participatory” peer review should be conducted throughout the study with the particular focus of the process that was followed in conducting the SSC assessment.

 

A participatory peer review process is essential for a SSHAC Level 3 project, and the SSHAC guidelines call for review of both the technical and process aspects of the project. The technical part of the review entails identifying any data, models, or methods that exist within the technical community that the TI Team may not be aware of, reviewing the evaluation process in workshops and working meetings to offer advice regarding hypotheses and views of members of the community, and reviewing the technical bases provided by the TI Team regarding their integration process to represent the center, body, and range of technically defensible interpretations in light of the data, models, and methods available in the larger technical community. The advantage of a participatory peer review process over a late-stage review process is that the review comments and advice of the PPRP can be used to make mid-course corrections. There were several such comments that led to improvements in the CEUS-SSC Project, as follows:

• Numerous data sets were identified by PPRP members for inclusion in the project database.

• New data compilations developed for the project (e.g., aeromagnetic anomaly, gravity, stress, and paleoliquefaction) were suggested and assisted by the PPRP.

• The structure of the database and associated metadata benefited from PPRP advice.

• The concept of more explicit data documentation led to the development of the Data Summary and Data Evaluation tables.

• Written comments after each workshop provided suggestions for process and technical aspects of the project.

• Detailed feedback and questioning at three PPRP briefings offered perspectives on the sensitivity and preliminary SSC models and on their success at representing current knowledge and uncertainties.

• Ongoing comments and suggestions led to improvement in addressing several key SSC issues, including approaches to Mmax assessment, magnitude conversions for the earthquake catalog, recurrence smoothing approaches, consistency of recurrence rates with observed frequencies, and application of criteria for identification of seismic sources.

The PPRP provided a review of the Draft Report and commented on its clarity and completeness in documenting the technical and process aspects of the project.

In addition to the PPRP, other groups provided review comments on the Draft Report. These included the sponsors, USGS, and others. The USGS provided a review of two important aspects of the project: advice regarding any data, models, or methods that should be considered by the TI Team, and review of the project earthquake catalog. Likewise, the sponsors of the CEUS-SSC Project provided their review comments in the spirit of ensuring a clear and complete documentation of the project and its technical assessments.

As summarized above, each of the methodology steps in the SSHAC guidelines for SSC was followed in the CEUS-SSC Project. In some cases, additional steps were added to ensure that the intent of the SSHAC assessment process was fulfilled.

Table 2.2-1
Technical Meetings Conducted as Part of the CEUS-SSC Project

Table 2.2-2
Contributors to the CEUS-SSC Project

Figure 2.3-1
CEUS-SSC Project organization

Figure 2.3-2
Lines of communication among the participants of the CEUS-SSC Project

Figure 2.4-1
Essential activities associated with a SSHAC Level 3 or 4 project (Coppersmith et al., 2010)