Study Overview: Independent Evaluation of FDA's First Cycle Review Performance – Final Report
PDUFA, enacted in 1992 and renewed in 1997 (PDUFA II), 2002 (PDUFA III), and 2007 (PDUFA IV), authorizes FDA to collect fees from companies that produce certain human drug and biological products. These revenues provide FDA with additional resources to expedite and improve the review of human drug and biologic product applications. In conjunction with the PDUFA III Goals, FDA agreed to meet specific performance goals to improve the effectiveness and efficiency of FDA review of NDAs and BLAs.  Several of these goals are focused on improving the review process activities occurring between initial submission of the application and subsequent FDA action regarding the application (i.e., first-cycle review). The focus of this study is to identify and examine factors, excluding scientific merit, that contribute to and detract from FDA’s ability to approve an application during the first-cycle review.
FDA, industry and the public benefit when a well-managed application review process allows sponsors with applications that meet the safety and efficacy standards for approval to resolve all issues (e.g., clarification of, or additional analyses, negotiation of labeling, and postmarketing study commitments) within a single review cycle. Efficient review allows FDA to fulfill its public health mission to ensure that safe and effective products are available to the public in a timely manner and allows for efficient use of resources. The public benefit from timely access to safe and effective therapies and first cycle approval allows the sponsoring company to market the product sooner and capture revenues. Yet, not all products should or can be approved in a single cycle. If FDA uncovers substantial deficiencies during an application’s first-cycle review or a sponsor does not respond to requests for information in a timely manner, then additional review cycles may be necessary to address all deficiencies in an application.
The PDUFA goals specify that FDA will retain an independent expert consultant to evaluate the application review process improvement initiatives and the impact of GRMPs. FDA has contracted Booz Allen to perform the independent program evaluation of the application review process.
The goal of the evaluation was to identify factors that contributed to or detracted from first-cycle Approvals, as well as determine the impact of FDA’s implementation of two initiatives to enhance first-cycle review processes and performance during the five-year period of PDUFA III. In the first initiative, CDER and CBER created a joint guidance, Good Review Management Principles and Practices (GRMPs), which describes a well-managed process for review of NDAs, BLAs and efficacy supplements, particularly during the first-cycle review. One of the goals of the GRMPs was to improve the rate of first cycle approvals without altering the approval standards. FDA believes that it is in everyone’s best interest , FDA, the sponsor, and society if an otherwise approvable drug is approved during the first cycle. For the second initiative, FDA agreed to provide applicants with early notification of potential NDA and BLA review issues identified during the filing review (i.e., the 74-Day letter). The scientific merit of an application, while of critical importance to the regulatory action, was beyond the scope of this study.
Booz Allen conducted a two-part study of first-cycle review initiatives: a Retrospective Analysis and a Prospective Analysis. The studies focused on the first-cycle review processes for marketing applications for NMEs and BLAs. The Retrospective Analysis, published in January 2006 , represented an interim analysis of first-cycle approval factors for NME applications submitted between FY2002 and FY2004. The Prospective Analysis analyzed the remaining PDUFA III applications. Specific objectives of the two Analyses are discussed in Exhibit 1.
This report reflects integrated findings from the entire first-cycle review initiative evaluation study (Retrospective and Prospective Analyses). This report also includes an assessment of GRMPs impact (Appendix A) and the sponsor perspective gained through interviews and focus groups (Appendix B).
The total product applications evaluated for this study included 185 NME NDAs and BLAs that were received during fiscal years 2002-2007 and reached first action by September 30, 2007 . The 185 products were divided into two cohorts, the Retrospective Analysis and Prospective Analysis.
- Retrospective Analysis Cohort includes 77 products submitted between FY2002 and FY2004 that received action by December 1, 2004.
Prospective Analysis Cohort includes 85 applications submitted between FY2005 and FY2007. This group includes 79 products that reached first action by September 30, 2007 and six additional products that were received in FY2006 - FY2007 but were delayed in reaching first action. For 41 products in the FY2006 - FY2007 cohort additional data was collected by attending review meetings and soliciting Regulatory Project Manager (RPM) input. In addition, for 20 products in the FY2006 - FY2007 cohort sponsor interviews were conducted post-action.
- Overall Study Cohort includes 185 applications submitted between FY2002 and FY2007. This cohort included the entire Retrospective Analysis and Prospective Analysis Cohorts. In addition, 23 products, submitted in FY2004 that reached first action after December 1, 2004, missing the cutoff for inclusion for in the Retrospective Analysis Report, were included.
For the Retrospective Cohort, Booz Allen’s primary source of information was FDA documented summaries of the application review outcomes, commonly referred to as Action Packages. For the Prospective Analysis Cohort, Booz Allen supplemented this information with additional FDA data, as well as solicited sponsor perspectives. A summary of the timeframes and data associated with each cohort is presented in Exhibit 2.
Exhibit 2. Overview of Study Cohorts and Data Sources
A more detailed discussion of our approach and sources for data collection is discussed in the following section.
Booz Allen followed a systematic methodology for conducti both the Retrospective and Prospective Analyses (Exhibit 3). The predominant difference between the Retrospective and Prospective Analyses were the data sources available for analysis.
Exhibit 3. Data Gathering and Analysis Methodology
|1. Identify Potential Drivers of Multi-Cycle Review||2. Gather Data||3. Analyze Data and Test Hypotheses||4. Develop Findings & Recommendations|
||Leverage FDA Documents and Systems
Note: Asterisk(*) denotes activities performed only in the Prospective Analysis
The following sections provide additional details of our four step methodology, with particular focus on the specific factors analyzed and the data sources used to support the analysis.
Step 1: Identify Potential Drivers of Multi-Cycle Review
The first step consisted of generating hypotheses of potential drivers of single/multiple review cycles and identifying the indicators and/or metrics appropriate for hypothesis testing (Appendix C contains a complete list of hypotheses generated). Booz Allen generated the initial study hypotheses based on our knowledge of the FDA review process and industry experience with the process, as well as consultation with FDA leadership. These hypotheses evolved throughout the duration of the study based on observations, additional analysis, as well as feedback from industry focus groups and FDA leadership. In addition, there were several hypotheses (e.g., application quality, reviewer/division workload) that were not evaluated based on the absence of appropriate indicators or metrics. The hypotheses can be categorized in five major categories: Product and Disease Characteristics, GRMPs Compliance, Issues and Communication, Sponsor Characteristics and FDA Characteristics (Exhibit 4). The full list of hypotheses identified is included in Appendix C: Study Hypotheses.
Exhibit 4. Drivers and Hypotheses of Multi-Cycle Reviews
Note: Red Italics indicate hypothesis categories that were not tested because appropriate test indicators or metrics could not be identified or insufficient data existed.
Step 2: Gather Data
In conducting this study, Booz Allen used multiple sources of data: FDA systems, Action Packages, review observation during FDA internal meetings, sponsor interviews, FDA focus groups and sponsor focus groups. Specific data sources are summarized in Exhibit 5.
Exhibit 5. Overview of Data Sources
|Data Source||Description of Data Source and Use|
|FDA Action Packages||A primary source of data for both the Retrospective and Prospective Analyses was FDA-compiled product Action Packages which contain records of FDA internal and FDA-sponsor communications and application review documents.|
|Systems that Support Application Review||During the Prospective Analysis, FDA systems that support NDA and BLA application review were used as a supplementary source of information on review communications, timelines, and issues.|
|Observation of Application Review Meetings||Since data available in FDA documents and systems were insufficient to test some hypotheses, additional data were gathered by observing the specific aspects of the application review process during the Prospective Analysis. Booz Allen observers attended review meetings (e.g., mid-cycle meetings, pre-approval safety conferences, and labeling meetings) that represented a broad spectrum of activities, including milestones as specified in GRMPs. Multiple divisions and offices were represented in the observed meetings. Beginning mid-2006, forty-one application reviews were observed for this activity.|
|Review Team Input||Where feasible and not available through other sources, input from FDA Regulatory Project Managers and other review team members was solicited regarding: application review activities, FDA-sponsor interactions (format, frequency and timing), nature and timing of issues identified.|
|Sponsor Interviews||Booz Allen conducted 20 sponsor interviews post-action, to gather data and industry perspectives on the FDA review processes. Interviews were held post-action to ensure the evaluation activities did not disrupt or potentially influence the review. Sponsors with applications submitted during FY2006 and FY2007, which reached action by June 2007 (33 products) were invited to participate in a post-action interview with Booz Allen. Of the invitees, 20 sponsors accepted and were interviewed; these included: 7 large pharmaceutical companies (35%), 3 medium pharmaceutical companies (15%), 5 small pharmaceutical companies (25%), 2 large biotechnology companies (10%), 2 small biotechnology companies (10%), and 1 research institution (5%). Interview discussions focused on FDA-sponsor communication, GRMPs impact, and specific application experiences.|
|Industry Focus Groups||Booz Allen conducted two, one day focus group meetings with industry to gather perceptions of the GRMPs and perspectives on the root causes for multi-cycle review. Communication effectiveness between sponsors and FDA was also discussed.|
|FDA Focus Groups||Booz Allen conducted two, half-day focus group meetings with FDA review divisions to gather perspectives on GRMPs implementation status, challenges with implementation, effectiveness of GRMPs training, and perspectives on root causes of multi-cycle review.|
|Publicly Available Data Sources||Data sources such as company websites were used to supplement product and sponsor company background information such as product/disease characteristics and sponsor profiles (e.g., company size, previous experiences with FDA).|
Step 3: Analyze Data and Test Hypotheses
Data were analyzed using both quantitative and qualitative methods to test hypotheses developed. However, the small numbers of product applications limited the ability to demonstrate statistically significant analyses. In addition, the number of product applications meeting the test criteria was even further limited (for example, novel mechanism of action coupled with product origin: in-house vs. acquired technology). As such, most quantitative analyses were limited to basic statistics (e.g., mean, frequency, range). Qualitative analyses were used to assess factors such as GRMPs impact and effectiveness of FDA/Sponsor communications. Where possible and practical, the study draws logical inferences based on Booz Allen observations and discussions with FDA, however, the small number of product applications (and subsets of applications) impacted the ability to generalize conclusions in some instances. In the process of testing hypothesis through data collection and analysis, we eliminated some hypothesis based on inadequate data quality or insufficient data.
Step 4: Develop Findings and Identify Recommendations
While many hypotheses were identified, only those hypotheses that had notable findings were retained. Using both qualitative and quantitative analyses, Booz Allen generated findings consistent with the study objectives, focusing on the following areas:
- Best practices that lead to first-cycle approval
- Relationship between review activities and the first-cycle review outcome
- Industry feedback on concerns and challenges in obtaining first-cycle review
- GRMPs impact on the review process overall and first-cycle approval specifically.
Based on these findings, Booz Allen developed recommendations focused on activities that FDA and/or industry could perform to address our findings, particularly to improve first review cycle outcomes. Recommendations were categorized based on the review phases (i.e., pre-submission, filing and planning, review, Advisory Committee, action, post-action).