Study Overview: Independent Evaluation of FDA's First Cycle Review Performance – Final Report

Table of Contents

EXECUTIVE SUMMARY

PDUFA, enacted in 1992 and renewed in 1997 (PDUFA II),  2002 (PDUFA III), and 2007 (PDUFA IV), authorizes FDA to collect fees from companies that produce certain human drug and biological products. These revenues provide FDA with additional resources to expedite and improve the review of human drug and biologic product applications.  In conjunction with the PDUFA III Goals, FDA agreed to meet specific performance goals to improve the effectiveness and efficiency of FDA review of NDAs and BLAs. ^[5] Several of these goals are focused on improving the review process activities occurring between initial submission of the application and subsequent FDA action regarding the application (i.e., first-cycle review).  The focus of this study is to identify and examine factors, excluding scientific merit, that contribute to and detract from FDA’s ability to approve an application during the first-cycle review.

FDA, industry and the public benefit when a well-managed application review process allows  sponsors with applications that meet the safety and efficacy standards for approval to resolve all issues (e.g., clarification of, or additional analyses, negotiation of labeling, and postmarketing study commitments) within a single review cycle.  Efficient review allows FDA to fulfill its public health mission to ensure that safe and effective products are available to the public in a timely manner and allows for efficient use of resources.  The public benefit from timely access to safe and effective therapies and first cycle approval allows the sponsoring company to market the product sooner and capture revenues.  Yet, not all products should or can be approved in a single cycle.  If FDA uncovers substantial deficiencies during an application’s first-cycle review or a sponsor does not respond to requests for information in a timely manner, then additional review cycles may be necessary to address all deficiencies in an application.

The PDUFA goals specify that FDA will retain an independent expert consultant to evaluate the application review process improvement initiatives and the impact of GRMPs.  FDA has contracted Booz Allen to perform the independent program evaluation of the application review process.

Objectives and Scope

The goal of the evaluation was to identify factors that contributed to or detracted from first-cycle Approvals, as well as determine the impact of FDA’s implementation of two initiatives to enhance first-cycle review processes and performance during the five-year period of PDUFA III.  In the first initiative, CDER and CBER created a joint guidance, Good Review Management Principles and Practices (GRMPs), which describes a well-managed process for review of NDAs, BLAs and efficacy supplements, particularly during the first-cycle review.  One of the goals of the GRMPs was to improve the rate of first cycle approvals without altering the approval standards. FDA believes that it is in everyone’s best interest , FDA, the sponsor, and society if an otherwise approvable drug is approved during the first cycle. For the second initiative, FDA agreed to provide applicants with early notification of potential NDA and BLA review issues identified during the filing review (i.e., the 74-Day letter). The scientific merit of an application, while of critical importance to the regulatory action, was beyond the scope of this study.

Booz Allen conducted a two-part study of first-cycle review initiatives: a Retrospective Analysis and a Prospective Analysis.  The studies focused on the first-cycle review processes  for marketing applications for NMEs and BLAs.  The Retrospective Analysis, published in January 2006 ^[6], represented an interim analysis of first-cycle approval factors for NME applications submitted between FY2002 and FY2004.  The Prospective Analysis analyzed the remaining PDUFA III applications.  Specific objectives of the two Analyses are discussed in Exhibit 1.

Exhibit 1. Study Objectives

Study	Objectives
Retrospective Analysis	Conduct a retrospective review of the Action Packages for completed application reviews for NDAs for NMEs and BLAs submitted from FY2002 through FY2004 (October 1, 2001 through September 30, 2004) and categorize applications by the number of review cycles. Review the available documentation for FDA review activities associated with each of these applications including the Action Package (e.g., discipline reviews, summary memoranda, meeting minutes, action letters, records of Advisory Committee meeting, if any, correspondence, and amendments) and resubmissions that address FDA's requests for the applicant to respond to application deficiencies. Document FDA’s requests for additional clinical data, clarifications, re-analyses, and other similar and related requests made prior to an Action Letter, along with the applicant's responses and identify any extensive requests. Identify reasons for multiple-cycle reviews, including deficiencies cited by FDA for each application not approved on any given cycle. Where possible, determine when each deficiency was first noted and when the sponsor was notified. Identify those cases where there was a significant deficiency that could not have been resolved during the first-cycle regardless of when the applicant had been notified. To the extent possible, identify deficiencies listed in an action letter that might have been resolved with earlier notification.
Prospective Analysis	Use the findings of the Retrospective Analysis as the basis for the Prospective Analysis Conduct a Prospective Analysis of PDUFA III NMEs and BLAs from FY2005 to FY2007 while they are under review, or immediately after review had been completed. Examine best practices that lead to first-cycle approvals, the underlying characteristics that result in multiple cycle reviews, and the impact that GRMPs have had on the review process. Determine whether there are correlations between review activities and the outcome of the first review cycle for NDAs for NMEs, and BLAs, submitted during PDUFA III. Solicit feedback from industry and FDA review staff about key concerns and challenges in obtaining first-cycle approvals and preventing multiple-cycle reviews. Gather information about the effect on review of implementing the GRMPs, including changes in workload, level of staff experience, opportunity cost of training, and other obligations. Determine the impact of the implementation of the GRMPs on the process of first-cycle reviews of NDAs for NMEs and BLAs submitted during PDUFA III. Performance before and after implementation of the first-cycle initiatives, including notification of filing issues, use of GRMPs, and training on GRMPs, will be evaluated.

This report reflects integrated findings from the entire first-cycle review initiative evaluation study (Retrospective and Prospective Analyses).  This report also includes an assessment of GRMPs impact (Appendix A) and the sponsor perspective gained through interviews and focus groups (Appendix B).

Study Cohorts

The total product applications evaluated for this study included 185 NME NDAs and BLAs that were received during fiscal years 2002-2007 and reached first action by September 30, 2007 ^[7].  The 185 products were divided into two cohorts, the Retrospective Analysis and Prospective Analysis.

• Retrospective Analysis Cohort includes 77 products submitted between FY2002 and FY2004 that received action by December 1, 2004.

• Prospective Analysis Cohort includes 85 applications submitted between FY2005 and FY2007.  This group includes 79 products that reached first action by September 30, 2007 and six additional products that were received in FY2006 - FY2007 but were delayed in reaching first action.  For 41 products in the FY2006 - FY2007 cohort additional data was collected by attending review meetings and soliciting Regulatory Project Manager (RPM) input. In addition, for 20 products in the FY2006 - FY2007 cohort sponsor interviews were conducted post-action.

   

• Overall Study Cohort includes 185 applications submitted between FY2002 and FY2007.  This cohort included the entire Retrospective Analysis and Prospective Analysis Cohorts.  In addition, 23 products, submitted in FY2004 that reached first action after December 1, 2004, missing the cutoff for inclusion for in the Retrospective Analysis Report, were included.

For the Retrospective Cohort, Booz Allen’s primary source of information was FDA documented summaries of the application review outcomes, commonly referred to as Action Packages.  For the Prospective Analysis Cohort, Booz Allen supplemented this information with additional FDA data, as well as solicited sponsor perspectives.  A summary of the timeframes and data associated with each cohort is presented in Exhibit 2.

 Exhibit 2. Overview of Study Cohorts and Data Sources

A more detailed discussion of our approach and sources for data collection is discussed in the following section.

Methodology

Booz Allen followed a systematic methodology for conducti both the Retrospective and Prospective Analyses (Exhibit 3).  The predominant difference between the Retrospective and Prospective Analyses were the data sources available for analysis.

Exhibit 3. Data Gathering and Analysis Methodology

1. Identify Potential Drivers of Multi-Cycle Review	2. Gather Data	3. Analyze Data and Test Hypotheses	4. Develop Findings & Recommendations
Product/Disease Characteristics GRMPs Compliance Issues and communication Sponsor Characteristics FDA Characteristics	Leverage FDA Documents and Systems Review Action Packages Search CDER and CBER systems* Observe the Review Process* Attend meetings and review internal communications* Conduct post-action sponsor interviews* Conduct Focus Groups on GRMPs and the Review Experience* Hold sponsor focus groups * Hold FDA focus groups*	Perform quantitative analysis by calculating basic statistics (e.g., mean, frequency, and range) Conduct additional qualitative analyses	Identify factors that contribute to, or detract from, the first-cycle approval of submissions Identify areas for improvement and recommendations

Note: Asterisk(*) denotes activities performed only in the Prospective Analysis

The following sections provide additional details of our four step methodology, with particular focus on the specific factors analyzed and the data sources used to support the analysis.

Step 1: Identify Potential Drivers of Multi-Cycle Review

The first step consisted of generating hypotheses of potential drivers of single/multiple review cycles and identifying the indicators and/or metrics appropriate for hypothesis testing (Appendix C contains a complete list of hypotheses generated).  Booz Allen generated the initial study hypotheses based on our knowledge of the FDA review process and industry experience with the process, as well as consultation with FDA leadership.  These hypotheses evolved throughout the duration of the study based on observations, additional analysis, as well as feedback from industry focus groups and FDA leadership.  In addition, there were several hypotheses (e.g., application quality, reviewer/division workload) that were not evaluated based on the absence of appropriate indicators or metrics.  The hypotheses can be categorized in five major categories: Product and Disease Characteristics, GRMPs Compliance, Issues and Communication, Sponsor Characteristics and FDA Characteristics (Exhibit 4).  The full list of hypotheses identified is included in Appendix C: Study Hypotheses.

 Exhibit 4. Drivers and Hypotheses of Multi-Cycle Reviews

Factors	Key Analysis Areas		Sample Hypotheses
	Review Designation Mechanism of Action Life Threatening Disease Licensing Status Impact of GRMPs Application Quality Issues/Deficiencies and Resolution Postmarketing Commitments Communication Effectiveness/Timing Company Size/Type Country of Origin Experience with FDA Experience in Therapeutic Area Reviewer/Division Workload Reviewer Experience Staff Turnover Inspections		Implementing GRMPs will contribute to an increase in the first-cycle approval rate and an increase in the transparency of the review process Regulatory Project Manager turnover will result in lower first-cycle approval Products developed by experienced companies are less likely to require multiple review cycles Submissions designated Priority are less likely to require more than one review cycle as compared to those designated as Standard

Note: Red Italics indicate hypothesis categories that were not tested because appropriate test indicators or metrics could not be identified or insufficient data existed.

Step 2: Gather Data

In conducting this study, Booz Allen used multiple sources of data:  FDA systems, Action Packages, review observation during FDA internal meetings, sponsor interviews, FDA focus groups and sponsor focus groups. Specific data sources are summarized in Exhibit 5.

Exhibit 5. Overview of Data Sources

Data Source	Description of Data Source and Use
FDA Action Packages	A primary source of data for both the Retrospective and Prospective Analyses was FDA-compiled product Action Packages which contain records of FDA internal and FDA-sponsor communications and application review documents.
Systems that Support Application Review	During the Prospective Analysis, FDA systems that support NDA and BLA application review were used as a supplementary source of information on review communications, timelines, and issues.
Observation of Application Review Meetings	Since data available in FDA documents and systems were insufficient to test some hypotheses, additional data were gathered by observing the specific aspects of the application review process during the Prospective Analysis. Booz Allen observers attended review meetings (e.g., mid-cycle meetings, pre-approval safety conferences, and labeling meetings) that represented a broad spectrum of activities, including milestones as specified in GRMPs. Multiple divisions and offices were represented in the observed meetings. Beginning mid-2006, forty-one application reviews were observed for this activity.
Review Team Input	Where feasible and not available through other sources, input from FDA Regulatory Project Managers and other review team members was solicited regarding: application review activities, FDA-sponsor interactions (format, frequency and timing), nature and timing of issues identified.
Sponsor Interviews	Booz Allen conducted 20 sponsor interviews post-action, to gather data and industry perspectives on the FDA review processes. Interviews were held post-action to ensure the evaluation activities did not disrupt or potentially influence the review. Sponsors with applications submitted during FY2006 and FY2007, which reached action by June 2007 (33 products) were invited to participate in a post-action interview with Booz Allen. Of the invitees, 20 sponsors accepted and were interviewed; these included: 7 large pharmaceutical companies (35%), 3 medium pharmaceutical companies (15%), 5 small pharmaceutical companies (25%), 2 large biotechnology companies (10%), 2 small biotechnology companies (10%), and 1 research institution (5%). Interview discussions focused on FDA-sponsor communication, GRMPs impact, and specific application experiences.
Industry Focus Groups	Booz Allen conducted two, one day focus group meetings with industry to gather perceptions of the GRMPs and perspectives on the root causes for multi-cycle review. Communication effectiveness between sponsors and FDA was also discussed.
FDA Focus Groups	Booz Allen conducted two, half-day focus group meetings with FDA review divisions to gather perspectives on GRMPs implementation status, challenges with implementation, effectiveness of GRMPs training, and perspectives on root causes of multi-cycle review.
Publicly Available Data Sources	Data sources such as company websites were used to supplement product and sponsor company background information such as product/disease characteristics and sponsor profiles (e.g., company size, previous experiences with FDA).

Step 3: Analyze Data and Test Hypotheses

Data were analyzed using both quantitative and qualitative methods to test hypotheses developed.  However, the small numbers of product applications limited the ability to demonstrate statistically significant analyses.  In addition, the number of product applications meeting the test criteria was even further limited (for example, novel mechanism of action coupled with product origin: in-house vs. acquired technology).  As such, most quantitative analyses were limited to basic statistics (e.g., mean, frequency, range). Qualitative analyses were used to assess factors such as GRMPs impact and effectiveness of FDA/Sponsor communications.  Where possible and practical, the study draws logical inferences based on Booz Allen observations and discussions with FDA, however, the small number of product applications (and subsets of applications)  impacted  the ability to generalize conclusions in some instances.  In the process of testing hypothesis through data collection and analysis, we eliminated some hypothesis based on inadequate data quality or insufficient data. 

Step 4: Develop Findings and Identify Recommendations

While many hypotheses were identified, only those hypotheses that had notable findings were retained.  Using both qualitative and quantitative analyses, Booz Allen generated findings consistent with the study objectives, focusing on the following areas:

• Best practices that lead to first-cycle approval
• Relationship between review activities and the first-cycle review outcome
• Industry feedback on concerns and challenges in obtaining first-cycle review
• GRMPs impact on the review process overall and first-cycle approval specifically.

Based on these findings, Booz Allen developed recommendations focused on activities that FDA and/or industry could perform to address our findings, particularly to improve first review cycle outcomes.  Recommendations were categorized based on the review phases (i.e., pre-submission, filing and planning, review, Advisory Committee, action, post-action).

FINDINGS