• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

For Industry

  • Print
  • Share
  • E-mail

Methodology Analysis Overview: CMA Pilot 1 Evaluation and Pilot 2 Preliminary Evaluation Studies -- Final Report

Table of Contents

Background and overview

The approach for the evaluation was based on a series of hypotheses formulated at the beginning of the task to capture costs and benefits to the FDA reviewers and sponsors.  Metrics were selected based on these hypotheses to direct data collection and ensure focused interview questions.  As themes emerged, hypotheses were narrowed to those that warranted further research/analysis.  From analyses, an assessment of the overall impact was determined which led to final conclusions and recommendations.  Exhibit 3–1 summarizes the overall evaluation approach.


Exhibit 3–1. Evaluation Approach

Exhibit 3–1. Evaluation ApproachD


 3.1           Hypotheses/Metrics Development

Hypotheses developed to evaluate both Pilots focused on benefits in terms of value and meeting FDA strategic goals (e.g., decreasing time to approval) and effort/cost to the FDA and sponsors.  Metrics derived from hypotheses captured both quantitative and qualitative aspects (see section 3.2).  Examples of measures used for the evaluation included:  first-cycle approval rate, number of amendments submitted per application, perceived workload distribution, and quality of submissions. 

Examples of hypotheses are provided in Exhibit 3–2.


Exhibit 3–2. Hypothesis Generation Approach

Evaluation Factors Key Hypothesis Areas Sample Hypotheses
Contributing Factors

•  RU interdependence

•  Issue Resolution

•  Application Quality

•  Communication

•  Sponsor Characteristics

•  Sponsor Motivations

•  Sponsors have more time to resolve issues prior to first action

•  Pilot 1 is more effective for large and US-based sponsors

•  Pilot 1 benefits some therapeutic areas more than others

•  RUs cannot be reviewed independently


•  PDUFA Goal

•  Time to Approval

•  First-Cycle Approval


•  FDA meets goal for RUs reviewed w/I 6 months

•  Pilot 1 applications have hogher single approval rate

•  Pilot 1 Priority reviews are more likely to be completed on time

•  Pilot 1 applications require fewer extensions


•  Resource Allocation

•  Workload Distribution

•  Incremental Effort (Start-Up, Recurring)


•  Pilot 1 will require a large effort to implement

•  Workload for Pilot 1 applications is distributed more evenly

•  Pilot 1 will displace other FDA activities


 3.2           Data Collection

Quantitative sources included:

  • Action Packages - a compilation of product review documents (e.g., discipline reviews and review meeting minutes); considered to contain the critical information required for Office/Division Directors to formulate an action
  • Discipline Review Letters - a formal FDA communication to the Sponsor at the conclusion of a discipline review identifying deficiencies only in that particular discipline's portion of an application
  • Time Tracking Databases - on-line system designed to capture the time spent by persons who are working to accomplish CDER's mission and goals; data are used to claim CDER's share of User Fees (PDUFA) collected by the FDA
  • Industry Meeting Tracking System (IMTS) - a database tracking  PDUFA mandated performance goals for all formal meetings between industry and FDA
  • Pilot Status Tracking Data - Pilot-specific data from the PDUFA goal data tracking database
  • Division File System - a database of all internally generated documents (i.e., discipline reviews, letters, meeting minutes, etc.) generated in response to a submission from an applicant or Sponsor
  • Sponsor Pilot 2 Agreements - an established communication schedule during product development; developed by both the FDA and Sponsor

Although these were important sources of data, there were some limitations including:

  • Action Packages were only available for some of the Pilot 1 products, and did not apply to Pilot 2 products
  • Discipline review letters were only applicable to Pilot 1 submissions
  • Time tracking database is not a continuous log of peoples' total work effort
  • Some Pilot-related documents (i.e., sponsor applications) were not entered into DFS

As a result, the evaluation also required qualitative information collected through interviews with FDA review teams and Sponsors (Exhibit 3–3).


Exhibit 3–3. Data Sources and Level of Contribution

Data from various information sources contributed significantly to the CMA Pilot
evaluations and FDA and sponsor interviews were a major contribution .

Data Source Pilot 1 Pilot 2
Action Packages 7 out of 11 N/A
Discipline review Letters 10 out of 11 Products N/A
Time-tracking data Limited Pilot data Limited Pilot data
Meeting tracking data (IMTS) Mtg timelines available Mtg timelines available
Pilot Status tracking data Status data available Status data available
Division File Ststem (DFS) Meeting minutes, etc Meeting minutes, etc
Sponsor Pilot 2 Agreements N/A Planned meetings
FDA Interviews Pilot 1 Pilot 2
Office/Deputy Office Director 2 2
Div. Dir./Review Team Leaders 7 3
Discipline Reviewers 7 0
Regulatory Project Managers 10 8
Sponsor Interviews Pilot 1 Pilot 2
Pilot 1 Sponsors 8 out of 11 N/A
Pilot 2 Sponsors N/A 7 out of 9
Comparison Cohort Sponsors 5 5

Pilot 1