• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

For Industry

  • Print
  • Share
  • E-mail

Pilot 1: CMA Pilot 1 Evaluation and Pilot 2 Preliminary Evaluation Studies -- Final Report

Table of Contents

Methodology Analysis Overview

Since introducing the Pilot 1 program in October 2003, FDA accepted 11 applications.  This relatively small number of products does not allow in-depth statistical analyses to substantiate findings.  Thus, the evaluation and corresponding recommendations are based on both the quantitative results and qualitative observations of FDA review participants, sponsor Pilot participants, and several sponsors eligible to participate, but who chose not to apply or participate.

The key findings of the Pilot are presented in the following four sections:

3.1 Sponsors Motivations for Participation-Presents a summary of the sponsor motivations and program expectations based on interview findings

3.2 Impact-Discusses the potential benefits (first-cycle approval rate, amendments triggering extensions, number of communications, etc.) of the program that were evaluated quantitatively and qualitatively

3.3 Incremental Effort for Sponsors and FDA-Presents the evaluation of the incremental effort to implement the Pilot

3.4 Summary/Recommendations-Discusses the findings and the reasoning behind the final recommendations

 

 4.1           Sponsor Motivations for Participation

FDA's commitment to an earlier review with a 6-month review clock was the primary motivator for companies to participate.  Additionally, sponsors cited a number of desired outcomes that made the Pilot opportunity attractive.  These were mostly related to the potential for increased communication and publicity, and the prospect of expedited review beyond a non-Pilot review (Exhibit 4–1).  In contrast, sponsors that were eligible for Pilot 1 enrollment but chose not to participate, noted that current access to the FDA under the Fast-Track/Rolling Review process was satisfactory and therefore found no additional incentives to enroll.

Exhibit 4–1.  Sponsor Motivations for Pilot 1 Participation

Exhibit 4–1. Sponsor Motivations for Pilot 1 ParticipationD

 4.2           Impact

The themes that emerged during the evaluation are addressed in the following four areas:

  • Program Outcomes-first-cycle approvals, application quality, and overall communications
  • RU Interdependencies
  • Timing of RU Submissions
  • Early Review

 

 4.2.1                Program Outcomes

Meeting the specific Pilot expectations/goals was a key evaluation factor.  The analysis examined first-cycle approval rates, quality of applications, number of deficiencies and effectiveness of communications. The first-cycle approval rate was of high interest since it is a highlighted PDUFA III goal as a means to ensure earlier access of new promising products that address a high unmet medical need.  Because of the Pilot 1 eligibility criteria, a majority of the products enrolled fit into this high unmet medical needs category.

Of the 11 Pilot products, one was withdrawn by the sponsor due to lack of efficacy shortly after the first RU was submitted.   As a result, only 10 products were included in the analyses.  Seven of the 10 Pilot 1 products received first-cycle approval (70%), compared to a 50% first-cycle approval rate comparison cohort products.  The Pilot result is similar to the first-cycle approval rate observed for New Molecular Entities (NMEs) submitted between 2002-2004, that received priority review (62%).  Because of the low number of products in the Pilot and the comparison cohort, the larger comparison group of priority reviewed NMEs addressing high unmet medical needs were included to provide an additional perspective on the first-cycle approval trends for Pilot 1 products (Exhibit 4–2).  

Exhibit 4–2. First-Cycle Approval Rate for Pilot 1 Products, the Fast-Track/Rolling Review Comparison Cohort, and Priority NMEs (FY02-FY04)

Exhibit 4–2. First-Cycle Approval Rate for Pilot 1 Products, the Fast-Track/Rolling Review Comparison Cohort, and Priority NMEs (FY02-FY04)D

While the first-cycle approval results are not materially higher for Pilot 1 products, it is important to note that the 3 products that did not receive a first-cycle approval failed to demonstrate efficacy.  Therefore, an early review would not be expected to provide significant benefit given the long lead times necessary to plan and conduct additional studies - a frequent requirement for such deficiencies.  In contrast, analyzing multiple cycle priority applications (Exhibit 4-2) from a previous study [1], factors influencing multiple cycles included:  application quality, CMC and safety issues.  These deficiencies were either not observed and/or were adequately addressed in the case of Pilot 1 products with first-cycle approvals.  While this may imply that the Pilot's early review structure has a positive influence, the impact of the small sample size remains unknown.   

In addition to first-cycle approvals, the Pilot 1 evaluation also analyzed the impact on application quality.  Using proxy indicators, quality was assessed through:

  • Number of amendments leading to goal extensions
  • Number of issues communicated between FDA and sponsors.

Applications that did not require amendments were considered complete and hence of higher quality compared to submissions that involved amendments.  Similarly, fewer number of communications, for example to discuss deficiencies and FDA requests for information, was used as a proxy indicator of higher quality submissions.

Two of the 10 Pilot 1 products had amendments that required goal extensions.  In these cases, the amendments requiring extensions were triggered by FDA requests for further information that would have occurred regardless of Pilot status (i.e., they were not the result of a discipline review letter).  This is consistent with the comparison cohort and the broader comparison group of priority NMEs, suggesting that there is no change in application quality (Exhibit 4–3).

Exhibit 4–3. Products with Amendments Requiring Goal Extensions

Exhibit 4–3. Products with Amendments Requiring Goal ExtensionsD

Furthermore, the Pilot did not have an impact in terms of number of deficiencies, requests for information from FDA to sponsors, and other clarification communications (Exhibit 4–4).  This may suggest that application quality neither increased nor decreased based on what could be observed.  Moreover, FDA interviews further confirmed that they did not perceive an impact on quality for Pilot 1 applications.

Exhibit 4–4.  Number of Issues Identified for Pilot Products Versus the Comparison Cohort As An Indicator of Application Quality

Exhibit 4–4. Number of Issues Identified for Pilot Products Versus the Comparison Cohort As An Indicator of Application Quality D

From the sponsor's perspective, a key outcome/expectation was the 6-month feedback for  each  RU submitted.  The FDA was able to meet the 6-month review commitment for early submissions for all Pilot 1 products (Exhibit 4–5).

Exhibit 4–5. FDA Met Pilot 1 Six-month Feedback Expectations

Exhibit 4–5. FDA Met Pilot 1 Six-month Feedback ExpectationsD

 

The secondary desired outcome expressed by sponsors, was to have increased communication with FDA during the Pilot 1 process.  As Exhibit 4–6 displays, in comparison to other Fast-Track/Rolling Review products, Pilot 1 products appear to have similar number of communications with FDA.

 

Exhibit 4–6.  Average Number of Meetings and Communications for Pilot 1 Products Versus the Comparison Cohort

Exhibit 4–6. Average Number of Meetings and Communications for Pilot 1 Products Versus the Comparison CohortD

 

 4.2.2                RU Interdependencies

At the onset of the Pilot 1 program, a concern for both FDA and sponsors was the impact of a segmented review, resulting in incomplete submissions or the need to repeat analysis of previously reviewed sections.  Therefore, the evaluation examined the perceptions of the interdependencies of the RUs for both FDA and sponsors, for example:

  • What difficulties did FDA encounter in reviewing RUs in isolation?  
  • What challenges did sponsors encounter submitting RUs prior to having the entire application ready for submission?
  • Did the submission of separate RUs benefit the process in any way?

In the case of the ten Pilot 1 applications, the four RUs were typically:

  • Chemistry, Manufacturing, and Controls
  • Nonclinical Pharmacology and Toxicology (P/T)
  • Clinical
  • BioPharm (in many cases, this section was included as part of another RU).

 In general, both FDA and sponsors agreed that CMC was the most independent RU and should be submitted as one of the first RUs.  The Clinical section on the other hand was broadly considered to be rate-limiting in most product development programs, and relying most heavily on other sections of an application.  Exhibit 4–7 summarizes the FDA and sponsor interview findings in regards to the interdependence of the RUs. 

 

Exhibit 4–7.  Reviewable Unit Interdependencies

The Reviewable Unit Interdependencies are rated on a scale of "0" for "Not
Interdependent" to "4" for "Very Interdependent".

 

 

Clinical

 

Pharm/Tox

 

BioPharm

 

Comments

 

 

Clinical

 

N/A

 

2

 

4

•  Clinical incorporates final label inputs from all disciplines

•  Availability of clinical data is the rate determining step in most development programs

 

Pharm/Tox

 

2

 

N/A

 

2

•  Pharm/Tox often interacts with Clinical and BioPharm; however, Pharm/Tox is commonly ready to submit first

 

BioPharm

 

4

 

2

 

N/A

•  BioPharm interacts heavily with Clinical and is often submitted at the same time

 

CMC

 

1

 

1

 

1

•  The CMC section would benefit most from early submission (e.g., manufacturing inspections could be scheduled earlier, especially if in a foreign country)

While early CMC submission was desirable, most sponsors expressed several practical constraints:

  • Sequencing and timelines of development programs in general will not allow CMC (or most other sections) to be ready that far ahead of other sections
  • Shelf-life expirations would not allow early batches to be produced too far in advance if they are to be marketed (if approved)
  • Sponsors will delay manufacturing/plant investment decisions until a complete NDA/BLA is certain

These considerations are further magnified in the case of biologics due to the relatively higher manufacturing costs, need for validation lots and often shorter shelf-lives.  The consensus among sponsors was that in general, CMC sections could not be prepared significantly earlier than 3-6 months prior to the rest of the submission.

Similar practical constraints were also noted for other sections, implying that the 12 month submission window between first and last sections of an application can rarely be fully leveraged (Exhibit 4–8).

 

Exhibit 4–8.  Limitations of Submitting Early Reviewable Units

 

 

CMC

•  Submitted as late as possible for shelf-life considerations and sale concerns

•  Stability data often not ready more than 3 months prior to full submission

•  Biologics: More validation lots, shorter shelf-life (stability), global shortage of bioprocess capacity, difficult to optimize full-scale production

 

Clinical

•  Most interdependent section, rate limiting

•  Early clinical results may become out-dated, submission/resubmission of updated information may trigger extension

Preclinical

•  Decision to conduct expensive carcinogenicity studies are generally started as late as possible to mitigate costs

 

 4.2.3 Timing of Reviewable Unit Submissions

Of the Pilot 1 products, only one sponsor was able to submit a RU greater than 6 months prior to the complete submission.  More than half of the participating companies submitted the first RU only within 3-6 months prior to the complete submission, and only a few of the companies submitted their first RUs within 0-3 months of the complete submission.  Sponsors noted that the uncertainty of product development and the challenges of preparing a comprehensive RU too far in advance of the complete submission impacted RU submission timing.  In addition, FDA's CMA Pilot guidance specified that no more than four RUs would be accepted per application.  One sponsor submitted four RUs-a majority submitted between two to three RUs (Exhibit 4–9).

Exhibit 4–9. Timing of Reviewable Units in Relation to the Complete Submission of the Pilot 1 NDA/BLA

Product Timeline of RU Submissions Prior to Complete Submission (Mos) # of RUs Time First-Last RU (mos)
A 12 & 8 2 12
B 4.5 & 3 2 4.5
C 4.5 1 4.5
D 4.5, 4 (2 RUs), &1.5 4 4.5
E 3.5 1 3.5
F 2 & 1 2 2
G 3 (3 RUs) 3 3
H 1 (2 RUs) 2 1
I 1 1 1
J 2.5 & .5 (2 RUs) 3 2.5

 

4.2.4                Early Review Analysis

Although there seems to be a positive trend in first-cycle approvals for Pilot 1 products versus the comparison cohort (Exhibit 4–2), this may not be solely due to the review of early submitted portions of an application and subsequent feedback.  Some other factors may come to play that contribute to the first-cycle approval include:

  • Timing of RU submissions relative to the complete application
  • Number of major application deficiencies and identification timeframe
  • Product indication/medical need
  • Product efficacy/safety profile.

A review of Pilot 1 product outcomes suggests that other factors beyond early review may have contributed to favorable first-cycle outcomes, irrespective of Pilot participation.  The early RU review for Product C for example, had no critical deficiencies that could have led to a delay in approval (Exhibit 4–10).  In the case of Product G, the majority of deficiencies were identified with the complete submission and not in the early submitted RUs (Exhibit 4–10).  Of the products analyzed, it appears that only Product E benefited significantly from early CMC RU review, which increased the time available for the sponsor to address the issues within the first-cycle (Exhibit 4–10).

Exhibit 4–10.  Factors Beyond the Pilot 1 Process Effecting First-cycle Review Outcome

Product Pilot 1 Benefit? Comment
B

no

•  CMC delayed and filed as final RU - several issues were identified requiring an amendment, and triggering a 3-month goal extension
C

no

•  Only minor issues were discussed during the review; did not provide a basis for an approval delay
E

yes

•  One RU, CMC, was submitted four months before the complete submission; early submission allowed CMC issues to be addressed easily
G

no

•  Major issue in CMC discipline which was the RU submitted - there was no added benefit from the pilot
H

no

•  Product addresses high unmet need; would have received high attention regardless of filing/review designation
I

no

•  CMC issues elevated a little sooner due to early RU; timing of other issues, e.g., immunogenicity, was similar to a regularly filed priority review
J

no

•  Product addresses high unmet need; would have received high attention regardless of filing/review designation

The outcome of three Pilot 1 products that did not receive first-cycle approval is likely to also be independent of the review process, since all three failed to adequately demonstrate efficacy: a deficiency that typically requires significantly longer resolution time than afforded by early feedback under Pilot 1 (Exhibit 4–11).

 

Exhibit 4–11. Multi-cycle Pilot 1 Products Would Have Had The Same Outcome Regardless of Pilot Status

Product Pilot 1 Benefit? Comment

A

no

Failed to meet non-inferiority criterion for primary efficacy; new study required

D

no

Failed to demonstrate efficacy due to an unacceptable study population; not approvable

F

no

Failed to meet clinically meaningful endpoints; new study required

 

In comparison, several of the Fast-Track/Rolling Review products that received early review and feedback prior to the complete submission, as identified by a FDA request for information, also received 1st-cycle approval.  Similar to the Pilot products however, additional factors may have influenced the outcome, such as the unmet medical need (Exhibit 4–12).

 

Exhibit 4–12. Some Fast-Track/Rolling Products Received Early Review and Feedback

Exhibit 4–12. Some Fast-Track/Rolling Products Received Early Review and FeedbackD

 

In general however, it is not clear whether an early review of Rolling Submissions occurs consistently.  Using FDA's request for information (RFI) as an indicator, 3 of 8 Fast-Track applications that submitted rolling portions of the application received an early review (Exhibit 4–12).  For the other 5 applications, it is difficult to ascertain whether or not a review began before the complete submission since the first RFI was not communicated until after the complete submission (Exhibit 4–13).

As with the Pilot products, the comparison cohort shows that the impact of early review on first-cycle approvals is still unclear.  Of the 4 comparison cohort products that received first cycle approval, only 2 received early review.  This again demonstrates that factors in addition to or independent of early review and subsequent feedback contribute to a favorable first-cycle outcome.

 

Exhibit 4–13. Fast-Track/Rolling Products With Limited Evidence of Early Review

Exhibit 4–13. Fast-Track/Rolling Products With Limited Evidence of Early ReviewD

 

 4.3           Incremental Pilot 1 Effort for Sponsors and FDA

This section analyzes the incremental effort to implement Pilot 1 for sponsors and FDA.  The incremental effort is related to activities specifically conducted to prepare and review products under the Pilot program above what would typically be expended for a Fast-Track/Rolling review product  The assessment of these additional activities was the basis for estimating the incremental effort related directly to the Pilot.

 4.3.1                 Sponsor Effort

Sponsors that participated in the Pilot program noted that the majority of activities related to application preparation and submission would have been performed irrespective of the submission route.  Only few additional activities were reported with minimal incremental effort (Exhibit 4–14).  These were comprised of:

  • Internal meetings to decide whether to participate in the Pilot
  • Preparation of the Pilot enrollment application
  • RU submission schedule negotiation with the FDA

 

Exhibit 4–14.  Major Fast-Track Activities Versus Additional Pilot-specific Activities Reported by Participating Sponsors

Major Sponsor Fast Track Activities

Pre-submission

Post-submission

Request for Fast-Track Designation

Respond to FDA RFIs

EOP1 Meeting

Submit Amendments (if applicable)

EOP2 Meeting

Receive Action Letter (at 6-10 months)

SPA

 

Pre-NDA/BLA Meeting

 

Prepare Application Sections

 

Submit Complete Sections

 

Additional Sponsor Pilot-Specific Activities

Pilot-specific Activities

Incremental Effort

(# Labor Hours)

Comment
Pilot Application Preparation

<100

Sponsors reported the Pilot 1 application effort was minimal
Internal Meetings

<100

Sponsors reported 1-3 additional meetings
Meetings with FDA

N/A

Sponsors reported no Pilot-specific meetings with FDA

Furthermore, many sponsors mentioned that the Pilot helped with internal workload distribution, a benefit especially highlighted by smaller companies with limited resources dedicated to application preparation and interfacing with the FDA review team.  Specifically, sponsors noted some pre-submission and post-submission workload benefits resulting from the Pilot:

  • Pre-submission: ability to prepare each unit separately helped focus on one area at a time, reduced complexity, reduced the need to coordinate multiple resources at the same time
  • Post-submission: more efficient resource management through earlier re-deployment once individual discipline reviews were completed
  • Focused attention on resolving discipline-specific issues raised in the discipline review letter

Overall, the sponsors perception of Pilot 1 was that the effort involved was not burdensome and there was no downside to participating in the program.

 4.3.2                 FDA Effort

Similar to the effort analysis conducted for sponsors, the Pilot effort for FDA was measured in terms of additional activities that FDA reviewers needed to perform beyond what is typically necessary for the Fast-Track/Rolling reviews.  The activities included start-up efforts, additional filing meetings, Pilot admittance/RU schedule negotiations, acknowledgement/discipline review letter writing and review, additional internal FDA meetings, and RU re-work.  The additional Pilot-specific activities for FDA are described below and are also summarized in Exhibit 4–15.    

Start-Up Effort
As this was a new program, each Division receiving a Pilot 1 product incurred start-up efforts associated with establishing how to approach the reviews, for example, by creating appropriate letter templates and/or ensuring the review team were aware of the requirements of the Pilot for review purposes.  By design, the CMA guidance did not specify Pilot implementation, therefore, each Division addressed this separately.  Divisions that had multiple Pilot 1 products noted that this effort was mostly incurred with the first Pilot submission.  Subsequent Pilot products accepted into the same review Division required minimal additional startup effort.

Additional Filing Meetings
In a typical Fast-Track/Rolling review, FDA review Divisions will hold one filing meeting after the complete submission. Under the Pilot, two of the Divisions interviewed held a filing meeting for each RU in addition to a filing meeting after the complete submission.  Other Divisions continued to hold one filing meeting upon receipt of the complete submission; however, a determination whether each RU is substantially complete and meets the conditions of the Agency agreement within 60 days must still occur.

Pilot Admittance/RU Schedule Negotiations
FDA incurred costs for reviewing each request for Pilot participation in order to determine eligibility.  For products that were ultimately accepted, some review team members reported minimal negotiation time with sponsors regarding the most appropriate RU schedule. 

Acknowledgement/Discipline Review Letters
The FDA incurred additional costs for letters acknowledging the receipt of RUs or reporting the results of discipline reviews.  These included direct costs associated with developing the communications as well as multiple levels of review. 

Internal FDA Meetings
Aside from start-up meetings to ensure that the review team was aware of the review process under Pilot 1, there were not many additional Pilot-specific meetings reported.  However, there were a few instances where a typical internal meeting that may only happen once during the course of a non-Pilot review, happened two or more times due to the review of multiple RUs plus the complete submission under the Pilot.

 RU Re-work
A concern voiced by FDA reviewers was the potential for duplicative review efforts-initially upon receipt of the RU, and again during review of later sections that required a reference to prior RUs.  Alternatively, certain deficiencies identified upon early RU review be resolved by the sponsor in time for re-submission during the first action.  In practice however, there was only one product where some incremental RU rework was required.  Most reviewers noted that RU reviews did not differ in terms of level of effort or approach compared to any product review with a 6-month clock.  In particular, it was noted that RUs were typically not reviewed more than once.  Two of the Pilot products had amendments to RUs that triggered clock extensions, but these amendments likely would have occurred regardless of Pilot status.  Therefore, the re-review work on the amendments was not viewed as a Pilot-specific incremental effort.

 

Exhibit 4–15.  Major FDA Fast-Track Review Activities Versus Additional Incremental  Pilot 1-specific Activities

Major Sponsor Fast Track Activities

Pre-submission Post-submission

Grant Fast-Track Designation Request

Filing Meeting

EOP1 Meeting

Discipline Reviews

EOP2 Meeting

Mid-cycle review Meeting

SPA

Advisory Committee Meeting (if applicable)

Pre-NDA/BLA Meeting

Discipline Review Complete/Leadership Signoff

 

Labeling (if applicacble)

 

Action Letter

Additional Sponsor Pilot-Specific Activities

Pilot Activities Recurring Activities "Startup" Activities
Additional Filing meeting

Some divisions have filing meeting for each RU

Additional meetings to clarify Pilot process for the review team
Pilot Admittance/RU Schedule neg.

Additional meetings to determine eligibility of applications and to negotiate RU schedule

Establishing which meetings (e.g., Filing) were necessary by RU
Ack/Discipline Letters

Sending an acknowledgement and discipline review letter for each RU

Pilot acceptance decision-making process
Internal FDA meetings

Typical internal meetings may happen for each RU instead of only once in standard review

Generating letter templates
RU Re-work (RU were typically only reviewed once per cycle except amendments)

Revisiting a discipline review in case of long lags between submissions (1 instance for 11 applications)

 

 

While the incremental Pilot-specific activities did not occur for every Pilot product, in order to estimate the potential incremental effort for FDA, the analysis assumed a scenario in which the all the incremental activities would take place (i.e., worst-case/conservative scenario). To quantify the incremental effort, FDA data sources such as interviews, time tracking data, and the document storage system were used to capture those involved in the various Pilot activities.  For each activity, the effort for discipline reviewers, regulatory project managers, team leaders, Division directors/deputy, office directors/deputy was estimated based on, for example, meeting preparation time, actual meeting time, and post-meeting efforts such as writing/reviewing meeting minutes for each review team member.  

Based on this analysis, the total incremental effort for the Pilot was estimated to be between 190-360 direct labor hours per application.  This effort is above what would typically be expended for a non-Pilot, Fast-Track/Rolling reviewed product in those Divisions included in the evaluation (i.e., experienced the Pilot 1 process).  The breakdown by Pilot-specific activity is presented in Exhibit 4–16.  The effort is displayed as a range, where the lower bars on the chart represent the low end range and the upper bars the high end range of effort as reported by review team members involved in the various activities.

 

Exhibit 4–16.  Incremental Effort for FDA of Pilot-Specific Activities per Application (1 cycle)

Baseline effort= Non-pilot Fast-Track/rolling submissions within divisions with pilot products

FDA Activities Additional Filing Meetings Pilot Application/ Negotiation Acknowledgement/ DR Letters Additional Internal Meetings Start Up Effort RU Re-work Total Range
Lower Limit Effort 44hrs 42hrs 35hrs 24hrs 33hrs 12hrs 190hrs
Upper limit Effort 90hrs 59hrs 72hrs 58hrs 69hrs 0 360hrs

The effort is not distributed evenly among the review team members (Exhibit 4–17).  Most of the incremental effort impacts the Regulatory Project Managers (RPMs), in charge of coordinating key review activities including: sponsor communications, scheduling reviewers, FDA-sponsor meetings, internal product-specific meetings and document reviews, and tracking PDUFA deadlines.

 

Exhibit 4–17. Incremental Effort of Pilot-Specific Activities by Function (1 cycle)

Baseline effort= Non-pilot Fast-Track/rolling submissions within divisions with pilot products

FDA Roles RPM Reviewers Team Leaders Division Director/Deputy Office Director/Deputy Total Range
Lower Limit Effort 53hrs 66hrs 45hrs 23hrs 3hrs 190hrs
Upper limit Effort 104hrs 158hrs 72hrs 0 0 360

 

For the Divisions that participated in Pilot 1, Exhibit 4–18 shows the incremental effort for these Divisions to perform a Pilot 1 style review on all of their Fast-Track product applications.  The estimated incremental effort increase is between 1,520-2,880 direct labor hours (based on the average number of Fast-Track applications submitted to those Divisions between FY 2003-2005).    

 

Exhibit 4–18. Extrapolation of the Incremental Pilot Effort to the Average Number of Fast-Track Products Submitted to Divisions with Pilot 1 Experience

Exhibit 4–18. Extrapolation of the Incremental Pilot Effort to the Average Number of Fast-Track Products Submitted to Divisions with Pilot 1 ExperienceD

The estimated incremental Pilot 1 effort may not be evenly distributed among the Divisions with Pilot experience based on historical Fast-Track submissions.  For example, between FY2003-FY2005, the Anti-infective and Ophthalmology and Drug Oncology Divisions received approximately 56% of all Fast-Track applications submitted in those fiscal years (Exhibit 4–19).  Based on interviews with reviewers in those Divisions, to accommodate for this influx of Fast-Track applications, they strive to perform a six month review on all products considered a high priority, irrespective of the review designation.  Because many of the procedures for early feedback and accelerated review have already been integrated, these Divisions may incur less Pilot 1 implementation effort (in the lower part of the range; Exhibit 3-18) compared to other Divisions.  The estimated incremental increase may not be transferable to Divisions that did not have a Pilot 1 product or who do not have significant experience reviewing Fast Track applications.  To understand the broader potential impact of expanding Pilot 1, FDA may consider prospectively monitoring the Pilot 2 products still in drug development and give sponsors the option to submit their applications as Pilot 1 products (if an application is submitted for those products).  This would allow for further evaluation of the Pilot 1 effort, including the evaluation of 2 additional Divisions that were not included in this Pilot 1 evaluation.

 

Exhibit 4–19.  Three Year Total Fast-Track Submissions by Divisions (FY2003-2005)

Exhibit 4–19. Three Year Total Fast-Track Submissions by Divisions (FY2003-2005)D

There is a possibility that the number of Fast-Track requests will increase if sponsors value the commitment of early review.  This would obviously increase the estimated incremental effort shown in Exhibit 4–18. 

If this increase were to occur, a major FDA concern is that important work without a PDUFA goal deadline may be shifted to a later date.  Consequently, delaying such activities may have a negative impact on public health issues and other FDA responsibilities.

On balance, this workload prioritization challenge is a broader concern for the FDA, one that exists regardless of this Pilot program and especially with product reviews with a 6-month clock.  The interviewed FDA Pilot participants did not report challenges completing other critical work specifically because of the Pilot.  To caveat this finding, interviewees were primarily asked to compare the Pilot effort to their experiences with other Fast-Track, priority reviewed products and were not specifically asked about the shifting of non-PDUFA work.  This is not to say, however, that high workload demands were not a concern, they were just not directly attributed to the implementation of the Pilot.

While this evaluation cannot validate or rule out the impact of non-PDUFA work being displaced, it is likely a broader concern for all products receiving a priority review.  Therefore, additional evaluation may be warranted to determine the impact of the displacement of non-PDUFA work for products receiving a priority review.

4.4           Summary/Recommendations

Overall, this evaluation of the CMA Pilot 1 program does not confirm nor does it discard the possible positive impact on first-cycle approvals and reducing time to approval for important drug/biologic products. The FDA also incurred some additional effort from implementing and administering the Pilot program; however, it is recognized by the FDA that earlier communication of issues has potential value (this was also concluded in the Retrospective study cited earlier).

Specifically, FDA and sponsor Pilot 1 interviews as well as an earlier retrospective product application analysis, show that CMC issues have the greatest potential to be resolved quickly.  Given earlier notification, the Sponsor would have a better chance of resolving such issues within the first cycle.  Further, earlier detection and communication of issues is generally a prudent practice that may, over time, yield shorter times to approval.

From an FDA perspective, Pilot 1 did not offer robust positives; however, FDA reviewers  involved in the Pilot program mentioned that the program was not overly burdensome and that the discipline reviews were effectively the same in terms of experience and effort (as compared to the Fast Track Program).  The most significant FDA concerns are related to the increase in RPM workload, and the potential for non-PDUFA work (e.g., review of annual reports, new protocol reviews) being de-prioritized as a result of the review.  In response to this general concern, reviewers who participated in the Pilot program did not explicitly express difficulty in fulfilling their responsibilities because of the Pilot.  Further, the effects of re-prioritizing work for early submitted RUs may not be dramatic, as the FDA would have received portions of the application at a maximum, 12 months, but typically only less than 4-6 months earlier. 

While largely not observed as part of this analysis, there is a possibility of additional work for reviewers within the first cycle Pilot 1 review as a result of conducting a fragmented review (blue circle, Exhibit 4–20) or if Sponsors rapidly address deficiencies identified early on, and submit amendments to resolve issues within the first review cycle (red circle, Exhibit 4–20).  For the first scenario, only one reviewer of a Pilot product reported the need for such a re-review. This finding is likely the exception as the RU was received 13 months prior to complete submission which from both the FDA and Sponsors' perspective is rarely feasible.  If under the second scenario early feedback ultimately leads to a first-cycle approval, then the increased first-cycle effort, may be off-set by not incurring the effort/costs of subsequent cycles (red circle, Exhibit 4–20).

Even if the need for a second review cycle is inevitable, a Pilot 1 submission may still reduce time to approval by offering more time to address deficiencies identified in early RUs.  For the Sponsor, preparing the resubmission may be expedited while the FDA can potentially conduct a more focused second cycle review with a lower staff burden.

 

Exhibit 4–20. Scenarios on Potential FDA Re-Work

Exhibit 4–20. Scenarios on Potential FDA Re-WorkD

 

Sponsors mostly value the FDA commitment to provide early review/feedback.  This not only offers additional time to address issues but also improves workload distribution and resource management.  This benefit was of particular value to resource-constrained sponsors.  Drawbacks included the uncertainty of product development preventing the sponsor from submitting timely RUs and the difficulty of preparing a comprehensive RU too far in advance.

In summary, due to several factors (e.g., the small sample size of the Pilot and the comparison cohort and the high unmet medical need nature of many of the products), there is no conclusive finding that indicates whether the Pilot 1 program should continue or be terminated.  The key findings of the evaluation showed:

  • Pilot 1 offers some positive aspects:

-     Sponsors valued FDA's 6-month RU review commitment

-    Helped distribute sponsors' workload

-    Additional time to address issues for early submitted RUs is a review process benefit

  • FDA Pilot-participants were not overly burdened by the Pilot
  • Both the Pilot 1 products and the comparison cohort products showed:

-    A strong first-cycle approval rate

-    Similar level of application quality

-    Similar levels of communication.  

Given that this evaluation focused on the comparison of the Pilot 1 program to the Fast Track/Rolling Review program, many Pilot participants offered their perceptions of the Fast Track program.  Industry strongly valued the subtle differences that Pilot 1 offered over the Fast Track/Rolling Review program where FDA remained more neutral. 

While there is no resounding reason to continue the Pilot as a separate program, there may be merit to integrating some positive attributes/lessons-learned from the Pilot 1 program into the existing Fast-Track/Rolling Review structure.  For example, some challenges with the current Fast Track/Rolling Review program are:

  • Rolling submission requirements are not specifically defined; therefore, sponsors do not have to submit complete rolling submission sections to FDA
  • FDA may or may not review a rolling submission prior to the complete application submission, depending on workload demands.

For these particular challenges, the Pilot 1 structure offers potential improvements over the Fast Track/Rolling Review program that include:

  • Requiring a well-defined, complete RU submission
  • Committing the FDA to a 6-month review of the early submitted RU.

If the Fast Track/Rolling Review program were modified with these Pilot 1 attributes, this would allow the FDA to plan better for reviews because they can expect a complete section for early review; early review would be conducted consistently across FDA divisions for early submissions; and issues would be identified earlier, and in some cases, may lead to resolution prior to the first action date, or may help reduce the time between cycles if sponsors can begin addressing deficiencies earlier.

Some considerations before deciding to make any modifications may include:

  • With the intent of further understanding the impact of and improving features of the Pilot 1 program, the FDA may consider prospectively monitoring the current Pilot 2 products, especially those in FDA Divisions that did not have a Pilot 1 product.  Also, ensure there is a mechanism to actively track metrics such as submission quality, review outcome, unforeseen additional FDA effort, and possible displacement of non-PDUFA work).
  • Conduct a detailed evaluation of the Fast Track program to determine if it merits a change.

Additionally, if modifications are implemented, other considerations may include:

  • If RUs are submitted close (e.g., 1 or 2 months) to the complete application submission, build in flexibility to allow the review to be conducted under the complete submission PDUFA clock rather than a separate 6 month RU clock.
  • Consider requiring electronic submissions for early submissions to be electronic.

Further, if the FDA decides to implement any modifications, additional resources would be required since the Agency would incur most of the additional workload burden.  If implemented, the FDA may incur, in addition to the incremental costs described in this report, additional costs during the transition phase as this program is rolled out more broadly to the Divisions which in parallel need to complete reviews of applications currently under review.  It is imperative that the FDA receive additional resources commensurate with the effort incurred to transition to and maintain the new process, in order to ensure that review Divisions are not overburdened

Pilot 2