Pilot 2: CMA Pilot 1 Evaluation and Pilot 2 Preliminary Evaluation Studies -- Final Report
Building on the interaction between the FDA and Sponsor during product development, the CMA Pilot 2 program promotes early scientific exchange in a more structured manner by formalizing certain interactions between the FDA and sponsors beginning with completion of Phase I through NDA/BLA submission.
At the time of evaluation, only one of nine drugs in the Pilot had submitted an NDA application (marking the end of the Pilot 2 process), thereby preventing a meaningful evaluation of the costs and benefits of CMA Pilot 2 to the drug development process. The other eight products are still in early to mid-stages of development, with many products having only just completed the first task of developing an Agreement. To the extent possible based on available data from both FDA sources and sponsors, the scope of this preliminary evaluation focused on:
- Meeting effectiveness
- The influence of structured meetings on PDUFA goals
- Expected incremental costs
- Motivation of Sponsors to participate
Although it is too early to draw unequivocal conclusions, this evaluation highlights trends and significant observations that may provide guidance on the program effort and possible improvements to the Pilot 2 program. In order to fully assess the program's benefits and costs, continued monitoring and evaluation is necessary as products complete development and regulatory review.
This section addresses the Pilot 2 observations of FDA and sponsors, and is organized by the following subsections:
- Program Implementation
- Potential Effort/Benefits
FDA and Sponsor interviews revealed that the first Pilot 2 activity-developing the meeting schedule agreement-was approached differently, which greatly influenced the interaction process and program effort. Two distinctive approaches were developed for sponsor/FDA interaction, the trigger method and the fixed schedule approach. Exhibit 5–1 details the two agreement approaches and FDA/Sponsor effort:
Meeting schedules are linked to the completion of milestones, and scheduling is ongoing as development hurdles are achieved. Sponsors especially valued the certainty of receiving FDA feedback during the drug development program with built-in schedule flexibility while the FDA appreciated the program efficiency.
Fixed Schedule Method
This approach was the most common interpretation and was also perceived to require relatively more effort to execute. Under this approach, the FDA and Sponsor plan out all future meetings/interactions with expected study completion dates. Participants using the Fixed Schedule Method found this process to be onerous in light of received benefits. In particular, the FDA RPMs and reviewers involved in this method reported process concerns and unanimously agreed that this approach imposed a great burden without adding much value. Specific difficulties mentioned were:
- Concerns over committing to future meetings without the ability to specify an agenda - quality of content and productivity may be low
- Negotiating and following the agreed to schedule is more effort than the resulting benefit - too much time/effort may be exerted upfront in developing the agreement, the schedule may force unnecessary and unproductive meetings and, additional administrative effort may be incurred if meetings must be re-scheduled
- Guidance is not clear on how often the agreement schedule should be updated if planned meetings are delayed.
|Fixed Schedule Method||"Trigger method"|
• Sponsor and FDA agree on schedule of meetings ahead of time
• FDA is obligated to grant every meeting in the schedule
• 7 of the 9 products followed this approach
• Meetings can be requested with shorter timelines than currently the norm
• Meetings are "triggered" by specific events in drug development process
• FDA not obligated to grant every meeting requested
• Multiple meetings required to negotiate schedule; negotiations take > 20 RPM and 20 reviewer hours
• Must be renegotiated for each sponsor
• Set of guidelines requires less preparation time than full schedule
• Templates can be re-used for other products, eliminating the need to constantly reinvent
• Meetings frequently rescheduled due to unpredictability in development
• High cost for adjusting schedules when deadlines lapse
• Flexible meeting schedule allows for meaningful discussion
• Driven by specific drug development events
|Unnecessary Meetings||• All planned meetings occur whether necessary or not||
• Meetings are triggered only if events occur
• FDA retains some level of discretion in granting meetings
• Avoids costly meetings
In contrast, feedback from FDA teams involved in the Trigger Method suggested little additional effort was necessary to launch the program, and ongoing activities are not considered significantly different to how these Divisions typically interact with sponsors:
- In some Divisions, all meetings requests are granted as requested; this is especially true for promising products addressing a high unmet medical need
- Agreement helped manage the Sponsors' expectations in terms of FDA timelines and type of expected feedback
- Agreement is flexible with no rigid deadlines to re-schedule
There are typically 3 types of FDA-Sponsor interactions that can occur during product development: Meetings in-person, teleconferences, and electronic communications (faxes & e-mails). To quantify program effort, the frequency of each interaction was determined from the Agreement and interactions documented in FDA's Document Filing System. For each type of interaction, an assessment of the average number of attendees by role was determined through FDA interviews. Next, an average level of effort (in hours) required to fully prepare, participate and follow through with any post-meeting action item was established for each person and type of interaction. Comparing corresponding data from the comparison cohort (non-pilot Fast-Track products received in 2003-2004), an overall program effort could be determined in hours by type of activity or by role.
Exhibit 5–2a shows the distribution of communication for the comparison cohort and Pilot 2 products. Dotted bars represent total communication counts based on the actual number that occurred to date, extrapolated to the full development cycle. Overall, FDA-Sponsor interactions have the potential to double compared to the routine interaction during the development stage of typical Fast-Track products (comparison cohort). Most significant are the additional in-person meetings introduced into the Pilot 2 program which represents the bulk of the newly incurred FDA effort (73%) - an estimated 376 hours out of the 512 additional hours required per Pilot 2 product (Exhibit 5–2b).
|Meetings (In person)||Telecon||Electronic Communication (fax, letter, email)|
|Pilot 2||9||5(10*)||13 (26*)|
* Since development is underway for nearly all Pilot 2 products, the expected
number of telecons and electronic communications were extrapolated based on
number of actual occurred through completion of the analysis.
|Startup||Meetings (In person)||Telecon||Electric Communications||Total|
|Estimated # hours for the Pilot 2 over the Baseline for Fast-Track||
Sponsors ascribe high value to the commitment from the FDA for frequent communication and early feedback (Exhibit 5–3a). This input can provide a level of certainty in decision-making that may translate to easier planning and significant development cost and time savings (Exhibit 5–3b). Whether this input will translate into higher quality product development and ultimately better applications will have to be assessed upon completion of the development programs.
As many products are still ongoing in the Pilot 2 program, evaluation topics such as meeting effectiveness and impact on PDUFA goals have yet to be determined. However, data has been collected on program implementation/maintenance effort and sponsor motivations. Both FDA reviewers and sponsors report positive attributes but also negative attributes associated with the program.
While most FDA Pilot participants agreed that the Pilot program could potentially influence the quality of applications from resource-constrained Sponsors, discipline reviewers are skeptical whether these will outweigh the negatives. The most common drawbacks mentioned included:
- A substantial increase in RPM and reviewer workload
- In some cases, the Agreement phase took an excessive amount of effort and was prolonged to a point that it may have hindered development progress
- A large administrative burden associated with the Fixed Schedule Method - most meetings had to be re-scheduled due to the uncertain nature of product development
However, some Divisions reported that their typical operating procedure for any product, regardless of formal designation (e.g., Pilot 2, Fast-Track, non-Fast-Track), already closely mirrors the Pilot 2 process. Hence, these Divisions do not anticipate a dramatic increase in workload, if any.
Resoundingly, sponsors who participated in the Pilot program most value the commitment for timely FDA feedback and further attribute this benefit to their ability to better plan their product development program. This may result in avoiding costly delays and streamlining the product development process. In addition, these Sponsors reported that the cost in terms of effort and time of the Pilot program was minimal. Overall, sponsors valued the positive attributes greater than negative aspects.
A number of eligible sponsors however, chose not to apply for Pilot inclusion based on the perception that the level of FDA involvement during the IND would not significantly differ for the products under consideration (life-threatening diseases). These sponsors view the relevant FDA Divisions as sufficiently engaged and motivated. Of interest for these sponsors however, would be to enroll products that generally receive a "standard" review designation. It should be noted, that sponsors falling into this category were generally the multi-national pharmaceutical companies, with significant prior FDA experience across in many therapeutic areas.
Overall, early observations indicate two key factors are influencing Pilot 2:
- Scheduling: use of the trigger method is a more efficient and logical approach for scheduling meetings/FDA feedback
- Communication: the frequency and methods FDA and sponsors use to communicate have a strong influence on whether the Pilot experience is a positive one
Exhibit 4.4 summarizes some of these early observations that should be considered for continued monitoring and in case new INDs are accepted into the program.
• Provide guidance to sponsors on using the trigger method
• Avoid delays and costs due to long negotiations of meeting schedule
• Agreements should focus on the timing of FDA feedback for certain meetings/reviews
• The variability of communications across FDA divisions is a concern with all sponsors
• Guidance on meetings (to be finalized in early '06) should help clarifiy "dos and don'ts"
• Communicate problems upfront
• Usage of "informal" communication can help clarify minor issues more quickly
• Non-binding/open dialogue can help sponsors move in the right direction; in particular for study design
|Pilot Scope||• Prospectively monitor the workload impact vs. value of the program|
APPENDIX A: Distribution of Pilot 1 and Pilot 2 Products In FDA's Office of New Drugs
 Independent Evaluation of FDA's First Cycle Review Performance - Retroactive Analysis Final Report, December 14, 2005