APPENDIX C: STUDY HYPOTHESES: Independent Evaluation of FDA's First Cycle Review Performance – Final Report

Table of Contents

APPENDIX B: SPONSOR AND FDA FOCUS GROUP FINDINGS

The metrics captured for hypotheses developed and tested in the First Cycle Approval Evaluation are detailed below by the following categories:

• Product/ disease characteristics
• Good Review Management Practices and Principles Compliance
• Review process issues and communication
• Sponsor characteristics
• FDA characteristics

Each hypothesis is accompanied by the anticipated metrics for assessment, as well as the data sources that were used or evaluated for testing.  Some hypotheses were tested, of these some had significant findings and others did not.  Some hypotheses could not be tested, either because appropriate data did not exist or the quality and quantity of such data was insufficient.  Each analysis is marked with a status that details the analysis outcome:

• Analyzed with Findings (AF)
• Analyzed with no Findings (ANF)
• Not Analyzed (NA) due to insufficient data

Product/Disease Characteristic Hypotheses

Hypotheses	Metric(s)	Data Source	Status
1. Products in the Fast-Track program have higher first-cycle approval rates	Approval rate* Drug development designation	Action Packages Data systems	AF
2. Products with Priority review have higher first-cycle approval rates	Approval rate* Review designation	Action Packages Data systems	AF
3. Products with Orphan status designation have higher first-cycle approval rates	Approval rate* Review designation	Action Packages Data systems	AF
4. Products with fee waived have higher first-cycle approval rates	Approval rate* Review designation	Action Packages Data systems	AF
5. Products with novel mechanism of action (MOA) have higher first-cycle approval rates	Approval rate* Novel MOA	Action Packages Data systems	ANF
6. Products for indications classified as life-threatening or for unmet medical needs have greater first-cycle approval rates	Approval rate* Indication characteristics	Action Packages Data systems	ANF
7. Products that have a novel MOA and are for life-threatening conditions have higher first-cycle approval rate	Approval rate* Review designation	Action Packages Data systems	AF
8. In-licensed drugs have greater approval rates	Approval rate* Drug origin Timing of acquisition	Action Packages Web research	ANF
9. Novel MOA under In-licensed have greater approval rates	Approval rate* Review designation	Action Packages Data systems	ANF
10. Products with significant public benefits are less likely to require multiple review cycles (treating life threatening disease?)	Available treatments Affected population Others Note – the indicator of public benefits will be further explored and defined	Web research Review team interviews Sponsor interviews	NA
11. Products for chronic conditions are more likely to require more than one review cycle	Chronic / Acute (based on indication)	Action Packages Web research	ANF
12. Products with more therapeutic areas involved are likely to require multi-cycle reviews	Select from therapeutic area list	Action Packages Web research	NA
13. Products addressing conditions with higher disease incidence are less likely to require multiple review cycles	Affected US population (appropriate ranges will be defined when data is analyzed)	Action Packages Web research	NA
14. Products with international approval are less likely to require multiple review cycles	Number of years approved Country or organizations that approved the drug	Action Packages Web research	ANF
15. Applications with more Indications submitted are less likely to have single review cycle	List indication(s)	Action Packages Web research	NA
16. Increasing disease severity is likely to decrease the likelihood of multiple review cycles	Disease severity Note – the severity scale will be further explored and defined	Action Packages Web research	NA
17. Products preceded by FDA approvals in same drug class are less likely to require multiple review cycles	Number of products in same drug class approved	Action Packages Web research	NA
18. Products treating conditions with strong public advocacy are less likely to require multi-review cycles for approval (e.g., HIV/AIDS)	Based on disease condition or therapeutic area Note – the rating of public advocacy will be further explored and defined	Action Packages Web research	NA
19. NDAs are less likely to require multi-cycle reviews than BLAs	NDA/BLA historical approval rate Center approval philosophy	Action Packages Web research Review team interviews	AF
20. Products that have secondary endpoints fail are more likely to require multiple review cycles	Y/N	Action Packages Data systems Sponsor interviews	NA
21. Applications with international clinical trial sites will increase the likelihood of multiple review cycles	Y/N Number of international sites used Counts of all non-US sites Percentage of non-US sites relative to total	Action Packages Data systems Sponsor interviews	NA
22. Applications with more clinical trial sites increase the likelihood of multiple review cycles	Total number of sites	Action Packages Data systems Sponsor interviews	NA

GRMPs Compliance

Hypotheses	Metric(s)	Data Source	Status
23. Implementing GRMPs will contribute to an increase in the first-cycle approval rate	Approval rate GRMPs implementation status within and across divisions	Action Packages Data systems Review team interviews	AF
24. Product reviews that adopt all activities and timeframes in the first phase (filing and review planning) of GRMPs Guidance have greater first-cycle approval rate	Approval rate Activities done/not done in the first phase Timing compliant/delayed Phase-specific Best Practices	Action Packages Data systems Review team interviews	ANF
25. Product reviews that adopt all activities and timeframes in the second phase (review) of GRMPs guidance have greater first-cycle approval rate	Approval rate Activities done/not done in the second phase Timing compliant/delayed Phase-specific Best Practices	Action Packages Data systems Review team interviews	ANF
26. Product reviews that adopt all activities and timeframes in the third phase (Advisory Committee meeting) of GRMPs guidance have greater first-cycle approval rate	Approval rate Activities done/not done in the third phase Timing compliant/delayed Phase-specific Best Practices	Action Packages Data systems Review team interviews	ANF
27. Product reviews that adopt all activities and timeframes in the fourth phase (Action) of GRMPs guidance have greater first-cycle approval rate	Approval rate Activities done/not done in the fourth phase Timing compliant/delayed Phase-specific Best Practices	Action Packages Data systems Review team interviews	ANF
28. To be compliant with GRMPs timelines will increase initial workload	Review hours distribution across review phase (self-reported) Total work hours (self-reported) Hours spent on non-review activities (self-reported)	Review team interviews	NA
29. Implementing GRMPs will result in earlier interactions between FDA and sponsor	Communication timing (letters, emails, fax, telecom, formal meetings and response time)	Action Packages Data systems Review team interviews	AF
30. Implementing GRMPs will result in more frequent interactions between FDA and sponsor	Communication frequency (letters, emails, fax, telecom, formal meetings and response time)	Action Packages Data systems Review team interviews	AF
31. Implementing GRMPs will result in earlier interactions within the review team	Communication timing (Internal meetings, informal contact)	Action Packages Data systems Review team interviews	NA
32. Implementing GRMPs will result in more frequent interactions within review team	Communication frequency (Internal meetings, informal contact)	Action Packages Data systems Review team interviews	NA
33. Implementing GRMPs will increase the review quality	Communication frequency Issues identified Labeling discussion timing Timing of the Office Director or Division Director receiving the Action Package Timing for AC preparation Note – the indicator of review quality will be further defined	Action Packages Data systems Review team interviews	NA
34. Implementing GRMPs will result in less compression towards the end of the review	Labeling discussion timing Timing of the Office Director or Division Director receiving the Action Package Timing for AC preparation	Action Packages Data systems Review team interviews	ANF
35. Implementing GRMPs will increase the efficiency	Total review effort (based on self-reported workload data) Overall approval time Note – the indicator of review efficiency will be further defined	Action Packages Data systems Review team interviews	NA
36. Implementing GRMPs will shorten review/ approval time	Overall approval time	Action Packages Data systems Review team interviews	NA
37. Implementing GRMPs will increase the clarity of FDA expectations (internally and externally)	Communication frequency Issues resolution Decision activities Note – the indicator of clarity will be further defined	Action Packages Review team interviews Sponsor interviews	ANF
38. Implementing GRMPs will increase the transparency between FDA and sponsor	Pre-defined review schedule Note – the indicator of review transparency will be further defined	Action Packages Data systems Review team interviews Sponsor interviews	ANF
39. Implementing GRMPs will increase the consistency in review process	GRMPs milestones across Divisions Note – the indicator of consistency will be further defined	Action Packages Data systems Review team interviews	ANF
40. GRMPs training could increase the review efficiency	Types of GRMPs training for current reviewers and new hires Review efficiency (will be based on the indicator defined earlier)	Action Packages Data systems Review team interviews	NA
41. Implementing GRMPs will increase the submission quality resulting in fewer IR letters and fewer filing or review issues	Number of IRs Number of issues Number of communications Submission format, issues Note – submission quality will be further explored and defined with FDA stakeholders	Action Packages Data systems Review team interviews Industry interviews	ANF

 

Issues and Communication

Hypotheses	Metric(s)	Data Source	Status
42. Sponsor response times to information requests correlate with first-cycle approval rate	Approval rate # of RFI/ IR and average response time Perception of value of the information request	Action Packages Sponsor interviews	ANF
43. Products in which the sponsor adequately prepares for the industry meetings are less likely to require multiple review cycles	Approval rate Total number of meetings Meeting preparation time	Action Packages Sponsor interviews	NA
44. Early and effective interactions between FDA and Sponsor to identify, prioritize, and resolve issues early are likely to increase the first-cycle approval rate	Timing of pre-submission meetings (EOP2, Pre-NDA/BLA and others) Format of pre-submission meetings Type and count of issues discussed in the meetings (as per Retrospective Study classification system) Response time between FDA and sponsor	Action Packages Data systems	AF
45. Products that have Pre-NDA/BLA meeting prior to submission are less likely to require multiple cycle reviews	NDA/BLA meeting held	Action Packages Data systems	AF
46. Products that have Pre-NDA/BLA meeting less than 6 months prior to submission are less likely to require multiple cycle reviews	NDA/BLA meeting timing	Action Packages Data systems	AF
47. Products that have EOP2 meeting prior to submission are less likely to require multiple cycle reviews	EOP2 meeting held	Action Packages Data systems	AF
48. Products that have both EOP2 and Pre-NDA/BLA meetings prior to submission are less likely to require multiple cycle reviews	EOP2 and NDA/BLA meetings held	Action Packages Data systems	AF
49. Frequent communication is likely to increase the first-cycle approval rate	Number and format of interactions (meetings, telecom, emails, faxes)	Action Packages Data systems	AF
50. Increased communication towards the end of the review cycle is likely to decrease the first-cycle approval rate	Number and format of interactions (meetings, telecom, emails, faxes) Timing of communications	Action Packages Data systems	AF
51. Increasing number of issues identified in pre-submission meetings and correspondence are more likely to predict multiple review cycles (Core issues not the number of issues)	Total number of issues raised	Action Packages Data systems Review Team interviews Sponsor interviews	AF
52. Increasing number of issues identified during review are more likely to result in multiple review cycles (Core issues not the number of issues)	Total number of issues raised in review Issues identified per phase (e.g., filing and primary review)	Action Packages Data systems Review Team interviews Sponsor interviews	AF
53. Applications with increasing number of issues raised under the Trial Design phase are likely to increase the number of review cycles	Identify stage of program in which each issue is raised: Trial Design Trial Execution Trial Analysis Manufacturing Clinical Facility Inspection	Action Packages Data systems Review Team interviews Sponsor interviews	AF
54. Reviews requiring Advisory Committee meeting are more likely to require multiple cycle reviews	Y/N Advisory Committee meeting	Action Packages Data systems Review Team interviews Sponsor interviews	AF
55. Types of Issues raised in Advisory Committee reviews may impact the number of review cycles required	List all issues raised Type of issue raised in Advisory Committee: Trial Design Trial Execution Trial Analysis Manufacturing Inspections	Action Packages Data systems Review Team interviews Sponsor interviews	NA
56. Necessity of more complex responses (e.g., new trial data) from FDA is more likely to result in multiple review cycles	List responses required Characterization of response (e.g., Length size of additional trials) Note – the complexity of response will be further defined	Action Packages Data systems Review Team interviews Sponsor interviews	ANF
57. Non-compliance of sponsor with specific FDA requests will increase the likelihood of multiple review cycles	List all non-compliance events (sponsor does not fulfill request of reviewer); Map to Issues involved in prior column	Action Packages Data systems Review Team interviews Sponsor interviews	ANF
58. FDA response to sponsor close to PDUFA goal date will result in multiple review cycles	Sample metrics would include FDA received date, first action date, performance goal date	Action Packages Data systems Review Team interviews Sponsor interviews	ANF
59. Changes in submission requirements will increase likelihood of multiple review cycles. Such requirements include: endpoints, safety, efficacy and manufacturing standards	Y/N (List) of changes in submission requirements	Action Packages Data systems Review Team interviews Sponsor interviews	NA
60. Products with a greater number of unsolicited amendments are less likely to be approved first-cycle	Number of unsolicited amendments	Action Packages Data systems	ANF
61. Products with a change in the safety profile (as defined by FDA) for a given product class are less likely to be approved first-cycle	Change in safety profile	Action Packages Data systems	NA
62. Products submitted in eCTD or electronic NDA/BLA have greater first-cycle approval	Submission method (electronic, paper, mixed)	Action Packages Data systems	ANF
63. Increased communication between FDA and the sponsor for products with Priority review designation leads to greater first-cycle approval	Number of communications	Action Packages Data systems	AF
64. Products with a clinical SPA review by FDA have greater first-cycle approval	SPA review conducted	Action Packages Data systems	ANF
65. Products with issues reported on the 74-Day letter that were discussed pre-submission are less likely to be approved first cycle	74-Day letter issues Pre-submission meeting issues Action letter issues	Action Packages Data systems	ANF
66. Products with any clinical or safety issue (not a data, data format, or label request) included in the 74-Day letter are less likely to be approved first cycle	74-Day letter issues Action letter issues	Action Packages Data systems	ANF
67. Products with 74-Day letter issues that are not addressed during the review are less likely to be approved first cycle	74-Day letter issues Action letter issues	Action Packages Data systems	AF
68. Products with unresolved safety and efficacy issues identified pre-submission (Pre-NDA/BLA, EOP2) are less likely to be approved first cycle	NDA/BLA and EOP2 issues Action letter issues	Action Packages Data systems	ANF
69. Products with more sponsor meetings during the review cycle are less likely to have multiple review cycles	Number of sponsor meetings	Action Packages Data systems Meeting Observation	AF
70. The use of checklists by reviewers increases the first-cycle approval rate	Use of checklists and number of checklists	Action Packages Data systems	ANF
71. Divisions that have structured internal team meetings and documentation practices have a higher approval rate	Observed meeting style/structure	Meeting Observation	ANF
72. High quality application (e.g., number of IRs, number of submission between application receipt date and 60-day filing date) submission have greater first-cycle approval rates	Review cycles and action dates Total review time Quality parameters (e.g., application completeness, data accuracy, technical data supported labeling, overall scorecard rating) Time interval between initial submission and complete submission if initial submission incomplete (Note: Parameters to assess quality are under development by the FDA and will evolve as data are gathered)	Action Packages Review team interviews Industry interviews	ANF
73. Large US-based companies are likely to have greater first-cycle approval rates	Approval rate Country of origin Size/ type of the sponsors Size of the regulatory unit	Action Packages Web research	AF
74. Companies with more experience with FDA have greater first-cycle approval rates	Approval rate Number of submissions Experience of regulatory staff Use of consultants (e.g., CRO)	Action Packages Web research	AF
75. Companies with more experience in the therapeutic area have greater first-cycle approval rates	Approval rate Number of submissions in each therapeutic area	Action Packages Web research	AF
76. Outsourcing consultant involvement is less likely to result in multiple review cycles	List any third parties involved; Specify nature of third party (e.g., CRO)	Action Packages Data systems Sponsor interviews	ANF
77. Sponsors whose prior submission characteristics shared with that of the current submission are less likely to require multiple review cycles	Number of submissions By phase By trial size Therapeutic area By type By year	Action Packages Data systems Sponsor interviews	NA
78. Sponsors with experience in gaining application approvals since PDUFA are less likely to require multiple review cycles	Number of submission gaining approvals since PDUFA began	Web research Sponsor interviews	AF

FDA Characteristics

Hypotheses	Metric(s)	Data Source	Status
79. Submissions received in the fourth quarter will have the lowest first-cycle approval rate	Number of active applications per quarter Approval rate	Product lists	AF
80. Products that have facility inspections early in the review cycle are more likely to increase first-cycle approval rates	Timing of notification of inspection Type of inspection site Timing of inspectors involved in the review team Approval rate	Action Packages Data systems	NA
81. Applications requiring foreign facility inspection have lower first-cycle approval rates	Inspection site location Timing of notification of inspection Type of inspection site Timing of inspectors involved in the review team Approval rate	Action Packages Data systems	AF
82. Reviewer or Division workload is likely to contribute to multi-cycle review	Approval rate Review cycles Number of active applications (INDs, NMEs, BLAs and efficacy supplements) per year Number of review teams or RPMs in a Division Non-review activities and Priority New initiatives and time spent	Review team interviews	NA
83. Applications that experience turnover of lead reviewers (i.e., Office Director, Clinical Team Lead) are more likely to result in multiple review cycles	Number of reviewer changes over course of review	Review team interviews	AF
84. Application that experience turnover of the RPM are more likely to result in multiple review cycles	Number of reviewer changes over course of review	Review team interviews	AF
85. Applications that are submitted to a Division in which the Division Director changes during application review result in more multi-cycle reviews	Division Director attrition rate	Review team interviews	AF
86. Applications that have more postmarketing commitment studies are less likely to require multiple review cycles	Review cycles Number of post-marketing studies Note – a separate task order will look into PMCs impact	Action Packages Data systems Review team interviews	AF
87. Drug-Device combinations requiring input from multiple divisions or areas are more likely to require multiple cycles	Drug-Device combination from multiple divisions? (Y/N)	Action Packages Data systems Review team interviews	NA
88. Drug-Drug combinations requiring input from multiple divisions or areas are more likely to require multiple cycles	Drug-Drug combinations from multiple divisions? (Y/N)	Action Packages Data systems Review team interviews	NA
89. Increasing numbers of consults (internal and external) are more likely to result in multiple review cycles	Total number of consults (internal and external) per application	Action Packages Data systems Review team interviews	ANF
90. Type and timing of consult (internal vs. external) may impact the number of review cycles required	List each consult, characterize as internal vs. external Timing of consults	Action Packages Data systems Review team interviews	ANF
91. Number and type of issues raised in consults may impact the number of review cycles required	List all issues raised in consults	Action Packages Data systems Review team interviews	NA
92. Applications with dissenting opinions on the Priority review designation amongst reviewers are more likely to require multiple review cycles	Priority designation status disputed – Y/N?	Action Packages Data systems Review team interviews	NA
93. An experienced reviewer (2+ years of regulatory review) will have higher first-cycle approval rates	Approval rate Reviewer experience (e.g. regulatory experience, experience in the therapeutic area)	Action Packages Data Systems (DFS, RMS-BLA) Review team interviews	NA

*Approval Rate metric will be based on specific hypothesis (e.g., for hypothesis related to the Fast-Track Program, the approval rate measured will be for approvals within this program category)

---

^[1] Independent Evaluation of FDA's First Cycle Review Performance – Retrospective Analysis Final Report, Booz Allen Hamilton Inc., January 2006 http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm119469.htm

^[2] For the purposes of evaluating GRMPs compliance and impact, five key activities and associated timelines were considered based on the importance of the activity, as well as the availability of information to assess compliance with these activities: hold filing meeting, communicate filing review issues to applicant, hold mid-cycle meeting, complete primary review, and hold labeling discussions (for approval and approvable actions).

^[3] Review issues are those issues identified during the filing of an application that may potentially impact approval.

^[4] Action packages are a collection of important documents generated during the review of an application.

^[5] The PDUFA III goals can be found at http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm118925.htm.  As of September 2007, FDA began operating under PDUFA IV.  PDUFA IV goals can be found at http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm119243.htm

^[6] Independent Evaluation of FDA's First Cycle Review Performance – Retrospective Analysis Final Report, Booz Allen Hamilton Inc., Jan. 2006 http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm119469.htm
^[7] Six of the 185 applications did not reach action until January 2008

^[8] The 77 products analyzed in “The Independent Evaluation of FDA’s First Cycle Review Performance – Retrospective Analysis Final Report” are included in the Overall Study Cohort.

^[9] A user fee waiver may be granted for a small business submitting its first application.  Also, a waiver may be granted where: it is necessary to protect the public health; assessment of the user fees would present a significant barrier to innovation due to limited resources; fees will exceed the anticipated costs incurred by FDA for conducting the application review; assessment of the fee for an application filed under section 505(b)(1) pertaining to a drug product would be inequitable because an application for a product containing the same active ingredient filed by another person under section 505(b)(2) could not be assessed user fees. (Section 736(d) of the Federal Food, Drug & Cosmetic Act (FDCA))

^[10] Guidance for Review Staff and Industry: Good Review Management Principles and Practices for PDUFA products. April 2005.

^[11] Only the FY2005-FY2007 cohort could be analyzed, since the data source for earlier products was limited to  Action Packages, which contain limited if any information regarding pre-submission meetings.

^[12] SOPP 8401.3.  Filing Action – Communication Options.  Version #1.  May 11, 2003

^[13] PDUFA III goals were to provide the sponsor a notification of deficiencies prior to the goal date for 50% of applications in FY 2003, 70% in FY 2004, and 90% in FY 2005, FY2006, and FY 2007.

^[14] Formal Meetings with Sponsors and Applicants for PDUFA Products Guidance

^[15] A major deficiency is defined as a product- or application-related issue that would contribute to preventing first-cycle approval if not adequately addressed.

^[16] Issues identified in either the EOP2 or Pre-NDA/BLA meeting are grouped in one pre-submission category because sponsors can take as much time as needed to resolve an issue before submitting the application, but must resolve them by the Action Date for issues identified during the review.

^[17] Title IX, Subtitle A, Section 901 of the Food and Drug Administration Amendments Act of 2007 (FDAAA) amends the FDCA to authorize FDA to require holders of approved drug and biological product applications to conduct postmarketing studies and clinical trials for certain purposes (section 505(o)(3)(A), 21 U.S.C. 355(o)(3)(A)). This provision took effect on March 25, 2008. The description in this report refers to policies in place prior to FDAAA implementation.

^[18] Analysis not shown.

^[19] Pre-submission documentation of RPM and Medical team lead were not available for all products.

^[20] Contributing factor was determined by reviewing deficiencies noted on the first-cycle Action Letter.

^[21] Review teams that performed each activity within one week of the guideline were considered fully compliant.  “Highly compliant” review teams performed at least 4 out of the 5 activities within one week of the guideline (80% compliant).

^[22] The activities include: hold filing meeting, communicate filing review issues to applicant, mid-cycle meeting, complete primary review, and labeling discussions (for approval and approvable actions).

^[23] Note: Large pharmaceutical companies had a market capitalization over $5B; large biotechnology had capitalization over $1B; small biotechnology had capitalization under $1B, small-medium pharma had less than $5B.

^[24] At the Application Orientation meeting, sponsors can walk the FDA through the format of their product application. However, discussion of data, results, and conclusions is not the focus of this meeting.

Table of Contents