FINDINGS: Issues and Communication: Independent Evaluation of FDA's First Cycle Review Performance – Final Report
Application issue identification and communication were key factors analyzed given the existence and severity of unresolved issues that result in multi-cycle reviews. Booz Allen examined interactions between FDA and sponsors prior to application submission and then during the review phase (Exhibit 16). Specific variables evaluated included the timing and frequency of meetings, timing of communication of application issues, and use of postmarketing commitments. Our analysis was limited to those communications and issues that were documented, reported, or observed. For pre-submission meetings, documentation was not available for all pre-submission events, but most significant milestone meetings (e.g., End of Phase 2) were included in the Action Packages or FDA systems, and comprised the basis for the pre-submission analyses. For review communications, we relied on Action Packages and FDA systems in combination with observations at review meetings and review team input.
Exhibit 16. Overview of FDA-Sponsor Communications
|Type of Communication||Timing||Expected Outcomes|
|1||End-of-Phase 2 Meeting||
|3||Filing Review Issues (74-Day Filing) Letter||
|4||Information Requests (e.g., letter, fax, email, phone)||
Pre-submission meetings between the FDA and sponsors provide an opportunity to review and discuss the status of the product in development and to agree on the planning for the subsequent stages of product testing and filing. The two main meetings that occur prior to application submission are the End of Phase 2 meeting(s) and Pre-NDA/BLA meeting. End of Phase 2 meetings are held after Phase 1 and Phase 2 clinical trials have been completed. These meetings are intended to determine whether a product is safe to proceed to Phase 3 clinical efficacy trials and agree on pivotal trial design. Pre-NDA/BLA meetings are conducted after the completion of all clinical studies that will be included in the submission.
Exhibit 17. Effect of End of Phase 2 Meetings on Approval Rate for FY2005-FY2007 Cohort
In the FY2005 - FY2007 cohort, EOP2 meetings, perhaps surprisingly, did not appear to have a positive impact on first-cycle approval rate. Of the 66 products with EOP2 meetings, 44% received first-cycle approval, while 79% of products that did not have EOP2 meetings were approved in the first cycle (Exhibit 17). Of the 37 multi-cycle application reviews that had EOP2 meetings, the deficiency that led to the failure to achieve approval was identified in the EOP2 meeting 43% of the time. However, this observation stands in contrast to the data from the FY2002- FY2004 cohort, in which 52% of the 46 products that had an EOP2 meeting were approved in the first cycle vs. 29% for the 21 products that did not. Booz Allen notes that there was a significant increase in the number of applications that had EOP2 (78%) or Pre-NDA/BLA meetings (93%) in the FY2005 - FY2007 cohort compared to the FY2002 - FY2004 cohort.
The main purposes of the Pre-NDA/BLA meeting are to discuss the efficacy evidence from the Phase 3 trials, identify unresolved issues and agree on the format for the submission, including data presentation methods. In the FY2005-FY2007 cohort, Pre-NDA/BLA meetings were held for virtually all applications, so the approval rate nearly matches that of all products in the Overall Study Cohort (Exhibit 18).
Exhibit18. Effect of Pre-NDA/BLA Meetings and Timing on Approval Rate for FY2005-FY2007 Cohort
There appeared to be a difference observed in the first-cycle approval rate for products that submitted their application within six months of the Pre-NDA/BLA (71%) vs. those that waited more than six months (39%). Additionally, this higher first-cycle approval rate (71%) for applications submitted within six months was in contrast to the data in the FY2002- FY2004 cohort, in which only 46% were approved in the first cycle. Based on interviews with sponsors, Booz Allen speculates this improvement may be explained by a shift in emphasis by FDA on the data review, rather than application format issues, during the Pre-NDA/BLA meetings.
Nearly three-quarters (73%) of product applications had both a Pre-NDA/BLA and EOP2 meeting in the Prospective Analysis Cohort, which was a significant increase from the Retrospective Analysis Cohort (57%). Interestingly, products that had both meetings had a lower first-cycle approval rate (44%) than products that had only one or neither meeting (Exhibit 19). Further analysis regarding sponsor experience and impact on first-cycle approval is discussed in Sponsor Characteristics.
Exhibit 19. Incidence of Pre-NDA/BLA and EOP2 Meetings
Filing Review Notification (74-Day Letter)
FDA evaluates the application within the first 60 days of its receipt to determine if it is sufficiently complete to conduct a full review. If FDA determines that the application can be filed (i.e., the application is sufficiently complete), the application review continues. Under PDUFA III, FDA agreed to communicate to applicants any significant review deficiencies identified during the filing review by day 74 of the review cycle. Sending the Filing Review Notification was also specified as an activity in the Filing Review phase of the GRMPs. FDA considers the Filing Review Notification, commonly referred to as the 74-Day Letter, a preliminary review. FDA does not consider this review to be comprehensive nor indicative of deficiencies that may be identified later in the review cycle . Similarly, FDA may not necessarily communicate deficiencies previously identified prior to the Filing Review (e.g., issues previously communicated during EOP2 or Pre-NDA/BLA meetings).
The 74-Day Letter was implemented to comply with PDUFA III goals in early 2003 (Exhibit 20). The 74-Day Letter was evaluated for effectiveness as a tool to provide earlier communication of issues to sponsors.
Exhibit 20. Filing Review Issues Identified by Year
Since implementation of the 74-Day Letter, the number of letters with no issues identified has remained fairly constant. This suggests that upon adoption, FDA immediately began using this new tool for its intended goal of early sponsor communication, instead of sending a letter (e.g., with no issues identified) primarily to comply with the PDUFA III goals. Significant deficiencies were identified in 59% (89 of 152) of the applications for which a 74-Day Letter was sent to the sponsor (Exhibit 21). Although the filing review is a preliminary and non-comprehensive review, it appears to provide early identification of applications that are at risk for not being approved in the first cycle.
Exhibit 21. Impact of Issue Identification in Filing Review Notification for FY2002-FY2007 Cohort
Of the 89 applications that had deficiencies identified in the 74-Day Letter, only 43% were approved in the first cycle, whereas 62% of applications with no significant deficiencies identified during the review were approved in the first cycle. Of the 24 multi-cycle applications that did not have any issues documented in the 74-Day Letter, 10 had issues identified in EOP2 or Pre-NDA/BLA meetings, and 14 had issues identified during the review.
The 74-Day Letter was an effective tool for early communication and issue resolution for those applications approved in a single cycle as well as those applications that were not approved in the first cycle. In the Prospective Analysis Cohort, 62% (29 of 47) of applications had all their potential review issues identified in the 74-Day Letter resolved by the action date (Exhibit 22). Of those that resolved all potential review issues identified in the 74-Day Letter, 62% were approved in the first cycle, indicating that the filing review successfully identified and communicated most of the major review issues to the sponsor in time to resolve them. As expected, a much smaller percentage (22%) of those applications with issues remaining unresolved from the 74-Day Letter at the action date were approved in the first cycle. For these approved applications, the unresolved review issues were not significant enough to negate the overall finding that the product was safe and effective, and were addressed as postmarketing study commitments.
Exhibit 22. Applications with All Filing Review Issues Resolved by Action Date for FY2005-FY2007 Cohort
Many of the issues identified in the 74-Day Letter can impact approvability. As the data on 74-Day Letters indicate, the resolution of potential approvability issues during the review process is related to higher levels of first-cycle approval. The case study below, which evaluates the potential for an increase in the first-cycle approval rate, further illustrates the importance of addressing these issues during the review cycle when feasible. (Exhibit 23)
Exhibit 23. Approvability Impact of 74-Day Letter Issues
A subset of 29 multi-cycle applications from FY2005 and FY2006 was analyzed to illustrate the impact of 74-Day Letter issues that were not resolved. In this case study group, 19 of the applications had issues reported on the 74-Day Letter, and 13 of the applications did not resolve the issue by the Action Date. In 10 of these cases, the issue directly impacted the approvability of the application, which included CMC, efficacy, and safety issues. This case study is not necessarily quantitatively representative of the Overall Study Cohort, but rather illustrates that the 74-Day Letter often identifies important issues that impact approvability, and that many of these issues could be addressed during the timeframe of the first-cycle review.
The FDA and sponsors frequently engage in informal communications (e.g., email, telephone, and fax) in addition to face-to-face meetings during review of a product application. Sponsors generally contact FDA to determine the status of their product’s review or to respond to an information request while FDA contacts sponsors to request critical information and provide sponsors with opportunities to justify their findings and conclusions.
Exhibit 24. FDA-Sponsor Communications for FY2005-FY2007 Cohort
Analysis of Action Packages and FDA systems revealed that the frequency of FDA-sponsor communications was similar for single-cycle and multi-cycle reviews for the first three-quarters of the cycle (Exhibit 24). However, in the final quarter of the review cycle, there was a significant increase in communications for single-cycle approvals compared to multiple-cycle applications. Approximately 70% of these additional communications were related to labeling and PMC issues, suggesting that likelihood of approval drives the additional communications, rather than the reverse. This pattern was similar for both Standard and Priority review applications.
Further analysis of the FDA-sponsor communications showed that the overwhelming majority of communications used teleconference or written formats (Exhibit 25). According to an FDA guidance, FDA minimizes the use of resource-intensive face-to-face meetings during the review, which are instead reserved for products with specific issues as outlined in the relevant guidance.
Exhibit 25. Communication Type for FY2005-FY2007 Cohort
Most product applications (61%) did not have any face-to-face meetings during the review cycle. Booz Allen speculates the review teams focus on quickly resolving the outstanding issues for potential single-cycle products toward the end of the review by using teleconferences.
When sponsors respond to FDA's information requests, they submit written amendments to the NDA or BLA application. Sponsors also submit amendments to provide the FDA with additional safety (120-day safety update) and efficacy information that may have been agreed upon in earlier discussions. FDA generally asks for timely submission of these amendments to avoid disruption of the application review.
Booz Allen performed a case study on those sponsors that submitted amendments in FY 2005. Sponsors submitted more amendments, on average, for first-cycle approval products (21.2 per application) than for those that were not approved in the first cycle (17 per application), for all product applications submitted in FY2005 (Exhibit 26). There was no discernible difference between the number of amendment submissions for Priority or Standard review applications. The difference in the number of amendments submitted was almost entirely due to submissions in the fourth quarter of the review cycle. Booz Allen speculates that as with FDA-Sponsor communications and meetings, more amendments are also submitted when a product is nearing the approval Action Date.
Exhibit 26. Amendments Submitted for Applications for FY2005 Products
While amendments submitted by sponsors are typically in response to specific requests from FDA, sponsors may also submit amendments without prompting from FDA in order to supplement their application. In the FY2005 cohort of applications analyzed, 18% of all documented amendments were submitted without an FDA request for information. The largest proportion of amendments were related to clinical issues, followed by those regarding CMC and labeling issues.
Exhibit 27. Types of Amendments Submitted for FY2005 Applications
Advisory Committee meetings provide FDA with an additional opportunity to discuss deficiencies of an application by soliciting independent, external advice from experts knowledgeable in specific areas related to drug and biologic products. Although the committee members provide advice to FDA and may recommend approval or disapproval of the application, the FDA is not bound to follow the recommendations of the Advisory Committee (Exhibit 28).
Exhibit 28. Advisory Committee Meetings and Review Designation
In the FY2002-FY2007 cohort, only 18% of the product applications had Advisory Committee meetings. The presence of an Advisory Committee meeting did not seem to impact the likelihood of an application being approved in the first cycle, regardless of whether it was a Priority or Standard review application.
Most Advisory Committee meetings took place in the latter half of the review cycle (Exhibit 29). The timing of the Advisory Committee meeting did not impact the first-cycle approval rate for Priority review applications. With the limited data, a slightly greater proportion of Standard review applications with an Advisory Committee meeting in the fourth quarter were not approved in the first cycle than those with the meeting in the third quarter of the review cycle.
Exhibit 29. Advisory Committee Meeting Timing and Priority/Standard Designation
Applications with major deficiencies  identified and documented either pre-submission or during the review were less likely to be approved in a single-cycle than those applications that did not have a major deficiency identified during the same timeframe (Exhibit 30).
Exhibit 30. Impact of Major Issues Identified on Approval Rate in FY2005-FY2007 Cohort
However, applications were more likely to be approved in the first-cycle if a major deficiency was identified pre-submission (40%) than if major deficiencies were only found during the review (19%). Applications for which no major deficiency was identified either pre-submission or during the review had a higher first-cycle approval rate (92%). The two multi-cycle products that did not have deficiencies identified pre-submission or during review had either marginal therapeutic benefit or a greater risk/benefit profile at the end of the review cycle.
The majority of multi-cycle applications have major deficiencies in only one or two categories (Exhibit 31), however, these were in the critical areas of safety, efficacy or chemistry, manufacturing and controls.
Exhibit 31. Major Deficiencies Cited in First Action Letter of Multi-Cycle Applications by Category for FY2002-FY2007 Cohort
Of the 92 applications requiring multiple cycles, 51 (55%) were cited for a single significant deficiency in the safety, efficacy or CMC categories. Thirty-two applications failed due to deficiencies in a combination of two of these categories, and two for a combination of application format and either CMC or safety. The nine remaining multiple cycle applications failed with significant deficiencies in more than two categories.
These deficiencies were further categorized into the areas of design (e.g., trial or manufacturing process design), execution (e.g., unacceptable clinical execution), or failure to meet study objectives (e.g., clinical endpoints) (Exhibit 32).
Exhibit 32. Major Deficiencies Cited in First-Cycle Action Letter of Multi-Cycle Applications by Area for FY2002-FY2007 Cohort
Of the 124 major deficiencies cited in 92 first-cycle action letters, 39 were related to trial design, 54 to execution and 25 to endpoints. The remaining six application format deficiencies were related to inconsistent documentation or record keeping, inability to locate information, or failure to translate from foreign languages into English. Exhibit 33 provides examples of the types of application issues that were identified by category type.
Exhibit 33. Examples of Issues Observed by Category
|Format (primarily in missing data)||
Postmarketing study commitments (PMCs), also referred to as Phase 4 commitments, are studies that are conducted by a sponsor after FDA has approved a product for marketing. FDA requires PMCs for products in certain situations, such as those approved under the accelerated approval provision, based on animal efficacy data, or not sufficiently labeled for pediatric use .
Exhibit 34. Products with Postmarketing Study Commitments in FY2002-FY2007 Cohort
Agreed-upon PMCs are intended to further characterize the safety, efficacy, or optimal use of a product, or to ensure consistency and reliability of product quality. Agreed-Upon PMCs can be used to resolve important issues that do not override the determination that the product is effective and safe for marketing. For the FY2002-FY2007 cohort products, nearly the same proportion of single-cycle (86%) and multi-cycle (84%) applications were approved with PMCs (Exhibit 34). While the percentage of applications with PMCs was similar between single and multi-cycle, there was a significant difference in the number of PMCs assigned.
Exhibit 35: Distribution of PMCs for Single- and Multi-Cycle Approvals
The average number of PMCs associated with applications that had PMCs was significantly different for single (7.5) and multiple cycle (4.4) review products (Exhibit 35). This difference was primarily due to a greater number of required and CMC PMCs associated with single-cycle applications. There were a similar number of agreed-upon PMCs for single (5.4) and multiple (4.6) cycle review applications. .
Exhibit 36 shows that the greatest proportion of PMCs are related to CMC issues (24%), followed by clinical safety (21%), clinical efficacy (16%) and clinical pharmacology (15%). A new clinical study request constituted 51% of PMCs. A detailed analysis of agreed-upon PMCs and their associated development and tracking processes was conducted in a separate study commissioned by FDA and completed in 2007.
Exhibit 36. Focus Area of Postmarketing Study Commitments for FY2002-FY2007 Cohort