Approach And Results
We addressed the assessment of the adjustment for changes in review activities for the PDUFA IV Workload Adjuster from three different perspectives. The first approach yielded an alternative model to allow for a comparative analysis to the baseline model described above. The second approach, the sensitivity analysis, was intended to generate alternative models that would serve as a basis for comparison to the baseline model. However, due to certain limitations described in the section that follows, we were unable to proceed with the sensitivity analysis approach. The third approach, the Monte Carlo Analysis, yielded results for both the baseline and the alternative models that served as a basis to provide meaningful insight on the reasonableness of the adjustments for changes in review activities.
The following section provides more detail regarding the three approaches outlined above.
Developing an Alternative Model
The alternative model was designed by considering three modifications to the current PDUFA IV adjustments for changes in review activity. These three modifications directly impact the calculations involved with the adjustments for changes in review activities. These modifications may enhance the current Workload Adjuster’s ability to measure the change in workload.
The first modification involved adding another review activity for the NDA/BLA applications. This additional activity captured the change in workload associated with the review of NDA/BLA Resubmissions since this activity could have a potentially significant impact on the workload. The activity factor for this review activity was computed using the same methodology that was used to calculate the five original application activity factors. However, discussions with the FDA team revealed this to be an imprudent modification due to the fact that FDA currently has multiple initiatives in place designed to achieve a higher percentage of first cycle review decisions and correspondingly reducing the number of application resubmissions. Inherent in this FDA initiative to increase the number of first-cycle decisions is an increase in the relative work and staff time spent during the initial review cycle. Therefore, the number of NDA/BLA resubmissions was expected to decline and as a result, not be relevant to the adjustments for changes in review activities. Consequently, this modification was excluded from the analysis.
The second modification involved excluding the denominator when calculating the activity factors for each of the five application review activities. The two denominators reflect the number of new NDA/BLA applications and the number of IND applications with activity. Excluding these two denominators in the calculations would avoid a risk of underestimating changes in workload by removing potential double-counting of submissions and further scaling down the year-to-year percent changes. It is important to mention that the workload has increased on a per-submission basis due to the increasing complexity of the applications received; however, an in-depth review led us to realize that certain review activities are not as directly related to the two denominators as other activities. For example, Annual Reports are largely independent of the number of new NDAs/BLAs received; whereas the number of NDA/BLA Meetings Scheduled is slightly more dependent on the number of new NDAs/BLAs received. Including a different NDA/BLA denominator in the calculation of the activity factors for both of these review activities would have potentially provided a better estimate of the relative change in workload. The ideal modification would involve excluding the denominator for some review activities and not others to avoid a double-counting of submissions. However, executing this modification with the existing calculations would have led to some review activities having a large activity factor percentage and other review activities having a small activity factor percentage. When totaled, those disproportionate activity factors (very large percentages and very small percentages) would have misrepresented the actual change in workload. As a result, a decision was made to exclude this modification from the alternative model as well.
The third modification involved weighting the time reporting percentages that were used to calculate each activity factor. The reason for applying weights to the time reporting percentages was because the existing 5-year time reporting averages (also used in the calculation) implicitly assume that the time reporting percentages are relative to each other on a yearly basis, which would result in accurately capturing workload changes. However, due to increases in FDA staff levels over time, the 5-year time reporting averages were only capturing average time spent in a particular year. Therefore, the time reporting weighting factors were generated based on average time spent year-to-year and fail to reflect relative time spent year-to-year. The proposed modification used weighted process costs to generate new time reporting weighting factors that were based on relative year-to-year changes for each review activity component. (The process costs for NDAs/BLAs and INDs, for FY 2006 through FY 2008, were obtained from the standard cost model.) The new weighting factors were then used in the remainder of the calculation that produced the new adjustments for changes in review activities for NDAs/BLAs and INDs. The new adjustments for changes in review activities were used to develop another Workload Adjuster (alternative model), shown in Figure 11 below. The specific modifications made to the existing calculations are included in Appendix D.
Figure 11 – Alternative Model
The adjustments for changes in review activities for NDAs/BLAs changed slightly from -0.55% in the baseline model to -0.46% in the alternative model. However, for INDs, the adjustment for changes in review activities remained the same for both models, with the percentage being 0.39%. The alternative model yielded a final Workload Adjuster value of 3.00%.
The alternative model was also subject to the Monte Carlo analysis technique, which is described at the end of this section.
A Sensitivity Analysis technique was intended to modify the Workload Adjuster inputs (the level of effort for each of the activity factors) to achieve a reasonable range of values for the final adjustment in workload. Evaluating this range of outputs would help determine whether or not the adjustments for changes in review activities were capturing the changes in workload. The inputs for the baseline model would have been treated as control data while the level of effort for each review activity would have been increased or decreased through a series of tests. The series of tests would have varied across univariate, bivariate and multivariate analysis. The results of this analysis would have been compared to feedback received from experienced FDA CDER and CBER personnel regarding hypothetical changes in the PDUFA-related workload. This comparison would have created a set of metrics to be used in making a final assessment on the reasonableness of the adjustments for changes in review activities. After discussing this approach with the FDA Project Officer and other key FDA team members, a decision was made not to proceed with this approach due to the lack of time available for FDA application review staff to respond to the questions due to application review workload demands.
Monte Carlo Analysis
The Monte Carlo methods are a class of computational algorithms that rely on repeated random sampling to generate results. These methods are often used when simulating physical and mathematical systems. In general, Monte Carlo methods are useful for modeling phenomena with significant uncertainty in their inputs. Because of the reliance on repeated computation and the generation of random or pseudo-random numbers, Monte Carlo methods are suited to calculation by a computer. These methods tend to be used when it is infeasible or impossible to compute an exact result with a deterministic algorithm.There is no single Monte Carlo method; instead, the term describes a large and widely-used class of approaches. However, these approaches tend to follow a particular pattern as described below:
- Define a domain of possible inputs to the model as random variables.
- Generate model inputs from the domain, and use those inputs with the model to perform a deterministic computation.
- Aggregate the results of a collection (large sample) of individual computations into the final result.
For the evaluation and study of the PDUFA IV Workload Adjuster, the Monte Carlo analysis could potentially add meaningful insight to the behavior of the model. We realize the limitations of this analysis given the limited number of years that the data is provided in the PDUFA IV Workload Adjuster, however, after discussing this approach with the FDA Project Officer and other key FDA team members, this approach was considered as possibly yielding value-added results that could be used for a final assessment. The Monte Carlo analysis was performed as described in the three steps above for the adjustments for changes in review activities. More specifically, the calculations for the five activity factors were subject to these techniques.
The first step of the Monte Carlo method was to define the domain for each input to the model as random variables. For example, one domain would be the collection of Labeling Supplements that are presented to FDA for review during FY 2008. This domain would then be defined by three parameters: the mean (average), the standard deviation, and the probability distribution function (e.g., normal, “bell curve”).
A random number generator in Excel was used to provide inputs to the random variables within the domain. When this process was repeated numerous times (e.g., 500, 1,000 or over 10,000 times) for a properly defined set of random variables, the behavior of the model was observed based on the outputs that were produced. In this study, the varying outputs of the analysis were the overall adjustments in workload (2.98% in the PDUFA IV Workload Adjuster).
The second step of the Monte Carlo method was to generate inputs from each domain for multiple simulations. This was accomplished using a random number generator for the seven model inputs associated with the five review activities and two application types (see Section 4.1).  The seven model inputs were Labeling Supplements; Annual Reports; NDA/BLA Meetings Scheduled; NDA/BLA Applications; SPAs; IND Meetings Scheduled; and IND Applications with activity. Each model input had seven data points associated with it (the 5-year averages from FY 2002 through FY 2007). The random variables for each of the model inputs were defined as the differences between the data point and the linear fit for the data (obtained from a regression analysis). The statistical properties of each random variable were described by the mean, standard deviation and a probability distribution function for each variable. The concept for random variables is depicted in Figure 12 below.
Figure 12 - Random Variables for Each Model Input
The simulated value in Figure 12 represents the FY 2008 value of each of the seven model inputs. The value was generated by first determining the formula for the linear fit through the data points. Since there is a degree of randomness associated with the value of each input, a random error was also added to the linear fit. 
The third step of the Monte Carlo method was to aggregate the results of a group of individual Monte Carlo simulations. These results provided a set of statistics which describe the behavior of the various review activities involved with the adjustments for changes in review activities. These statistics were used to evaluate the adjustments for changes in review activities across a reasonable range of values.
The Monte Carlo analysis was applied to the PDUFA IV Workload Adjuster (baseline model) as well as the alternative model. The results for both models are discussed below.
|Statistic||Baseline Model||Alternative Model|
|Number of Trials||10,000||10,000|
Figure 13 shows a summary of the results of the Monte Carlo analysis for both of the models. Each model was analyzed using the 5-year Average data (FY 2002 through FY 2008) simulated 10,000 times.
The first observation made from the results was that the Monte Carlo analysis generated a mean value of 3.11% for the final adjustment in workload. The output of the baseline model determined by FDA was 2.98%. The second observation is that the weighted time reporting modification used to develop the alternative model had little effect on the final output of the PDUFA IV Workload Adjuster. The Monte Carlo analysis observed that the mean for the alternative model involving weighted time reporting was also 3.11%. The 3.11% observed mean values is based on the assumption that the seven model inputs are normally distributed. Based on our analysis and discussions with key FDA personnel, the 2.98% output in the baseline model and the 3.00% in the alternative model was an expected result due to the reasons outlined below:
- The 5 year moving averages for the number of SPAs and IND meetings per Active IND increased at slower pace than the PDUFA III years.
- The 5-year moving averages for NDA/BLA annual reports and meetings had declined in FY 2008.
- There was an increase in the 5-year moving average of NDA/BLA labeling supplements, but the number of NDA/BLA applications had increased at a faster pace, leading to an additional decline in the workload adjustment reflected from the NDA/BLA review activity components.
- There was a relatively high number of NDAs received in FY 2007. In addition, the number of BLAs received by CBER increased by a factor of nearly 3 over FY 2006 and FY 2007 levels. The increase in the number of these applications was significantly higher relative to the trends found in the other years; thus, leading to a lower adjustment for FY 2008.
This translated in a decline or a slower increase in workload for FY 2008 as opposed to the increasing trend from the review activity components during the PDUFA III years.
Given the assumptions made above, one could conclude that there is a 50% chance that the FY 2009 adjustment in workload will be greater than 3.11%, and a greater than 50% chance that the FY 2009 adjustment in workload will be greater than 2.98% for the baseline model and a greater than a 50% chance that the FY 2009 adjustment will be greater than 3.00% for the alternative model. (The detailed results of the analysis for each model can be found in Appendix B.)
Given the assumptions made in the Monte Carlo analysis described above, we conclude that the adjustment for changes in review activities has a reasonable degree of stability in the scenarios examined.
The chart represents how linear fit data was calculated for the regression analysis. The simulated value is generated by determining the formula for the linear fit through the data points. The x-axis represents the complexity factor component values and the y-axis represents the recoded and simulated 5-year moving average values for FY 2002 through FY 2008. The recorded 5-year moving averages are plotted for each fiscal year and since there is a degree of randomness associated with the value of each input, a simulated 5-year moving average was also added as a random error. A best-fit linear line is shown based on the recorded and simulated 5-year moving average data points.