• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

For Industry

  • Print
  • Share
  • E-mail

Data Collection And Validation

Table of Contents

Approach and Results

Data validation efforts associated with the adjustments for changes in review activities involved comparing the PDUFA IV Workload Adjuster data with the data provided from CDER and CBER for the following seven data elements: 1) Labeling Supplements; 2) Annual Reports; 3) NDA/BLA Meetings Scheduled; 4) NDA/BLA Applications; 5) SPAs; 6) IND Meetings Scheduled and 7) IND Applications. The data provided by CDER and CBER included submission counts for NDAs/BLAs and INDs for PDUFA years 2002 through 2008. CDER and CBER also provided activity factor counts for the five review activities mentioned above for PDUFA years 2002 through 2008. The CDER and CBER time reporting data for FY 2006 through FY 2008 was deemed out of scope of this evaluation due to the sensitive nature of the data.

Data requests were submitted to CDER and CBER to obtain the counts associated with the seven data elements. The data was obtained by each center from the systems where the information is stored. The data validation efforts are summarized in Figure 3 below for each column in the PDUFA IV Workload Adjuster.

Column Data Element Data Validation
1 Submission Counts:
- Labeling Supplements
- Annual Reports
- NDA/BLA Meetings Scheduled
- NDA/BLA Applications
- SPAs
- IND Meetings Scheduled
- IND Applications
Validated against data received from CDER and CBER
2a Submission Counts:
- Labeling Supplements
- Annual Reports
- NDA/BLA Meetings Scheduled
- NDA/BLA Applications
- SPAs
- IND Meetings Scheduled
- IND Applications
2b Activity Factor Data:
- Labeling Supplements
- Annual Reports
- NDA/BLA Meetings Scheduled
- SPAs
- IND Meetings Scheduled
2c Derived from Columns 2a and 2b Not Applicable
3 Derived from Columns 1 and 2c Not Applicable
4 Weighting Factors:
- NDAs/BLAs
- INDs
Validated against weighting factors obtained from the Standard Cost Model. The Standard Cost model is beyond the scope of this project, therefore no validation is performed on any of the Standard Cost Model data
5 Derived from Columns 3 and 4 Not Applicable

Figure 3 – Summary of Data Validation Efforts for the PDUFA IV Workload Adjuster

 

The results of the data validation for each of the seven data elements are shown in the tables below (Figures 4 through 10), and is followed by an explanation of any variances found.

Labeling Supplements
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 761 74 835 815 20 2.40%
2003 773 93 866 836 30 3.46%
2004 1,011 70 1,081 1,040 41 3.79%
2005 776 50 826 787 39 4.72%
2006 885 49 934 902 32 3.43%
2007 1,024 53 1,077 1,037 40 3.71%
2008 912 48 960 914 46 4.79%

Figure 4 – Data Validation Results for Labeling Supplements

 

Annual Reports
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 2,672 226 2,898 2,897 1 0.03%
2003 2,651 275 2,926 2,925 1 0.03%
2004 2,575 187 2,762 2,761 1 0.04%
2005 2,651 158 2,809 2,808 1 0.04%
2006 2,581 191 2,772 2,771 1 0.04%
2007 2,676 194 2,870 2,868 2 0.07%
2008 2,672 222 2,894 2,891 3 0.10%

Figure 5 – Data Validation Results for Annual Reports

 

NDA/BLA Meetings Scheduled
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 342 74 416 416 0 0.00%
2003 412 80 492 492 0 0.00%
2004 366 55 421 399 22 5.23%
2005 328 42 370 346 24 6.49%
2006 328 58 386 372 14 3.63%
2007 285 44 329 316 13 3.95%
2008 238 62 300 296 4 1.33%

Figure 6 – Data Validation Results for NDA/BLA Meetings Scheduled

 

NDA/BLA Applications
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 90 9 99 99 0 0.00%
2003 113 8 121 115 6 4.96%
2004 129 6 135 138 -3 -2.22%
2005 105 9 114 117 -3 -2.63%
2006 128 4 132 133 -1 -0.76%
2007 109 15 124 116 8 6.45%
2008 141 4 145 138 7 4.83%

Figure 7 – Data Validation Results for NDA/BLA Applications

 

SPAs
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 204 0 204 206 -2 -0.98%
2003 295 17 312 314 -2 -0.64%
2004 329 13 342 341 1 0.29%
2005 384 10 394 397 -3 -0.76%
2006 423 12 435 437 -2 -0.46%
2007 421 11 432 433 -1 -0.23%
2008 387 10 397 403 -6 -1.51%

Figure 8 – Data Validation Results for SPAs

 

IND Meetings Scheduled
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 758 283 1,041 1,040 1 0.10%
2003 1,016 316 1,332 1,338 -6 -0.45%
2004 1,338 196 1,534 1,537 -3 -0.20%
2005 1,649 196 1,845 1,819 26 1.41%
2006 1,733 200 1,933 1,942 -9 -0.47%
2007 1,670 191 1,861 1,861 0 0.00%
2008 1,533 210 1,743 1,767 -24 -1.38%

Figure 9 – Data Validation Results for IND Meetings Scheduled

 

Active INDs
PDUFA
Year
CDER
Data

[a]
CBER
Data

[b]
Total
CDER & CBER

[a] + [b] = [c]
PDUFA IV
Workload
Adjuster

[d]
Variance

[c] - [d] = [e]
Percentage
Difference

[e] / [c] = [f]
2002 3,659 1,320 4,979 4,982 -3 -0.06%
2003 3,764 1,352 5,116 5,123 -7 -0.14%
2004 4,766 888 5,654 5,661 -7 -0.12%
2005 5,047 843 5,890 5,900 -10 -0.17%
2006 5,385 829 6,214 6,252 -38 -0.61%
2007 5,168 824 5,992 5,843 149 2.49%
2008 5,521 864 6,385 5,832 553 8.66%

Figure 10 – Data Validation Results for IND Applications

 

According to the FDA, the discrepancies found between values that were used in the PDUFA IV Workload Adjuster and the values provided to us by the FDA for verification purposes were due to querying the source databases at different points in time. Variances may occur in the data because of the dynamic nature of the source data and constant updates to the data. Properties and attributes of a given application may be adjusted as necessary during the course of a review and any updates to that application are then made within the system accordingly. Examples affecting submission counts and complexity factors may include:

  • Data Entry Lag: It may take a few days from the receipt of an application to its entry into the database. If the database is queried by a user during this window, the results of the query may underestimate the total submission count for that application type. For example, if three new submissions are received on June 30th (the cut-off date for the PDUFA year used in the Workload Adjuster), they may not be captured in a query that is run immediately after that date. FDA analysts try to mitigate this risk by deferring the database query until the last possible moment.
  • Data Update Lag: Since individual application submissions include many fields that need to be accurately flagged, these critical flags may not be known until substantial review of the submission package has been performed; this process can take up to two months (for example, an NDA needs to be categorized as a New Molecular Entity (NME) or a non-NME, and also as to whether or not it includes clinical data). Even after adding flags there may be some time before FDA has time to update the submission in the system because of competing priorities. If the database is queried while these submissions are being processed, the results of the query may not represent the final number of submissions until all entries have been updated. It is important to note that the data update lag will affect the NDA/BLA sub-types but it will not have a significant impact on the overall count.
  • FDA Data Reporting Method:  Under the current methodology for collecting and reporting the data used in the PDUFA workload adjuster, the data is collected for each 12 month period ending on June 30th, and recorded in mid July.  To make periods from earlier years comparable with later years, the amounts from earlier years are not restated - they always remain fixed at the values initially reported for each year. Naturally, restating these values for purposes of our evaluation produced values that generally differed slightly from the values initially recorded and repeated here without change for the associated year.
  • Human Entry Errors: Errors may occur by the user when the application is entered into the system or updated. Multiple levels of quality assurance are performed on a frequent basis, but resource constraints make this difficult at times.
  • New Data Systems: During the past few years, FDA’s CDER has been moving towards a new consolidated database for submission types, namely, the Document Archiving, Reporting, and Regulatory Tracking System (DARRTS). CDER IND applications were moved to DARRTS in 2007 and CDER NDA applications are planned for migration in 2009. This new database features a greater reliance on automated updates to data based on clearly defined business rules which reduces the need for manual data entry (which would help minimize data entry and update lags as well as human entry errors). This database also includes a robust trace-back feature that will document changes made to the database. However, moving to a new database does introduce several issues that may affect the stability of the data:
    • When data are migrated from one system to another, there is a possibility that data may be lost or altered during the move. FDA exercises extensive quality assurance measures to maintain data integrity during the migration.
    • Any new database structure will also require the development of new queries that would provide the same results that were obtained from the older database. FDA has experienced challenges building comparable queries using the new Business Object query interface. Training is being provided to support the development of queries capable of navigating the intricacies of the data being queried. Until all data migration issues have been addressed, some variability in the query results will be expected.

We performed a simulation analysis of the adjustments for changes in review activities for the PDUFA IV Workload Adjuster by utilizing the data received in support of our validation effort (see the 2nd, 3rd and 4th columns of figures 4-10) as new inputs. Based on the results of the simulations we believe that the data variances do not represent a significant impact to the adjustments for changes in review activities. However, we believe that FDA could implement additional procedures to reduce these data variances (see observations section below).

Approach and Results