U.S. flag An official website of the United States government
  1. Home
  2. Medical Devices
  3. Science and Research (Medical Devices)
  4. CDRH Research Programs
  5. Statistical Assessment Methodology for Diagnostics and Biomarkers
  1. CDRH Research Programs

Statistical Assessment Methodology for Diagnostics and Biomarkers

Contact

Frank Samuelson, Ph.D.
Weijie Chen, Ph.D.

 

Score distributions and ROC curve

Upper left: densities of reader scores; Lower left: cumulative distributions of reader scores; Right: ROC curve, a plot of one distribution against another.


 An antler plot of AUC

This plot shows the average AUC performance of a classifier on a new data set (open symbols) and on the same data set used to train the classifier (filled symbols) as a function of the dimensionality (D) of the data and the sample size used to train the classifier.

Summary

The ability of the FDA to assess the effectiveness of imaging devices and predictive biomarkers depends strongly on the FDA’s ability to evaluate (1) study designs for these devices, and (2) the statistical significance of data collected from these studies.

The program aim is to develop mathematical and statistical methodologies for designing, analyzing, and evaluating studies of diagnostic devices in imaging, predictive biomarkers, and computer algorithms based on imaging or biomarker data. Our research includes methodologies for evaluating the effectiveness of these devices at various levels: image quality assessment with model observers; the assessment of the device’s stand-alone performance with clinical data; and, when necessary, the assessment of physicians’ performance in using the device for a diagnostic task (i.e., reader studies). This includes methods for establishing the effectiveness of biomarkers or algorithms from survival outcome data and two-class receiver operating characteristic (ROC) data, which have a long history at the FDA. One emphasis of our research is to appropriately account for various sources of variability in an evaluation study for valid statistical inference; for example, case variability and reader skill are two important sources of variability.

In addition, the program examines data from published studies and studies submitted to the FDA to guide our methodological research and to produce recommendations for future FDA submissions.

Personnel

FDA Staff:
Frank Samuelson, Ph.D.
Weijie Chen, Ph.D.
Brandon Gallas, Ph.D.
Berkman Sahiner, Ph.D.
Nick Petrick, Ph.D.
Kyle Myers, Ph.D.

ORISE Fellow:
Zhipeng Huang, Ph.D.

External collaborators

Steve Hillis, Ph.D., University of Iowa
Craig Abbey, Ph.D,, University of California Santa Barbara
Le Kang, Ph.D., Virginia Commonwealth University

Public domain software

iMRMC Software disclaimer icon
The primary objectives of iMRMC applications are to assist investigators with analyzing and sizing multi-reader multi-case (MRMC) reader studies that compare the difference in rates of performance or agreement (binary data) or the difference in the area under ROC curves (AUCs from ROC data).

Assessment of Classifiers disclaimer icon
These are software tools for the evaluation of classifiers/algorithms that predict binary or survival time outcomes.

IQmodelo: Statistical Software for Image Quality Assessment with Model Observersdisclaimer icon

Selected peer-review publications