U.S. flag An official website of the United States government
  1. Home
  2. Medical Devices
  3. Science and Research | Medical Devices
  4. iMRMC: Software for the statistical analysis of multi-reader multi-case reader studies
  1. Science and Research | Medical Devices

iMRMC: Software for the statistical analysis of multi-reader multi-case reader studies

Catalog of Regulatory Science Tools to Help Assess New Medical Devices

Technical Description

The primary objective of the iMRMC statistical software is to assist investigators with analyzing and sizing multi-reader multi-case (MRMC) reader studies that compare the difference in the area under Receiver Operating Characteristic curves (AUCs) from two modalities.  The iMRMC application is a software package that includes simulation tools to characterize bias and variance of the MRMC variance estimates.

The core elements of this application include the ability to perform MRMC variance analysis and the ability to size an MRMC trial, including study designs that are not "fully-crossed."

  • The core iMRMC application is a stand-alone, precompiled, license-free Java application and source code. It can be used in GUI mode or on the command line.
  • There is also an R package that utilizes the core Java application. Examples for using the programs can be found in the R help files.

Intended Purpose

The iMRMC package analyzes data from MRMC studies. MRMC stands for Multiple Readers and Multiple Cases (MRMC). MRMC studies are often imaging studies where clinicians (readers) evaluate patient images (cases). The MRMC methods apply to any scenario in which clinicians interpret data to make decisions. 

The iMRMC package calculates the reader-averaged area under the curve (AUC) of the receiver operating characteristic curve (ROC). AUC is a diagnostic performance measure. Additional functions analyze other endpoints, such as binary performance and score differences. This package also estimates variances, confidence intervals, and p-values. These uncertainty characteristics are needed for hypothesis tests to size and assess the efficacy of diagnostic imaging devices and computer aids, including artificial intelligence-based devices. 

Many imaging studies are designed so that every reader reads every case in all modalities, or in a fully-crossed study. In this case, the data is cross-correlated, and we consider the readers and cases to be cross-correlated random effects. An MRMC analysis accounts for the variability and correlations from the readers and cases when estimating variances, confidence intervals, and p-values. In addition, the functions in this package can treat arbitrary study designs and studies with missing data, not just fully-crossed study designs. 

The package permits industry statisticians to use a validated statistical analysis method without having to develop and validate it themselves.


Please cite these references when you use the iMRMC software.


The tool has been characterized through simulations (bias and variance of the MRMC variance estimates) and was found to have similar characteristics as other methods in the literature. The simulations model the data collected in reader studies: the diagnostic scores from readers interpreting images. The reader-study data yields a reader-averaged AUC and an MRMC variance estimate for each simulated reader study. Repeating this many times using a Monte Carlo simulation allows for the estimation of the bias and variance of the MRMC variance estimates.

The testing is summarized in the following articles:


The tool may produce negative variance estimates for studies where the dataset is small.

Supporting Doumentation

Tool websites:

Supplementary materials:

Related FDA Product Codes

Related FDA product codes include:

  • KPS: System, Tomography, Computed, Emission
  • LLZ: System, Image Processing, Radiological
  • PAA: Automated Breast Ultrasound
  • POK: Computer-Assisted Diagnostic Software For Lesions Suspicious For Cancer
  • QDQ: Radiological Computer Assisted Detection/Diagnosis Software For Lesions Suspicious For Cancer
  • QPN: Software Algorithm Device To Assist Users In Digital Pathology

Related Work

Current research


Tool Reference

In addition to citing relevant publications, please reference the use of this tool using DOI: 10.5281/zenodo.6628838.

For more information

Back to Top