FY 2001 Statistical Issues of Diagnostic Modalities
- Software Development for Multivariate ROC Assessment of Diagnostic Modalities and Systems for Computer-Aided Diagnosis
- Tissue Characterization
- The Medical Image Perception Conference IX (MIPC IX)
- Factors That Affect the Performance of Self-Monitoring Glucose Meters
The clinical assessment of medical imaging systems is complicated by the great variability observed in readers in radiology. This variability leads to the necessity of a multivariate approach that includes the range of patient case difficulty, the range of reader skills, and correlations among the patients, readers, and imaging technologies under comparison. Thus, a primary goal of this program is to develop statistical methods for analyzing the performance of imaging systems within the context of reader variability. At the same time, new and increasingly sophisticated computer techniques for medical diagnosis are being developed by academia and industry to aid/augment the human reader in the interpretation of high-dimensional image data sets. A second major goal of this program is to develop study designs, objective measurements, and analytical methods for the laboratory and clinical assessment of imaging and other diagnostic systems, systems for computer-aided diagnosis (CADx) used in medical imaging, and stand-alone image-based computerized diagnostic modalities such as high-dimensional DNA micro-arrays (DNA chips).
The methodological tools being developed for analyzing the performance of imaging systems within the context of reader variability are referred to broadly as the multiple-reader, multiple-case (MRMC) receiver operating characteristic (ROC) paradigm. The paradigm is a multivariate analysis of the map of reader true-positive rates versus false-positive rates as a function of the variables listed above. A key question OST scientists are investigating is that of analyzing not only reader or system average performance but also the multivariate uncertainties that result from the finite sample of patients and readers.
The approach to the assessment of systems for computer-aided diagnosis and high-dimensional DNA arrays is an extension and application of the multivariate approach to ROC analysis. In the case of CADx and DNA arrays, the key question is that of analyzing the multivariate uncertainties that result from the finite sample of patients used to train the system, the finite sample of patients used to test the system and their interaction. The general subject of uncertainty analysis also addresses the classical problem of the "generalizability" of performance of a CADx or micro-array algorithm. OST makes use of advanced statistical tools for diagnostic decision-making under uncertainty, including classical Bayes' discriminants, neural-network architectures, and fuzzy logic, in studies of CAD algorithms and their performance.
In the last two years CDRH has been receiving an increasing number of premarket submissions for digital imaging modalities and modalities used for CADx, not only in imaging but also for clinical laboratory diagnostic tests. Statistical and analytical methods developed in the OST imaging group have been directly used to assist with both the design and the data analysis for several of these submissions, both in imaging and CADx. OST has played a significant role in the statistical evaluation of device submissions such as those for automated Pap smear readers, lung cancer, and breast cancer detection devices. A current emphasis is on developing a draft guidance document, in collaboration with scientists in OSB and ODE, to provide industry and academia with "best" and "acceptable" practices for the laboratory and clinical assessment of diagnostic devices. OST is also pursuing the potential for coordinating MRMC ROC clinical study designs in such a way as to optimize the expenditure of resources over the total product life cycle of an imaging technology—from university research, through pilot clinical trials and pivotal FDA studies, to confirmatory ACRIN (American College of Radiology Imaging Network) trials, through downstream cost/benefit studies of interest to public policy makers and insurers working at that higher level.
Software Development for Multivariate ROC Assessment of Diagnostic Modalities and Systems for Computer-Aided Diagnosis
Key words: computer-aided diagnosis, clinical trial design
This program addresses the development of analytical methodology and software for assessing diagnostic imaging modalities and systems for computer-aided diagnosis in the presence of multiple random effects (patient cases, image readers, and multiple correlations). OST scientists have previously developed methods and software for the efficient design and sizing of clinical trials within this paradigm in imaging. OST has now extended this work to include the effects of the finite sample sizes used for training and testing in the development of algorithms for computer-aided lesion detection and classification. This work is immediately relevant to the design and sizing of databases for use by industry sponsors of such systems and public-policy-making bodies such as the National Cancer Institute and the FDA, who either sponsor such databases (NCI) or may depend on them as part of submissions from industry sponsors of such computer algorithms (FDA).
Key words: ultrasound, magnetic resonance imaging, spectroscopy, tissue characterization
This project involves applying quantitative methods for tissue characterization using ultrasound and magnetic resonance. An understanding of the physics of these modalities and the statistical issues involved in multi-parameter tissue characterization is important in reviewing diagnostic devices. OST continued to participate with the American Institute of Ultrasound in Medicine Technical Standards Committee Working Group on Backscatter Measurements to develop a standard for ultrasonic backscatter measurements in tissue. OST continued to participate in developing the ASTM "Standard Test Method for Evaluation of MR Image Artifacts from Passive Implants." This standard was voted on and approved recently and is now an official ASTM standard. Researchers are still collecting ultrasound and pathology data from ex vivo prostates at the University of Vermont to assess the usefulness of combining clinical, sonographic, and elastographic features to improve the detection of prostate cancer. Software was developed to compute backscatter coefficient, texture, and elastographic parameters from prostate samples.
Key words: medical imaging, radiologists, assessment methodology
The ninth biennial conference on Medical Image Perception was held at the Airlie Conference Center in Warrenton, VA from September 20-23, 2001. This conference was organized by researchers in OST’s Division of Electronics and Computer Science. Over 80 conferees and speakers came from 6 European countries and the United States. The topics of the conference were reader strategies, variability, and error in diagnostic imaging; statistical assessment methodologies; model observers and image quality assessment; computer-aided diagnosis; and problems in medical image display. There were invited tutorials, proffered papers, and workshops for each topic. Representatives from CDRH, the National Cancer Institute, the American College of Radiology Imaging Network (ACRIN), academia, and industry provided state-of-the-art science and methodology that is useful for the process of consensus discovery and the development of FDA guidance for industry submissions in these areas of diagnostic medicine.
Key words: glucose, diabetes
Results from this project revealed that operator variability, meter precision, and inter-meter difference were minor contributors to the inaccuracy of blood glucose meters. The main factor appeared to be systematic errors in the meter calibration. Meter strip lot-to-lot variation was found to be up to ± 10% for the four meters. The 14 test assay kits from the hexokinase comparative method had 0.2 to 4.8% deviation from the true glucose values certified for NIST’s standard SRM sera.
These findings have provided the FDA first hand independent information to support better guidelines for the manufacturers’ submissions and also provide reliable information regarding the performance limits of the glucose meters.