Key words: classifiers, ROC analysis, CADx, analysis of variance, tech support, research
OST scientists have collaborated with the University of Michigan over the last 2 years to study the problem of generalizability of the results of performance assessment of systems for computer-aided diagnosis (CADx). Generalizability here refers to the issue of how applicable the measured uncertainties in performance assessment will be when the clinical trial is repeated and new patients are used to train and assess the CADx algorithm. All performance assessment is carried out within the context of Receiver Operating Characteristic (ROC) curve analysis in which test sensitivity is mapped as a function of its specificity. In collaboration, OST has developed computer simulations of the process of training and testing classical and neural-network CADx algorithms that fuse up to 16 image or laboratory test features. From the simulations, OST has developed models of the bias, variance, and generalizability of the performance assessment results of such emerging modalities. Classical statistics as well as several contemporary resampling strategies are included in the models. This work falls within the paradigm of analysis of variability of the performance assessment of diagnostic modalities in general, an area OST will continue to pursue.