Comprehensive Introduction to the Evaluation of Neural Networks and other Computational Intelligence Decision Functions: Receiver Operating Characteristic, Jackknife, Bootstrap and other Statistical Methodologies

Sponsor: International Neural Network Society and the IEEE Computational Intelligence Society

Event name: 2011 International Joint Conference on Neural Networks

Date: July 31, 2011

Title: "Comprehensive Introduction to the Evaluation of Neural Networks and other Computational Intelligence decision functions: Receiver Operating Characteristic, Jackknife, Bootstrap and other Statistical Methodologies"

Presenter: David Brown

Course Description: The course will cover the broad range of performance metrics that go beyond the simple calculation of success rate or accuracy on a limited test set. These include sensitivity, specificity, positive and negative predictive value, receiver operating characteristic (ROC) curve, and area under the ROC curve (AUC). Particular attention will be paid to the ROC curve formalism and its extension beyond the binary categorization problem. Both parametric and nonparametric uncertainty calculation techniques will be discussed, with particular emphasis on jackknife and bootstrap methods. The importance of good data hygiene in maintaining complete separation of training and test data will be illustrated, and the significance of understanding training as well as test-set variability (and human observer variability where appropriate) will be examined.

David Brown is a veteran of 30 years of research in the field of medical imaging and computer-aided diagnosis performance evaluation.  He is a scientist at the US Food and Drug Administration with responsibility for the assessment and approval of imaging and CI devices.