In this program we develop, demonstrate, and explore two categories of evaluations needed to determine the safety and effectiveness of digital pathology systems, including the whole slide imaging (WSI) device: 1) technical evaluations of the characteristics (especially color transfer) of a digital pathology scanning and display system, and 2) observer studies that evaluate the diagnostic abilities of pathologists interpreting digital pathology images. Our program will develop information on which technical characteristics are important, how they impact pathologist interpretations, and how they can be measured. Adequately outlining the technical characteristics can reduce the size, cost, and duration of clinical studies. Our program will also lead to clinical trial strategies that balance the enormous scope of general indications for use against feasibility, rigor, and meaning.
Key elements of our evaluation environment for digital and analog pathology: 1) Camera, 2) Microscope, 3) Computer controlled stage holding multiple slides, 4) Stage Controller, 5) Joystick for user control of stage. Not shown: computer and monitor.
Here we show the registration of the camera image of the real-time microscope view and the digital whole slide image. This allows the pathologist to evaluate the exact same locations and features of a slide in the digital and analog (microscope) domains.
With clinical collaborators we design, execute, and analyze reader studies where pathologists evaluate slides on the microscope, images on a computer display, or both. These studies inform us on the complexities of reader studies and give us the experience and expertise to review studies proposed by industry.
We are leading a working group in this area to engage the community, access expertise, and build consensus on the evaluation of technical and pathologist performance. Information can be found at https://nciphub.org/groups/wsi_working_group. Please check out the blog to see our announcements, past business and current plans.
One of our research efforts has been to design eeDAP, an Evaluation Environment for Digital and Analog Pathology (see description under “Public domain software”). We are currently pursuing a Medical Device Development Tool (MDDT) qualification of eeDAP as a “Clinical Outcome Assessment tool used in reader studies for whole slide imaging premarket submissions (PMA or 510k deNovo) to compare the accuracy or reproducibility of pathologist evaluations of digital images on a display to those of glass slides on a microscope. The pathologist evaluations of patient tissue are the clinical outcomes. The accuracy or reproducibility is the clinical outcome assessment; this assessment reflects image quality.” We are sharing the submission materials and feedback from the FDA with the WSI working group in order to expose them to this program and the device evaluation culture at the FDA. For more information go to https://nciphub.org/groups/wsi_working_group/blog/2017/01/time-to-submit-proposal-fdacdrh-medical-device-development-tool-for-digital-pathology and future updates to the WSI working group blog.
In research recently funded by the FDA Critical Path Initiative, we are conducting a multi-site study with pathologists from FDA, Johns Hopkins Hospital, and George Washington University Medical School, focusing on primary diagnosis of ovarian cancer. Through several targeted observer studies we are a) examining sources of discrepancies between pathologists for classifying different subtypes of ovarian cancer, b) developing a panel of histopathology patterns and related decision support tools toward improving pathologist performance for this task, and c) assessing pathologist performance with WSI and traditional optical microscopy (OM). Variables to be included in upcoming reader studies include the classification of histological subtype as well as characterization of histopathological patterns and associated criteria related to this task. The study will contribute to better characterization of WSI by providing valuable data to regulators regarding: a) measures of pathologist performance for both primary diagnosis and pattern quantification tasks, b) the link between section-based and case-based discrimination in digital pathology, and c) the effect of tumor heterogeneity on ovarian diagnosis and the development of computer aids.
Related work includes a completed project focusing on quantifying the histological expression of the protein HER2/neu, which is crucial for accurate treatment selection for breast cancer patients. Through our collaboration with the National Cancer Institute, we developed algorithms for quantitative assessment of HER2/neu, examined reproducibility of this biomarker to different image acquisition protocols, and conducted observer studies demonstrating benefit of computer-aided assessment on pathologist performance. Our studies provided evidence for the guidelines on Quantitative Analysis of HER2 Immunohistochemistry for Breast Cancer by the College of American Pathologists and American Society of Clinical Oncology.
We are now undertaking a project on the assessment of digital pathology and computer-aided diagnosis devices for the detection of metastatic ovarian cancer on lymph node tissue. The project aims to 1) assess WSI performance for the detection of ovarian micrometastases as a function of micrometastasis size and technical parameters and 2) develop assessment methodologies to address unique challenges presented by CAD systems for WSI.
In addition, our group participates in the Hematology and Pathology Devices Panel which hold meetings to seek the panel’s expert opinion and input on the replacement of Hematoxylin and Eosin stained glass slides and conventional microscopy by digital whole slide imaging, specifically for purposes of making routine surgical pathology diagnosis of a disease or condition.
Current funding sources
Brandon D. Gallas
Weijie Chen, Ph.D.
Wei-Chung Cheng, Ph.D.
Marios A. Gavrielides, Ph.D.
Jeff D. Seidman, M.D.
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, National Institutes of Health
- Leeds Teaching Hospitals NHS Trust and University of Leeds
- The George Washington University School of Medicine and Health Sciences, Department of Pathology
- Johns Hopkins Department of Pathology, Division of Gynecologic Pathology
- Olympus Microscope with mounted camera and computer controlled stage
- Zeiss Axio Scan.Z1 Whole Slide Imaging scanner
- Zeiss Microscope (on loan from collaborator) with mounted camera and computer-controlled stage
Public domain software
eeDAP: Evaluation Environment for Digital and Analog Pathology
eeDAP is a software and hardware platform for designing and executing digital and analog pathology studies where evaluation regions of interest (ROIs) in the digital image are registered to the real-time view on the microscope. This registration allows for the reduction or elimination of a large source of variability in comparing these modalities in the hands of the pathologist, i.e., the field of view, or tissue, under evaluation. In fact, the current registration precision of eeDAP allows for the presentation of the same individual cell in both domains. As such, a study can be designed where pathologists are asked to evaluate a pre-selected list of individual cells in Digital mode and Microscope real-time mode (MicroRT mode). eeDAP collects the pathologist evaluations while cycling through the list of cells. In Digital mode, the pathologist can enter the evaluations himself or herself. In MicroRT mode, an administrator enters the evaluations while confirming and maintaining a high level of registration precision. The paired observations allow for comparisons of WSI and traditional optical microscopy, using several forms of agreement or performance, allowing for a reference standard to be established.
Relevant standards & guidances
Selected peer-review publications
- Gavrielides, et al. Inter-observer variability in the classification of ovarian cancer cell type using microscopy: a pilot study, in SPIE Medical Imaging 2015: Digital Pathology, 2015.
- Treanor et al., Evaluating whole slide imaging: A working group opportunity, J Pathol Inform, 2015.
- Gallas et al., Evaluation Environment for Digital and Analog Pathology: a platform for validation studies, J Med Img, 2014.
- Gavrielides et al. Observer Performance in the Use of Digital and Optical Microscopy for the Interpretation of Tissue-Based Biomarkers, Analytical Cellular Pathology, 2014
- Keay et al., Reproducibility in the automated quantitative assessment of HER2/neu for breast cancer, Journal of Pathology Informatics, 2013.
- Keller et al., Quantitative assessment and classification of tissue-based biomarker expression using color content analysis, Archives of Pathology and Laboratory Medicine, 2012
- Gavrielides et al., Observer variability in the interpretation of HER2/neu immunohistochemical expression with unaided and computer-aided digital microscopy, Arch Pathol Lab Med, 2011.
- Masmoudi, et al., Automated quantitative assessment of HER-2/neu immunohistochemical expression in breast cancer, IEEE Transactions on Medical Imaging, 2009.