SUMMARY OF SAFETY AND EFFECTIVENESS DATA
A. GENERAL INFORMATION
Device Generic Name: System, Image Processing, Radiological
Device Trade Name: ImageChecker® CT CAD Software System
Applicant’s Name and Address: R2 Technology, Inc.
1195 W. Fremont Avenue
Sunnyvale, CA 94087
PMA Number: P030012
Date of Panel Recommendation: February 3, 2004
Date of notice of approval to Applicant: TBD
B. INDICATIONS FOR USE
The ImageChecker CT CAD Software System is a Computer-Aided Detection (CAD) system designed to assist radiologists in the detection of solid pulmonary nodules during review of multi-slice CT (MSCT) scans of the chest. It is intended to be used as a “second reader,” alerting the radiologist – after his or her initial reading of the scan – to regions of interest (ROIs) that may have been initially overlooked.
In this way, the system reduces the chance of observational lapses by the reader due to fatigue, distraction, satisfaction of search, or other recognized causes of a sub-optimal review. Thus, the system increases the chances that a clinically significant nodule will be not be overlooked, resulting in improved patient care (e.g., detection at an earlier, more treatable stage).
C. CONTRAINDICATIONS
There are no contraindications for the use of this device.
D. WARNINGS AND PRECAUTIONS
Warnings and Precautions for use of the device are stated in the attached product labeling.
E. DEVICE DESCRIPTION
System
Overview
The
ImageChecker CT System is an image analysis and visualization system designed
to assist radiologists in the review of multi-slice CT (MSCT) exams of the
chest and in the detection of solid pulmonary nodules.
The ImageChecker CT
System is a combination of dedicated computer software and hardware. The system is comprised of the ImageChecker
CT Workstation (K023003) and the ImageChecker CT CAD software that is the
subject of this PMA filing. The two
components are related as indicated in Figure 1.
Figure
1. Overview
of the ImageChecker CT system illustrating the two devices that comprise the
system and the relevant hardware and software components.
Developed by R2 Technology, Inc., the ImageChecker
CT Computer-Aided Detection (CAD) System is intended to be used as an
adjunctive aid for radiologists interpreting MSCT scans of the chest –
providing detection of solid parenchymal densities that may represent
clinically significant nodules.
The ImageChecker CT CAD software does this by
applying proprietary signal processing algorithms to the large digital datasets
generated during scanning. These algorithms, in effect, analyze the complete
set of images and search for findings with features suggestive of a solid
pulmonary nodule. The system conveys information regarding “candidate” nodules
to the workstation for consideration by the reader.
The ImageChecker CT CAD Software System is not
intended to be used principally in a lung-cancer screening environment, where
the vast majority of scans would be expected to be free of clinically
significant disease. Rather, it is intended as a “back-up” nodule
identification and reporting system for radiologists reading everyday
diagnostic chest CTs – many of which will have other pathologic processes
competing for their attention.
After the initial study
acquisition by the CT scanner, copies of the study are sent automatically to
the review workstation for the user to review as well as to the processing
server for segmentation and CAD analysis. When the processing server finishes,
it sends a report to the workstation.
All image and information exchanged between the components of the system
and external devices (such as CT acquisition devices, PACS systems) are
controlled using standard DICOM (Digital Imaging and Communications in Medicine)
protocols.
The following sections give
a brief overview of the individual devices that comprise the ImageChecker CT
System.
ImageChecker CT Workstation
The ImageChecker-CT
Workstation is indicated for use as a general medical imaging workstation, and
is used to receive, store, transmit, and display images from a Multislice CT
scanner (MSCT). The FDA cleared the
workstation for marketing on 11/4/2002 (K023003) as ImageChecker CT Model
LN-500. The ImageChecker CT Workstation
combined with the ImageChecker CT CAD Software that is the subject of this PMA
submission will be distributed as the Model LN-1000.
The ImageChecker-CT
Workstation LN-1000 is comprised of two off-the-shelf personal
computers, one with a Linux-based operating system (OS) and one with a
Microsoft Windows-based OS, and a display monitor.
The processing
software performs several functions:
· Receives MSCT exams using the DICOM standards.
· Takes the CT images and segments different anatomical structures into normal structures (e.g. vessels) and other composite features.
· Stores the location and characteristics of the segmented composite features in a DICOM Structured Report object.
The workstation display
software provides tools for the radiologist that aid in the review process. During the review, the radiologist instructs
the display software by means of a standard keyboard and mouse. The images and findings are communicated to
the radiologist by means of a color Flat
Panel
display. When the user completes his or
her review, the system makes a summary report available that lists any findings
and measurements associated with the study. The user can also print this
summary.
The
workstation is also able to display the findings that are identified by the CAD
software (see next section). The
radiologist using the workstation is able to view the CAD findings – after a
preliminary review of the study – using a simple button press on the user
interface. The user can then recheck
the areas that the CAD software points out to determine if a change in his or
her interpretation of the exam is warranted.
ImageChecker CT CAD software
The ImageChecker CT
CAD software is an adjunctive software package that analyzes the CT images
after they have been pre-processed by the workstation software and identifies
regions of interest that may be solid pulmonary nodules. The regions of interest are identified by
means of the propriety signal processing algorithms that analyze the images and
search for findings with features suggestive of a solid pulmonary nodule (see
Figure 2).
The location
information about these identified regions of interest is sent to the
workstation using a DICOM CAD Structured Report (DICOM CAD SR).
Figure 2. Examples of CAD-marked pulmonary nodules
F. ALTERNATIVE PRACTICES
AND PROCEDURES
Currently, there is no alternative to assist radiologists
in the detection of lung nodules on multi-slice CT chest scans other than
double reading.
G. MARKETING HISTORY
The ImageChecker CT CAD Software System was launched for commercial sale in the European Union at the European Congress of Radiology in March 2003.
H. POTENTIAL ADVERSE
EFFECTS OF DEVICE ON HEALTH
There are no known direct risks to safety or health caused by, or related to, the use of the device. The indirect risks are that the device may fail to identify and mark some actionable lesions and will mark some lesions that do not require further action.
I. NON-CLINICAL STUDIES
There are no non-clinical studies to report for this device.
J. CLINICAL STUDIES
R2 Technology, Inc. has conducted two pivotal clinical studies to evaluate the safety and effectiveness of the ImageChecker CT CAD Software System. The studies were based on a retrospective case collection project involving multiple clinical sites in various regions across the U.S.
Objectives
The first pivotal study was designed to generate a “truth” set of cases containing solid pulmonary nodules, as well as cases with no nodules, to be used as a reference truth for subsequent studies. The second pivotal study, conducted in two phases, was an Observer/ROC (Receiver Operating Characteristic) Study, designed to measure the performance enhancement of radiologists using the System.
Sites and Cases for Case Collection Project and Subsequent Studies
Five (5) regionally diverse sites contributed 151 cases to the study; 2 sites in the Northeast, and 1 site each from the South, the Midwest, and the West. Of these sites, 3 were private imaging centers and 2 were academic medical centers. These sites are listed in Table 1.
|
Table 1. Sites contributing nodule and non-nodule
CT chest cases in the Case Collection Project and subsequent clinical studies |
||||
|
Name |
State |
Number of Nodule-present Cases Used in Studies |
Number of Non-nodule Cases Used in Studies |
Total Cases |
|
Atlantic Medical Imaging |
NJ |
11 |
10 |
21 |
|
MRI &
CT Diagnostics |
VA |
15 |
23 |
38 |
|
South Jersey Radiology |
NJ |
14 |
35 |
49 |
|
University of Iowa |
IA |
18 |
9 |
27 |
|
UC, San Francisco |
CA |
5 |
11 |
16 |
|
Total |
|
63 |
88 |
151 |
All cases were culled consecutively from the sites’ digital archives according to the inclusion and exclusion criteria identified in the case collection protocols.
The nodule-present cases collected included only those in which a diagnosis of cancer, either primary lung cancer or an extrathoracic neoplasm, had been documented. Other co-existing disease processes resulting in the formation of nodules (e.g. TB, histoplasmosis, rheumatoid lung) were allowed, as were cases containing other “background” pathology such as lobar pneumonia, emphysema, and heart failure.
A total of 63 nodule-present cases dating from November 2001 through December 2002 were included in the studies. The study population consisted of 56% females and 44% males, with a median age of 66 and a range of 20-86. The malignancy consisted of primary lung cancer in 24 (38%) of these cases, and documented extra-thoracic primary cancer with suspected metastatic disease to the lung in the remaining 39 (62%) cases.
The “normal” cases collected were those in which no nodules were deemed to be present by the principal investigator at each site. Other disease processes could be present, including the presence of pulmonary masses (>3cm). Histories of cancer, radiation therapy, or even previous thoracotomy, were allowed. In all, 88 nodule-absent cases dating from June 2002 through December 2002 were included in this retrospective case collection.
STUDY #1 – IDENTIFICATION OF REFERENCE TRUTH
The objective of this study was to generate a “truth” set of unanimous actionable nodules, as identified by a panel of 3 experienced radiologists, to serve as a reference truth for all subsequent studies.
To achieve this objective, multiple panel sessions were scheduled in which 3 radiologists independently read a variable number of cases (min = 12, max = 25) until all 151 study cases had been interpreted by all 3 readers.
The three panelists identified a total of 142 findings in the 151 cases that met the size (4-30mm) and peak density (> -100HU) requirements, and which all three panelists agreed were actionable. The presence or absence of at least one of these findings in a quadrant was used as the reference truth for study #2 below.
The findings ranged in size from 4-28mm. It is of note that the majority of these findings were between 4mm and 8mm in diameter (46%, 66/142), with the largest categories being the 5 – 6 mm (15%, 21/142) and 6 – 7 mm (15%, 22/142) findings.
STUDY #2 – EFFECT OF CAD SYSTEM ON IMPROVING ACCURACY OF IDENTIFICATION OF ACTIONABLE NODULES
The objective of this study was to demonstrate that review of CAD output improves performance of radiologists reviewing MSCT with respect to their ability to accurately identify actionable nodules. The study employed a Receiver Operating Characteristic (ROC) methodology that has become the standard in the radiology community for evaluating imaging modalities. The study was conducted in two phases, with the first phase using a smaller case set (n=32 cases) and the second phase using a larger case set (n=90 cases). The study methods and analysis were the same for both phases of the study.
For Phase I of the study, thirty-two (32) cases were randomly selected from the 151 cases evaluated in study #1. These cases were each divided into 4 quadrants, yielding a total of 128 regions for evaluation. Each of 15 radiologists independently reviewed the 128 quadrants, first without computer-aided detection (CAD) and then immediately with CAD.
For Phase II of the study, ninety (90) cases were randomly selected from the remaining 119 cases (151 – 32 used in Phase I). Again, the cases were divided into 4 quadrants, yielding 360 regions for evaluation. Each of 15 radiologists (9 returning radiologists from Phase I, 6 new radiologists) independently reviewed the 360 quadrants, first without computer-aided detection (CAD) and then immediately with CAD
For both phases, each reader rated each quadrant on a 0-100 “actionability scale” as to his or her level of confidence that the quadrant contained at least one actionable nodule. Ratings were provided both before and after viewing the CAD marks.
For purposes of measuring reader performance, quadrants were defined as “actionable” if the reference truth consensus panel described in study #1 unanimously agreed that at least one of the findings in the quadrant was actionable.
An area under the receiver operator curve (Az) was computed for each of the 15 radiologists before and after CAD. The average curve is shown below in Figure 3 for the Phase I study and Figure 4 for the Phase II study. In both cases the area under the ROC curve increased with the use of CAD. If the full plot is viewed as a unit square, the area separating the two curves in the Phase I study is 0.014 and the area separating the two curves in the Phase II study is 0.024. Although the second study shows greater separation and a slightly smoother pair of curves, the basic shape of the curves in the two studies is similar.
Az post-CAD = 0.8880 Az pre-CAD = 0.8739 p = 0.108
Figure 3. Average ROC curve showing pre-CAD (dashed line) and post-CAD (solid line) for Phase I (32 case, 128 quadrant) study.
Az post-CAD = 0.9047 Az pre-CAD = 0.8808 p = 0.003
Figure 4. Average ROC curve showing pre-CAD (dashed line) and post-CAD (solid line) performance for Phase II (90 case, 360 quadrant) study.
The primary analysis of statistical significance was based on the Dorfman-Berbaum-Metz (DBM) ANOVA-after-jackknife approach[1] (adapted for the fact that the quadrants are “clustered” data) and the results are presented in Table 2 below.
In the Phase II study (Figure 4), the average reader improvement in AZ (estimated using the ANOVA-after-jackknife) was 0.0240 ± 0.0077 (p=0.0033) with a 95% confidence interval of (0.0084, 0.0395). By comparison, in the Phase I study (Figure 3), the average reader improvement in Az (estimated using the ANOVA-after-jackknife) was 0.0140 ± 0.0084 (p=0.1083) with a 95% confidence interval of (-0.0033, 0.0313). The combined 122 case study demonstrated that the average reader improvement in AZ (estimated using the ANOVA-after-jackknife) was 0.0209 ± 0.0062 (p=0.0013) with a 95% confidence interval of (0.0085, 0.0333). Thus both the Phase II study by itself and the pooled study results showed a statistically significant improvement in the area under the ROC curve with the use of CAD.
The secondary analysis of statistical significance was conducted to determine the dependence of the study results on the use of the consensus reference truth from study #1 as the gold standard for the ROC study. In general, a reference truth based on a consensus panel assessment of actionability is weaker than one based on a more invariant gold standard such as biopsy. To examine the effect of variability in the unanimous three-panelist reference truth, two-panelist reference truths and single-panelist reference truths were constructed from the data collected in study #1.
Implementing this variable truth is difficult within the framework of the ANOVA-after-jackknife analysis, therefore the secondary analysis employs a bootstrap analysis[2]. The bootstrap is a computationally-intensive non-parametric method that allows complex analyses to be repeated many times using different randomly generated datasets (all based on the original data) to approximate the variability that would occur if the entire study were repeated many times. As a test of the validity of the bootstrap mechanism, the analysis was performed first using the unanimous reference truth. As shown in Table 2, the results of the ANOVA-after-jackknife analysis and the bootstrap analysis using the unanimous reference truth are very similar for the Phase I, Phase II and pooled data.
|
Table 2.
Study analyses of the significance of the Improvement in Area under
ROC curve for the Phase I, Phase II and pooled study data. |
|||||
|
Analysis Method |
Study |
Estimated Improvement in Az |
p-value |
95% CI |
Statistical Significance |
|
Primary: ANOVA-after-jackknife, with unanimous reference truth |
Phase I |
0.0140 |
0.108 |
(-0.0033, .0313) |
No |
|
Phase II |
0.0240 |
0.003 |
( 0.0084, .0395) |
Yes |
|
|
Pooled |
0.0209 |
0.001 |
( 0.0085, .0313) |
Yes |
|
|
Secondary: Bootstrap with
unanimous reference truth |
Phase I |
0.0139 |
0.126 |
(-0.0041, .0354) |
No |
|
Phase II |
0.0246 |
<0.001 |
( 0.0089, .0446) |
Yes |
|
|
Pooled |
0.0192 |
<0.001 |
( 0.0074,
.0345) |
Yes |
|
|
Secondary: Bootstrap with random 2-panel reference truth |
Phase I |
0.0168 |
0.058 |
(-0.0004, .0396) |
No |
|
Phase II |
0.0216 |
0.002 |
( 0.0077, .0387) |
Yes |
|
|
Pooled |
0.0186 |
<0.001 |
( 0.0073, .0314) |
Yes |
|
Finally, several approaches were used, based on the bootstrap re-sampling approach, to incorporate random reference truths for the random cases against which the random readers’ performance could be estimated. All methods showed similar results; therefore a representative approach that selected two panelists at random is shown here. Based on varying the reference truth in this way, in the Phase II study, the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0216 (p=0.002) with a 95% confidence interval of (0.0077, 0.0387). By comparison, in the Phase I study, the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0168 (p=0.058) with a 95% confidence interval of (-0.0004, 0.0396). The combined (pooled) data demonstrated that the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0186 (p<0.001) with a 95% confidence interval of (0.0073, 0.0314). Thus again, the results with the varied reference truth show that both the Phase II study by itself and the pooled study results showed a statistically significant improvement in the area under the ROC curve with the use of CAD. The Phase I study by itself demonstrated a consistent trend, but 32 cases were insufficient to achieve statistical significance.
The primary and secondary analysis from the Phase II study alone or the two studies combined shows that the ImageChecker CT CAD Software System significantly improves radiologists’ ROC performance for detecting solid pulmonary nodules between 4 and 30 mm in diameter. Also, this result is robust when different reference truth definitions are used in the analysis.
K. CONCLUSIONS DRAWN FROM STUDIES
For multi-slice CT exams of the chest:
The ImageChecker CT CAD Software System significantly (p=0.003) improves radiologists’ ROC performance for detecting solid pulmonary nodules between 4 and 30mm in size.
L. PANEL RECOMMENDATIONS
(To be added by FDA)
M. CDRH DECISION
(To be added by FDA)
N. APPROVAL SPECIFICATIONS
(To be added by FDA)