SUMMARY OF SAFETY AND EFFECTIVENESS DATA

 

 

A.        GENERAL INFORMATION

 

Device Generic Name:                                System, Image Processing, Radiological

 

Device Trade Name:                                   ImageChecker® CT CAD Software System

 

Applicant’s Name and Address:                  R2 Technology, Inc.

                                                                  1195 W. Fremont Avenue

                                                                  Sunnyvale, CA 94087

 

PMA Number:                                           P030012

 

Date of Panel Recommendation:                  February 3, 2004

 

Date of notice of approval to Applicant:      TBD

 

 

B.         INDICATIONS FOR USE

 

The ImageChecker CT CAD Software System is a Computer-Aided Detection (CAD) system designed to assist radiologists in the detection of solid pulmonary nodules during review of multi-slice CT (MSCT) scans of the chest.  It is intended to be used as a “second reader,” alerting the radiologist – after his or her initial reading of the scan – to regions of interest (ROIs) that may have been initially overlooked.

 

In this way, the system reduces the chance of observational lapses by the reader due to fatigue, distraction, satisfaction of search, or other recognized causes of a sub-optimal review.  Thus, the system increases the chances that a clinically significant nodule will be not be overlooked, resulting in improved patient care (e.g., detection at an earlier, more treatable stage).

 

C.         CONTRAINDICATIONS

 

There are no contraindications for the use of this device.

 

 

D.        WARNINGS AND PRECAUTIONS

 

Warnings and Precautions for use of the device are stated in the attached product labeling.  

 

 

E.         DEVICE DESCRIPTION

 

System Overview

The ImageChecker CT System is an image analysis and visualization system designed to assist radiologists in the review of multi-slice CT (MSCT) exams of the chest and in the detection of solid pulmonary nodules.

The ImageChecker CT System is a combination of dedicated computer software and hardware.  The system is comprised of the ImageChecker CT Workstation (K023003) and the ImageChecker CT CAD software that is the subject of this PMA filing.  The two components are related as indicated in Figure 1.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Figure 1.  Overview of the ImageChecker CT system illustrating the two devices that comprise the system and the relevant hardware and software components.

 

Developed by R2 Technology, Inc., the ImageChecker CT Computer-Aided Detection (CAD) System is intended to be used as an adjunctive aid for radiologists interpreting MSCT scans of the chest – providing detection of solid parenchymal densities that may represent clinically significant nodules.

The ImageChecker CT CAD software does this by applying proprietary signal processing algorithms to the large digital datasets generated during scanning. These algorithms, in effect, analyze the complete set of images and search for findings with features suggestive of a solid pulmonary nodule. The system conveys information regarding “candidate” nodules to the workstation for consideration by the reader.

 

The ImageChecker CT CAD Software System is not intended to be used principally in a lung-cancer screening environment, where the vast majority of scans would be expected to be free of clinically significant disease. Rather, it is intended as a “back-up” nodule identification and reporting system for radiologists reading everyday diagnostic chest CTs – many of which will have other pathologic processes competing for their attention.

After the initial study acquisition by the CT scanner, copies of the study are sent automatically to the review workstation for the user to review as well as to the processing server for segmentation and CAD analysis. When the processing server finishes, it sends a report to the workstation.  All image and information exchanged between the components of the system and external devices (such as CT acquisition devices, PACS systems) are controlled using standard DICOM (Digital Imaging and Communications in Medicine) protocols.

 

The following sections give a brief overview of the individual devices that comprise the ImageChecker CT System. 

 

ImageChecker CT Workstation

 

The ImageChecker-CT Workstation is indicated for use as a general medical imaging workstation, and is used to receive, store, transmit, and display images from a Multislice CT scanner (MSCT).  The FDA cleared the workstation for marketing on 11/4/2002 (K023003) as ImageChecker CT Model LN-500.  The ImageChecker CT Workstation combined with the ImageChecker CT CAD Software that is the subject of this PMA submission will be distributed as the Model LN-1000. 

 

The ImageChecker-CT Workstation LN-1000 is comprised of two off-the-shelf personal computers, one with a Linux-based operating system (OS) and one with a Microsoft Windows-based OS, and a display monitor. 

 

The processing software performs several functions:

·        Receives MSCT exams using the DICOM standards.

·        Takes the CT images and segments different anatomical structures into normal structures (e.g. vessels) and other composite features.

·        Stores the location and characteristics of the segmented composite features in a DICOM Structured Report object.

The workstation display software provides tools for the radiologist that aid in the review process.  During the review, the radiologist instructs the display software by means of a standard keyboard and mouse.  The images and findings are communicated to the radiologist by means of a color Flat Panel display.  When the user completes his or her review, the system makes a summary report available that lists any findings and measurements associated with the study. The user can also print this summary.

The workstation is also able to display the findings that are identified by the CAD software (see next section).  The radiologist using the workstation is able to view the CAD findings – after a preliminary review of the study – using a simple button press on the user interface.  The user can then recheck the areas that the CAD software points out to determine if a change in his or her interpretation of the exam is warranted.

 

ImageChecker CT CAD software

 

The ImageChecker CT CAD software is an adjunctive software package that analyzes the CT images after they have been pre-processed by the workstation software and identifies regions of interest that may be solid pulmonary nodules.  The regions of interest are identified by means of the propriety signal processing algorithms that analyze the images and search for findings with features suggestive of a solid pulmonary nodule (see Figure 2).

 

The location information about these identified regions of interest is sent to the workstation using a DICOM CAD Structured Report (DICOM CAD SR). 


 


 

 


Figure 2. Examples of CAD-marked pulmonary nodules

 


F.         ALTERNATIVE PRACTICES AND PROCEDURES

 

Currently, there is no alternative to assist radiologists in the detection of lung nodules on multi-slice CT chest scans other than double reading.

 

 

G.        MARKETING HISTORY

 

The ImageChecker CT CAD Software System was launched for commercial sale in the European Union at the European Congress of Radiology in March 2003.

 

 

H.        POTENTIAL ADVERSE EFFECTS OF DEVICE ON HEALTH

 

There are no known direct risks to safety or health caused by, or related to, the use of the device.  The indirect risks are that the device may fail to identify and mark some actionable lesions and will mark some lesions that do not require further action.

 

I.                   NON-CLINICAL STUDIES

 

There are no non-clinical studies to report for this device.

 

J.         CLINICAL STUDIES

 

R2 Technology, Inc. has conducted two pivotal clinical studies to evaluate the safety and effectiveness of the ImageChecker CT CAD Software System.  The studies were based on a retrospective case collection project involving multiple clinical sites in various regions across the U.S.

 

Objectives

 

The first pivotal study was designed to generate a “truth” set of cases containing solid pulmonary nodules, as well as cases with no nodules, to be used as a reference truth for subsequent studies. The second pivotal study, conducted in two phases, was an Observer/ROC (Receiver Operating Characteristic) Study, designed to measure the performance enhancement of radiologists using the System. 

 

Sites and Cases for Case Collection Project and Subsequent Studies

 

Five (5) regionally diverse sites contributed 151 cases to the study; 2 sites in the Northeast, and 1 site each from the South, the Midwest, and the West. Of these sites, 3 were private imaging centers and 2 were academic medical centers. These sites are listed in Table 1.

 

Table 1.  Sites contributing nodule and non-nodule CT chest cases in the Case Collection Project and subsequent clinical studies

Name

State

Number of

Nodule-present Cases Used in Studies

Number of Non-nodule Cases

Used in Studies

Total Cases

Atlantic Medical Imaging

NJ

11

10

21

MRI & CT Diagnostics

VA

15

23

38

South Jersey Radiology

NJ

14

35

49

University of  Iowa

IA

18

9

27

UC, San Francisco

CA

5

11

16

Total

 

63

88

151

 

All cases were culled consecutively from the sites’ digital archives according to the inclusion and exclusion criteria identified in the case collection protocols.

 

The nodule-present cases collected included only those in which a diagnosis of cancer, either primary lung cancer or an extrathoracic neoplasm, had been documented. Other co-existing disease processes resulting in the formation of nodules (e.g. TB, histoplasmosis, rheumatoid lung) were allowed, as were cases containing other “background” pathology such as lobar pneumonia, emphysema, and heart failure.

 

A total of 63 nodule-present cases dating from November 2001 through December 2002 were included in the studies.  The study population consisted of 56% females and 44% males, with a median age of 66 and a range of 20-86.  The malignancy consisted of primary lung cancer in 24 (38%) of these cases, and documented extra-thoracic primary cancer with suspected metastatic disease to the lung in the remaining 39 (62%) cases. 

 

The “normal” cases collected were those in which no nodules were deemed to be present by the principal investigator at each site. Other disease processes could be present, including the presence of pulmonary masses (>3cm). Histories of cancer, radiation therapy, or even previous thoracotomy, were allowed. In all, 88 nodule-absent cases dating from June 2002 through December 2002 were included in this retrospective case collection. 

 

 

 


STUDY #1 – IDENTIFICATION OF REFERENCE TRUTH

 

The objective of this study was to generate a “truth” set of unanimous actionable nodules, as identified by a panel of 3 experienced radiologists, to serve as a reference truth for all subsequent studies.

 

To achieve this objective, multiple panel sessions were scheduled in which 3 radiologists independently read a variable number of cases (min = 12, max = 25) until all 151 study cases had been interpreted by all 3 readers.

 

The three panelists identified a total of 142 findings in the 151 cases that met the size (4-30mm) and peak density (> -100HU) requirements, and which all three panelists agreed were actionable. The presence or absence of at least one of these findings in a quadrant was used as the reference truth for study #2 below. 

 

The findings ranged in size from 4-28mm. It is of note that the majority of these findings were between 4mm and 8mm in diameter (46%, 66/142), with the largest categories being the 5 – 6 mm (15%, 21/142) and 6 – 7 mm (15%, 22/142) findings.

 

 

STUDY #2 – EFFECT OF CAD SYSTEM ON IMPROVING ACCURACY OF IDENTIFICATION OF ACTIONABLE NODULES

 

The objective of this study was to demonstrate that review of CAD output improves performance of radiologists reviewing MSCT with respect to their ability to accurately identify actionable nodules. The study employed a Receiver Operating Characteristic (ROC) methodology that has become the standard in the radiology community for evaluating imaging modalities.  The study was conducted in two phases, with the first phase using a smaller case set (n=32 cases) and the second phase using a larger case set (n=90 cases).  The study methods and analysis were the same for both phases of the study.

 

For Phase I of the study, thirty-two (32) cases were randomly selected from the 151 cases evaluated in study #1.  These cases were each divided into 4 quadrants, yielding a total of 128 regions for evaluation.  Each of 15 radiologists independently reviewed the 128 quadrants, first without computer-aided detection (CAD) and then immediately with CAD.

 

For Phase II of the study, ninety (90) cases were randomly selected from the remaining 119 cases (151 – 32 used in Phase I).  Again, the cases were divided into 4 quadrants, yielding 360 regions for evaluation.  Each of 15 radiologists (9 returning radiologists from Phase I, 6 new radiologists) independently reviewed the 360 quadrants, first without computer-aided detection (CAD) and then immediately with CAD

 

For both phases, each reader rated each quadrant on a 0-100 “actionability scale” as to his or her level of confidence that the quadrant contained at least one actionable nodule.  Ratings were provided both before and after viewing the CAD marks.

 

For purposes of measuring reader performance, quadrants were defined as “actionable” if the reference truth consensus panel described in study #1 unanimously agreed that at least one of the findings in the quadrant was actionable. 

 

An area under the receiver operator curve (Az) was computed for each of the 15 radiologists before and after CAD.  The average curve is shown below in Figure 3 for the Phase I study and Figure 4 for the Phase II study.  In both cases the area under the ROC curve increased with the use of CAD. If the full plot is viewed as a unit square, the area separating the two curves in the Phase I study is 0.014 and the area separating the two curves in the Phase II study is 0.024.  Although the second study shows greater separation and a slightly smoother pair of curves, the basic shape of the curves in the two studies is similar. 

 

Az post-CAD    = 0.8880

Az pre-CAD      = 0.8739

 

p =  0.108      

 

Figure 3.  Average ROC curve showing pre-CAD (dashed line) and post-CAD (solid line) for Phase I (32 case, 128 quadrant) study.

 

Az post-CAD    = 0.9047

Az pre-CAD      = 0.8808

 

p =  0.003      

 

Figure 4.  Average ROC curve showing pre-CAD (dashed line) and post-CAD (solid line) performance for Phase II (90 case, 360 quadrant) study.

 

The primary analysis of statistical significance was based on the Dorfman-Berbaum-Metz (DBM) ANOVA-after-jackknife approach[1] (adapted for the fact that the quadrants are “clustered” data) and the results are presented in Table 2 below. 

 

In the Phase II study (Figure 4), the average reader improvement in AZ (estimated using the ANOVA-after-jackknife) was 0.0240 ± 0.0077 (p=0.0033) with a 95% confidence interval of (0.0084, 0.0395).  By comparison, in the Phase I study (Figure 3), the average reader improvement in Az (estimated using the ANOVA-after-jackknife) was 0.0140 ± 0.0084 (p=0.1083) with a 95% confidence interval of (-0.0033, 0.0313).  The combined 122 case study demonstrated that the average reader improvement in AZ (estimated using the ANOVA-after-jackknife) was 0.0209 ± 0.0062 (p=0.0013) with a 95% confidence interval of (0.0085, 0.0333).  Thus both the Phase II study by itself and the pooled study results showed a statistically significant improvement in the area under the ROC curve with the use of CAD.

 

The secondary analysis of statistical significance was conducted to determine the dependence of the study results on the use of the consensus reference truth from study #1 as the gold standard for the ROC study.  In general, a reference truth based on a consensus panel assessment of actionability is weaker than one based on a more invariant gold standard such as biopsy. To examine the effect of variability in the unanimous three-panelist reference truth, two-panelist reference truths and single-panelist reference truths were constructed from the data collected in study #1. 

 

Implementing this variable truth is difficult within the framework of the ANOVA-after-jackknife analysis, therefore the secondary analysis employs a bootstrap analysis[2].  The bootstrap is a computationally-intensive non-parametric method that allows complex analyses to be repeated many times using different randomly generated datasets (all based on the original data) to approximate the variability that would occur if the entire study were repeated many times.  As a test of the validity of the bootstrap mechanism, the analysis was performed first using the unanimous reference truth.  As shown in Table 2, the results of the ANOVA-after-jackknife analysis and the bootstrap analysis using the unanimous reference truth are very similar for the Phase I, Phase II and pooled data.

 

Table 2.  Study analyses of the significance of the Improvement in Area under ROC curve for the Phase I, Phase II and pooled study data.

Analysis Method

Study

Estimated Improvement

in Az

p-value

95% CI

Statistical

Significance

Primary:

ANOVA-after-jackknife, with unanimous reference truth

Phase I

0.0140

0.108

(-0.0033, .0313)

No

Phase II

0.0240

0.003

( 0.0084, .0395)

Yes

Pooled

 

0.0209

0.001

( 0.0085, .0313)

Yes

Secondary: Bootstrap with unanimous reference truth

Phase I

0.0139

0.126

(-0.0041, .0354)

No

Phase II

0.0246

<0.001

( 0.0089, .0446)

Yes

Pooled

0.0192

<0.001

( 0.0074, .0345)

Yes

Secondary:

Bootstrap with random 2-panel reference truth

Phase I

0.0168

0.058

(-0.0004, .0396)

No

Phase II

0.0216

0.002

( 0.0077, .0387)

Yes

Pooled

0.0186

<0.001

( 0.0073, .0314)

Yes

 

 

Finally, several approaches were used, based on the bootstrap re-sampling approach, to incorporate random reference truths for the random cases against which the random readers’ performance could be estimated.  All methods showed similar results; therefore a representative approach that selected two panelists at random is shown here.  Based on varying the reference truth in this way, in the Phase II study, the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0216 (p=0.002) with a 95% confidence interval of (0.0077, 0.0387).  By comparison, in the Phase I study, the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0168 (p=0.058) with a 95% confidence interval of (-0.0004, 0.0396). The combined (pooled) data demonstrated that the average reader improvement in AZ (estimated using the 1000 bootstrap samples with variability in the reference truth) was 0.0186 (p<0.001) with a 95% confidence interval of (0.0073, 0.0314).  Thus again, the results with the varied reference truth show that both the Phase II study by itself and the pooled study results showed a statistically significant improvement in the area under the ROC curve with the use of CAD. The Phase I study by itself demonstrated a consistent trend, but 32 cases were insufficient to achieve statistical significance.

 

The primary and secondary analysis from the Phase II study alone or the two studies combined shows that the ImageChecker CT CAD Software System significantly improves radiologists’ ROC performance for detecting solid pulmonary nodules between 4 and 30 mm in diameter.  Also, this result is robust when different reference truth definitions are used in the analysis.

 

 

 

K.         CONCLUSIONS DRAWN FROM STUDIES

 

      For multi-slice CT exams of the chest:

 

The ImageChecker CT CAD Software System significantly (p=0.003) improves radiologists’ ROC performance for detecting solid pulmonary nodules between 4 and 30mm in size.

 

 

L.         PANEL RECOMMENDATIONS

 

                  (To be added by FDA)

 

M.        CDRH DECISION

                 

                  (To be added by FDA)

 

N.         APPROVAL SPECIFICATIONS   

 

                  (To be added by FDA)

 

 

 

 

                       

 

 

 



[1] Dorfman DD, Berbaum KS and Metz, CE.  Receiver Operating Characteristic Rating Analysis.  Invest Radiol 1992;  27: 723-731.

 

[2] Rutter, C.  Bootstrap Estimation of Diagnostic Accuracy with Patient-clustered Data.  Acad Radiol 2000; 7: 413-419.