• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

Medical Devices

  • Print
  • Share
  • E-mail

FDA Workshop - Seizure Detection, Cognitive Function, and TBI/Concussion Devices: Issues in Their Evaluation June 2-3, 2011 Breakout Questions

Questions Related to Seizure Detection Devices
1. Typically neurodiagnostic devices are validated with
sensitivity/specificity or positive percent agreement (PPA) and
negative percent agreement (NPA) and compared against a
clinical reference standard (“ground truth”).

a. A lack of agreement on the exact definition of a seizure causes
inherent issues with the validation of seizure detection
algorithms. Would a standardized database of previously
identified seizures address this issue without requiring a
prespecified definition of seizure?

b. What is an acceptable Clinical Reference Standard (CRS)
against which to compare the algorithm’s assessments?

c. In the case of seizure detection devices, specificity is
incalculable, so false detection rate (FDR) is typically
substituted. Is there a preferable statistical method to assess
adequate performance of seizure detection devices?

d. Identification of false positive (FP) detections is important in
the characterization of device performance. What should be
considered a FP detection? Should FPs be defined differently
depending on the type of EEG assessed (scalp vs. intracranial)
or the situation in which the device will be used (ICU vs.
EMU)? How?
2. Currently marketed seizure detection devices have a wide range of
sensitivities and false detection rates. What is the best way to deal
with these broad ranges?

3. How should seizure detection algorithms address differences
between scalp and intracranial EEG (e.g., specify on which type of
EEG they are intended to be used and/or be validated using only
that type of EEG data)?
4. EEGs and seizures in ICU patients may differ significantly from
those in EMU patients. How should this difference be captured in
the validation of seizure detection algorithms that are intended to
be used in ICU patients? If the validation and/or performance
goals should be different, are there minimal acceptable results ?
5. Context of how the device will be used is important in the
validation of a seizure detection device. Devices that are used for
retrospective (post hoc) analysis, early detection, or
prognostication of seizures may require different levels of evidence
or methods of validation.

a. What should be the minimal acceptable performance results (if
any) for devices to be used for post-hoc review of long EEG
recordings?

b. A seizure detection algorithm with a real-time alarm may imply
that the device can be useful for early detection of seizures.
Would this type of algorithm require different performance
goals from those used to validate post-hoc review seizure
detection software? If so, what should those values ideally be?

c. Should these values change if the EEG data will be reviewed
by clinicians untrained in interpretation of EEGs (e.g.,
intensivists, nurses, emergency physicians)? If so, how?

d. Should these values change if the device with a real time alarm
is used in an ambulatory setting (i.e., no healthcare
professional nearby)? If so, how?

e. In the future, a seizure detection algorithm may be intended to
predict an upcoming seizure. Would this type of algorithm
require better performance than that used to validate post-hoc
review seizure detection software? If so, what should those
values ideally be?

f. “Any overlap” is the method of agreement used in the
validation of post hoc review algorithms. What method should
be used for validation when there is a real time alarm or
seizure prediction?

6. Does transparency of the algorithm to FDA or the user mitigate
any of the above performance “requirements”? Is there any
specific information that should be included in the labeling that
will provide for improved usage of these devices?


Questions Related to Computerized Cognitive Assessment Devices
1. A paper-based test’s documentation typically includes a significant
amount of descriptive, technical and validation information to help
users make an informed decision about how or when to use the
test. Should computerized tests be held to the same or different
standards as other tests in regard to these aspects of
documentation? What specific information should be included?

2. Computerized tests raise a number of additional issues that paperbased
tests do not.

a. What computer-specific issues should computerized tests at
least acknowledge and address in some way?

b. For each factor identified in (a) above, what is the
preferred method(s) of mitigating the variation in
performance due to this factor (e.g., validation studies,
labeling)?

3. What should be the type or level of evidence that should be
included in a user manual for claims related to either a domain of
cognitive function or related to diagnosis of a specific disease
process? Specifically:

a. For claims related to domains of cognitive function, what
evidence should be included in the manual to support the
validity of the claim? What data are needed? How should
“impairment” be defined (e.g., 1 or 2 standard deviations
from mean)?

b. For claims related to diagnosis of specific disease, what
evidence should be included in the manual to support the
validity of the claim? What data / types of studies are
needed? What should be the reference standard and how
should the disease population be defined? How should
“impairment” be defined?

c. For both type of claims, is it enough to demonstrate
correlation with other tests or are convergent and
divergent validity data needed?

4. Several patient-related factors can cause variations in device
performance.

a. What are the patient-related factors that computerized
cognitive tests should at least acknowledge and address in
some way?

b. For each factor identified in (a) above, what is the
preferred method(s) of mitigating the variation in
performance due to this factor (e.g., “pre-test”, validation
studies, labeling)?

Questions Related to TBI/Sports Concussion Devices
1. How should diagnostic tests for TBI/Sports Concussion (both
cognitive and non-cognitive) be validated?

2. What does a TBI/Sports Concussion diagnostic device need to
measure clinically and what types of information (e.g., instruction
manual) should be available to the user regarding the device
output?

3. What specific clinical performance data should be included in
clinical validation studies to:

a. Determine that a diagnostic device for TBI/Sports
Concussion reliably assesses the key clinical features of
these disorders?

b. Determine if a change on a test score in a device’s output
from one test session to another translates into meaningful
clinical information for the user of the device?

4. What does the test user need to know about the test to make an
informed decision about how or when to use the test? Specifically,
what do they need to know about:

a. The specifics of the actual test instrument (e.g., cognitive
scales, EEG analysis parameters, biomarker analysis, etc)?

b. The algorithm(s) that analyze the patient’s response?

c. The norms to which the patient is compared?

d. Other processing steps the software might include (e.g.,
imputation of missing data, alternate test forms, automaticadaptive
testing)?

e. Limitations of use (e.g., limits to validation data, lack of
understanding about practice effects, etc)

5. What additional information would be needed for a TBI/Sports
Concussion diagnostic device to be used to support clinical
decision making for:

a. The acute phase of a suspected TBI at the time of the injury
(e.g., sports field, battlefield, emergency department)

b. The sub-acute phase of a suspected TBI (e.g., clinician
office, outpatient clinic, etc.)

c. Clearance to return to usual premorbid level of activity
following a TBI (e.g., clearing an individual to return to
his/her usual premorbid level of functioning (e.g., return to
work, athletic competition, military duty, etc.)?

d. The longer-term or chronic phase of a suspected TBI (e.g.,
determining that an individual should no longer participate
in their usual premorbid level of functioning (e.g.,
permanently unable to return to work, athletic competition,
military duty, etc.)?