Final Report of a Study to Evaluate the Feasibility and Effectiveness of a Sentinel System for Adverse Event Reporting of Medical Device Use in User Facilities
PREPARED UNDER CONTRACT # 223-96-6052
JUNE 16, 1999
Office of Surveillance and Biometrics
Center for Devices and Radiological Health
Food and Drug Administration
Department of Health and Human Services
1350 Piccard Drive, 3rd Floor
Rockville, MD 20850
RESULTS OF A FEASIBILITY STUDY FOR A SENTINEL SURVEILLANCE SYSTEM
FOR ADVERSE MEDICAL DEVICE EVENTS
TABLE OF CONTENTS
- 1. BACKGROUND
- 2. SCHEDULE FOR THE FEASIBILITY STUDY
- 3. EXPLORATORY PHASE
- 4. RECRUITMENT ACTIVITIES
- 5. TRAINING
- 6. DATA COLLECTION
- 7. RESULTS OF THE QUALITATIVE ANALYSIS
- 8. QUANTITATIVE ANALYSIS OF STUDY RESULTS
- 9. SUMMARY
- Progress in Meeting Study Goals
The mission of the FDA's Center for Devices and Radiological Health (CDRH) is promoting public health by ensuring the safety and effectiveness of medical devices. CDRH’s oversight authority for medical devices comes through the 1976 Medical Device Amendments to the Food, Drug and Cosmetic Act, the 1990 Safe Medical Device Act (SMDA), the 1992 Medical Device Amendments and the 1992 Mammography Quality Standards Act. The CDRH pursues its mission through several avenues: evaluating devices for marketing, developing and monitoring product quality and performance standards, acting against firms that violate the law, educating professionals and consumers on the safe use of devices, and performing research on device problems. Much of this work draws upon information from FDA reporting programs such as those for manufacturers, health care facilities, health care professionals, and patients.
In 1993, the FDA consolidated its various voluntary and mandatory reporting programs under MEDWATCH, the agency’s postmarket surveillance program for reporting adverse events associated with all medical products regulated by FDA. Medical Device Reporting (MDR) refers to the MEDWATCH reporting program under which device manufacturers and certain health care facilities designated as "user facilities" are required to report device-related serious injuries and deaths.
In addition to the changes created by MEDWATCH, the Safe Medical Device Amendments in 1990 and 1992 had significant impact on CDRH’s postmarket surveillance program. The amendments require user facilities -- hospitals, nursing homes, ambulatory surgical facilities and outpatient diagnostic and treatment facilities -- to report any incidents in which medical personnel reasonably suspect that a medical device has caused or contributed to the death of a patient or to a serious injury or illness. Facilities are also required to establish and maintain adverse event files. A user facility must submit an adverse event report within 10 days of becoming aware of a death, serious injury or illness, using FDA Form 3500A. In case of death, the user facility is required to submit the report to both the device manufacturer and FDA. In case of serious injury or illness, it must submit the report to the device manufacturer or directly to FDA if the manufacturer is unknown. Facilities that reported to a manufacturer and/or the FDA during the previous year also must submit an annual summary report to FDA covering all reports submitted that year. (This summary report was submitted semi-annually until revised regulations took effect starting with calendar 1998 reports.) Health care facilities also may submit reports on incidents that do not fall under mandatory reporting guidelines but that illustrate the potential for harm. For these incidents they may use <a href=">Form 3500A or Form 3500 for voluntary reporting, which does not require as much information as the form for mandatory reporting.
Although SMDA made the reporting of adverse events involving serious injury or death mandatory for user facilities, it is a completely passive surveillance system, relying on the consumer or health care professionals within the user facility to recognize that an event may be device-related and then to submit a report to the manufacturer (in the case of serious injury) or to both the manufacturer and FDA (in the case of death). FDA does not have the resources to train personnel in an estimated 50,000 to 60,000 user facilities concerning when and how they should report, nor does it have the resources to routinely monitor whether or not facilities report when an adverse medical device event occurs. One proposed solution to this problem is to establish a sentinel surveillance system, that is, to identify a sample of all facilities and then to establish a more proactive surveillance network involving them. The Food and Drug Modernization Act of 1997 (FDAMA) required FDA to report to Congress in late 1999 about progress towards a sentinel system.
In September 1996, CODA Inc. was awarded a contract to conduct a study to evaluate the feasibility and effectiveness of a sentinel reporting system for adverse event reporting of medical device use in user facilities for the Office of Surveillance and Biometrics of CDRH. The National Association of Health Data Organizations (NAHDO) served as a subcontractor to CODA to provide support, primarily in the area of facility recruitment and retention. Mr. Elliot Stone, Executive Director of the Massachusetts Health Data Consortium, also assisted CODA as a consultant in the exploratory research stage of the project.
The study was intended to evaluate the usefulness of an active sentinel reporting system in postmarket surveillance. In the beginning of the contract, the sentinel system was seen as a supplement to the existing passive system. However, after passage of the Food and Drug Administration Modernization Act in 1997, the sentinel system approach was viewed as a substitute, in future years, for the existing passive system for user facility reporting (with non-sampled facilities reporting to manufacturers and to FDA on a voluntary basis). CODA was to recruit health care facilities to participate in the year long study, train facility staff in correct reporting procedures, and provide support to the facilities in submitting reports on adverse device events. The purpose of this study was to evaluate the feasibility and effectiveness of a sentinel system for reporting adverse events connected with the use of medical devices. To be effective, the system had to achieve several goals:
- Increased reporting by user facilities of adverse events involving medical devices, and
- Higher quality of event reports submitted.
In order to achieve these primary goals, it was thought to be necessary to accomplish the following secondary goals:
- Increased awareness of user facility staff about the importance of reporting such events,
- Increased level of knowledge among user facilities about what events are reportable,
- Decreased burden on user facilities, and
- Better communication between facilities and the office to which they report, FDA’s Center for Devices and Radiological Health.
This report describes the schedule of events for the feasibility study, the exploratory research performed before the study was launched, how the participating facilities were recruited and trained, the process of data collection, and how channels of communication were established among the participating facilities, CODA and CDRH. The final two sections discuss the results of the qualitative and quantitative portions of the study. Tables documenting the findings are contained in an appendix to the report being distributed under separate cover.
2. SCHEDULE FOR THE FEASIBILITY STUDY
Project staff began the planning and development phase of the feasibility study in October 1996 with an exploratory research effort. We developed a draft study plan, and after it was approved, we developed the materials and systems needed to implement the study plan and simultaneously conducted identification and recruitment of user facilities. In each facility contacted, we asked that a member of the staff be designated as the Study Coordinator; usually this was the Risk Manager.
CODA developed a two-pronged approach to educating facility staff about the feasibility study and the requirements of adverse device event reporting: 1) developing a training video suitable for use with nursing and allied health staff in the participating user facilities, and 2) holding a one-day orientation program for the Study Coordinators, who were responsible for reporting adverse events to the project.
In order to keep the project salient in the minds of the participants, we decided that the feasibility study should have a distinctive name that was easy to remember and that would give us an easy way of referring to the project; we selected the name "DEVICENET."
The study year was designated as Oct. 1, 1997, through Sept. 30, 1998. Participating facilities were asked to report events involving medical devices that occurred during that time. The major activities of the year of the project were: conducting orientations for Study Coordinators from participating facilities; collecting and processing event reports; providing technical assistance to facilities in their reporting activities; and maintaining communication with participating facilities on other matters, such as FDA reporting requirements, emerging medical device issues, and FDA safety notices.
Orientation sessions for Study Coordinators took place in October and November 1997. (Although the training overlapped the reporting period, we asked Study Coordinators to report any incidents that had occurred since the study year had begun on Oct. 1, 1997.) In order to allow facilities to report on incidents that had occurred late in the study year, CODA continued to receive and process reports until the end of November 1998.
During the data collection period, CODA provided technical assistance to Study Coordinators upon request, trained new Study Coordinators as necessary, and kept user facility staff informed about news regarding medical devices through a DEVICENET newsletter. Following the close of the data collection period, project staff began an evaluation of the impact of DEVICENET on the quantity and quality of the reports received.
3. EXPLORATORY PHASE
The exploratory research phase of this project was designed to provide CODA and CDRH with essential information about knowledge, attitudes and practices of user facilities with respect to the Safe Medical Devices Act (SMDA) and MEDWATCH, which would be used to inform the design of the feasibility project.
Initially, CODA project staff conducted four focus groups with persons responsible for MDR reporting from hospitals in the Washington/Baltimore area and Boston, staff from eight outpatient diagnostic and treatment facilities in Raleigh/Durham, and three staff members from nursing homes in Northern Virginia. At each session, participants were given an overview of the feasibility project and a brief review of SMDA, MEDWATCH and the MDR program. Participants were encouraged to talk about any problems they perceived with the current regulations and reporting requirements.
CODA project staff also made site visits to a number of hospitals and nursing homes in Washington and in Raleigh/Durham to learn about their internal decision-making processes for reporting adverse events. Additionally, project staff from CODA and the Office of Surveillance and Biometrics of CDRH reviewed a sample of the types of adverse medical device event reports submitted to FDA and spoke with representatives of several professional organizations whose members are subject to SMDA requirements.
The exploratory phase revealed a high level of concern about potential liability implications of MDR reporting. Reluctance to submit MDR reports when use error may have been involved was particularly high. We also learned that facility staff had problems with the definition of a reportable event, and found that it was not always clear or easy to apply. We heard many complaints about the problems of complying with the requirement to report within ten days of being aware of the adverse medical device event, which facilities saw as an unrealistically short period to gather definitive information about the event. Finally, lack of feedback to user facilities from the MDR system was repeatedly mentioned by facility staff, who described the experience of submitting reports to the system as like sending information into a "black hole."
In meetings with CDRH staff during the developmental phase of the project, several discussions took place that indicated that the most valuable reports often pertained to problems with devices that had the potential to cause harm, even though no serious harm had actually occurred.
Drawing upon the lessons gleaned from the exploratory phase, CODA and the CDRH staff made the following decisions about the design of the feasibility study:
- Given the difficulty in interpreting the MDR rules and deciding when an adverse event had occurred, combined with the value of reports indicating potential for harm, we decided to ask participating facilities to submit reports indicating potential for harm in addition to reports on adverse events.
- CODA would act as an intermediary, providing a channel of communication between facilities and CDRH. With this mechanism, CDRH could ask for feedback from facilities as needed (e.g., when considering policy changes or when interested in the dissemination of safety notice), and facilities could have quick access to information from CDRH. Feedback mechanisms would be built into the design of the project, so that CODA could pass on information about the types of reports received and FDA actions related to medical device problems and so forth. Facilities could also request information on specific device-related topics, which project staff would seek to obtain from appropriate sources at FDA.
- The information submitted by facilities to the project would be confidential. CODA would de-identify reports after a brief period of time (30 days for assigned ID number, within days of receipt for names of facilities and reporters), to allow for follow-up when project staff had questions about individual reports. For report-writing purposes, data would be categorized by type and size of facility (e.g., large hospital), but CODA would not release any information identifying reporters or reporting institutions.
- CODA would provide technical assistance to facilities in the reporting process as requested, including assistance with coding, preparation of MEDWATCH forms, assistance with MEDWATCH software, etc. However, CODA would not submit reports to manufacturers or CDRH, nor attempt to ascertain if facilities had done so.
- CODA would encourage participating facilities to submit reports as soon as possible, but would not insist on any particular time period.
These design features were described in the Memorandum of Understanding, a document provided to facilities at the time of recruitment which outlined the responsibilities of the project team, CDRH, and the participating facilities.
4. RECRUITMENT ACTIVITIES
Recruitment took place between late May 1997 and September 1997. The goal was to obtain a sample of four or five nursing homes with a minimum of 100 beds, 10 to 12 hospitals with a minimum of 200 beds, and at least one large hospital that routinely trained new physicians. Recruitment activities were focused primarily on three geographic areas: Washington/-Baltimore/Richmond, Raleigh/Durham, NC, and Boston, MA.
Facilities that had taken part in the exploratory phase were the first to be considered. These were facilities that had sent staff to participate in focus groups and/or allowed us to make site visits. We thought that enlisting these facilities would expedite the recruitment process, since we had a contact person in each of them that had a basic knowledge of the project. Additional candidates for participation were identified using the American Hospital Association guide and NAHDO contacts.
Initial contacts. Contacts from facilities that had participated in focus groups or site visits received a recruitment package. The package consisted of the following: a letter thanking them for their earlier input and inviting them to participate in the feasibility study, a brochure describing the project, and brochures describing CODA and NAHDO, the organizations conducting the feasibility study.
Approximately one week after the recruitment package was mailed, DEVICENET project staff from CODA or NAHDO followed up with a phone call to provide additional information and to determine the next steps for gaining the facility’s participation in the study. If the contact indicated interest, a site visit was scheduled.
For other facilities, recruitment began when a member of the DEVICENET project staff called to determine who might be the best contact person. The person identified was given a brief description of the study over the phone, and if he or she expressed interest in the project, a recruitment package was mailed. After that point, the sequence of events was the same as for facilities with previous knowledge of the project. Only three facilities refused to participate in the study during the first phone call to them.
During the first recruitment effort in May and early June 1997, we contacted 19 hospitals and 10 nursing homes. Of these, eight hospitals and three nursing homes had participated in focus groups or site visits. Within three weeks of the initial mailing, six facilities had indicated that they were not interested in participating. At other facilities, the approval process was slower than we had hoped, requiring the involvement of corporate offices and/or legal departments. Given the project’s time constraints, we decided to approach additional facilities in late June and early July. Twenty more hospitals and eight additional nursing homes were contacted.
Site visits. Site visits were conducted by teams of CODA and NAHDO project staff members who met with the facility contact and others from the administration or safety staff of the facility. In some cases, the CDRH Project Director accompanied the project staff. During the site visits, the project staff reviewed MEDWATCH and described the DEVICENET project and why it was needed. They discussed the project’s goals and anticipated results, what participation would entail, the different types of support that would be made available to facilities that decided to participate, project confidentiality procedures, and the benefits of participation. They also described the Memorandum of Understanding, a document that was intended as a signed commitment by CODA and CDRH to participating facilities regarding the types of support that would be available to them, what their participation would entail, and how any information they submitted would be protected. During the site visit, facility staff were provided ample time to ask questions about the project.
Twenty-three facilities -- 17 hospitals and six nursing homes -- eventually agreed to participate in the year long DEVICENET study. The persons designated as Study Coordinators by the participating facilities were generally risk managers for hospitals and directors of nursing for nursing homes, although there were also a quality management specialist, a hospital safety director, and a director of clinical engineering (representing hospitals) as well as an assistant director for operations and a residential care coordinator from nursing homes.
Orientation for Study Coordinators. During recruitment, Study Coordinators were given the choice of attending a one-day group orientation session in the Washington, DC, area or having DEVICENET staff conduct a 2- to 3-hour individual orientation at their facilities. Separate group sessions were scheduled for hospital and nursing home Study Coordinators.
An exception to the above plan involved a group of four hospitals and one nursing home in Richmond, Virginia. For this cluster of facilities, it was decided that it would be efficient and convenient to conduct an orientation session in Richmond.
In late September participating facilities were provided with materials for clinical staff training (a video and summaries of each facility’s internal procedures for reporting adverse medical device events). Although the study year was Oct. 1, 1997, to Sept. 30, 1998, the orientation sessions were scheduled for October and early November, with OSB approval, in order to accommodate facilities that agreed to participate late in the recruitment process. Participating facilities were asked to send the project reports of events that had occurred since Oct. 1, 1997.
The group and individual orientation sessions for hospital and nursing home Study Coordinators covered many similar topics: the purpose and importance of a sentinel system, the project background and goals, SMDA and mandatory reporting, the project design and plans, and confidentiality procedures. Participants asked questions of the speakers throughout the session, and a specific time at the end of the session was set aside for discussion and answering Study Coordinators’ questions and concerns. Since it was believed that Nursing Home Coordinators had less experience filling out MEDWATCH reports, CDRH staff presented a detailed review of the MDR reporting requirements for that session.
Orientation for hospital staff. Representatives from 14 hospitals attended the orientation session for Hospital Coordinators on October 9, 1997. Another four attended the session in Richmond. Two Study Coordinators who could not attend either session received individual orientations – one at the facility and the other by telephone. Orientation sessions for replacement staff were held on-site in one case and by phone in another case where the replacement Study Coordinator was very familiar with mandatory reporting requirements.
Orientation for nursing home staff. An orientation session for Nursing Home Coordinators was held on Oct. 15, 1997, at CODA’s home office in Silver Spring, MD; however, only two nursing home Study Coordinators were able to participate in this session. Three others were trained individually at their facilities later in October, and one attended the Richmond training session described above.
Training video for clinical staff. Our exploratory work revealed large gaps in the knowledge of facility clinical staff regarding the importance of reporting adverse medical device events. Therefore, we decided to develop a training video suitable for use during in-house staff orientation and in-service training sessions. Copies of the video, "The Role of Health Care Professionals in Detecting Medical Device Problems," were distributed to the participating facilities. The video encouraged health professionals to follow their facility’s internal procedures for reporting of adverse events. In order to be certain that clinical staff in each facility were knowledgeable about the appropriate steps to take to report adverse events, we contacted each Study Coordinator to obtain a description of his or her facility’s internal reporting procedures and the names and phone numbers of those responsible for adverse event reporting. Then we developed a one-page summary of each facility’s procedures. In the video, we told staff that a summary of their facility’s reporting procedures would be available for distribution at the end of the video. Copies of these summaries were then sent to each facility.
6. DATA COLLECTION
In this section we discuss the communication CODA staff established with participating facilities during the data collection period and the procedures used to gather additional information from facilities when event reports were not completely clear or consistent.
6.1 Communication with Facilities During Data Collection
DEVICENET was designed to provide a channel for communication between health care facilities and the Center for Devices and Radiological Health, especially concerning adverse event reporting. Project staff could ask Study Coordinators questions about their reports in order to obtain more complete information. Feedback from the DEVICENET project staff and FDA to the participating facilities was intended to provide Study Coordinators with information about what was being reported to the project and how FDA uses such information. A bimonthly newsletter was one major part of this communication; it was sent to the Study Coordinators, who often distributed it within their facilities. The DEVICENET project itself was to be a vehicle Study Coordinators could use to ask questions about the reportability of various events and to obtain information about what CDRH was learning about device problems that might be of use in their facilities. The specific ways in which communication with facilities took place are described below.
DEVICENET newsletter. The project newsletter, called the DEVICENET NEWS, was intended to inform participating facilities of the project’s progress and to keep the project salient in their minds. It also was designed to answer one of the primary complaints that CODA and CDRH heard during the exploratory phase -- that there was very little feedback from the FDA reporting system. Many of the representatives said they liked the FDA User Facility Bulletin, which had been distributed quarterly until recent years, as well as the FDA safety notices such as Alerts and Advisories. However, they also wanted more feedback about the uses of the reports and, in some cases, about the receipt and review of reports they had submitted.
The DEVICENET NEWS typically contained a letter from the Project Director, articles from FDA nurse analysts concerning various device problems, a listing of the DEVICENET reports received since the last issue, notices of teleconferences or articles that might be of interest, and other articles concerning patient safety issues. In some issues, articles were taken from the CDRH User Facility Bulletin, a quarterly publication that is no longer sent to facilities, but is available by fax and on the Internet. The newsletter was mailed to Study Coordinators every other month (starting with the November 1997 issue). Facilities received 2-5 copies, with additional copies available on request.
Informal communication. DEVICENET staff generally contacted Study Coordinators for two reasons: to apprise them of new information from FDA (FDA Advisories, Alerts, Warning Letters, etc.) or to discuss reports they had submitted. New FDA information was faxed to all Study Coordinators; reports were discussed by fax or telephone and sometimes by e-mail. Study Coordinators could contact DEVICENET staff on our toll-free line for fax and telephone.
After the receipt of a DEVICENET report, CODA staff called the Study Coordinator who submitted it to thank him or her for the submission and to ask any questions that may have come up during the review of the report. These questions usually concerned the details of the event description and a review of the coding. Often the Study Coordinator was able to provide more information about the event at the time of the phone call either because more information now was available or because the Incident Report provided details that were not written on the MEDWATCH report form. Informal communication often took the form of a phone call, although e-mail was also a useful way to reach several Study Coordinators.
Study Coordinators also initiated contact with DEVICENET staff during the project. They asked questions related to their work as Risk Managers or Biomedical Engineers, such as names of contacts at FDA, specifics of device tracking regulations, or how to use MEDWATCH software for filing reports. On several occasions, Study Coordinators also asked whether they were required to report certain events to FDA and manufacturers. In such cases, DEVICENET staff played an intermediary role, gathering information from FDA and reporting it back to Study Coordinators and, when appropriate, passing on the names and phone numbers of FDA staff who could be contacted for any additional questions.
Informal communication was a useful aspect of the DEVICENET project, allowing facilities to ask questions that they might not want to ask directly of FDA staff. On several occasions, FDA also made use of this aspect of the project when it wanted a quick reading on what reaction health facilities would have to certain draft regulations, guidance documents, or policy decisions.
Project reminders. CODA sent Study Coordinators desk clocks that said "DEVICENET" on the case at the end of 1997, as a continuing reminder of their facility’s participation in the project. In addition, in the Spring of 1998, DEVICENET staff sent Study Coordinators copies of a poster designed for use in the project. The colorful poster, which showed a hand writing a report (with many people poised on the sleeve of the reporter), read, "Report Problems Involving Medical Equipment or Products. Your report could save a life. For more information contact (blank space)", with the name and telephone extension of the Study Coordinator to be filled in on site. Several Study Coordinators used the poster throughout their facilities, in places such as staff lounges, offices, and conference rooms, where it served as a reminder of the importance of reporting problems with medical equipment or products.
6.2 Data Retrieval and Follow-up
Once adverse event reports arrived at CODA, project staff reviewed them with a nurse consultant and a Medical Informatics specialist, who did not know the identity of the facilities making the reports. The specialists helped CODA determine if certain questions about the reports needed to be pursued in follow-up calls with the Study Coordinators. They also assisted with the review of codes for patient effects and device problems that had been entered on the reports, helped in the determination of the quality of those codes, and often recommended other codes that were more precise or more appropriate.
Study Coordinators were contacted either by phone or e-mail to discuss the reports. In most cases, they were very willing to answer questions. Some Study Coordinators were very difficult to reach or did not return calls from project staff, but these were the exception to the rule. Additional information obtained through contacts with the Study Coordinators was written into the report file and subsequently entered into the study database.
CODA staff evaluated the clarity of the event descriptions, the clarity of the patient outcome information, and the internal consistency of the reports, and recorded the codes for these judgments. If further information was received from the Study Coordinator that changed the evaluation of any aspect of the report, another code was entered in the database to update the original determination.
Items that were added to the records included CDRH nurse analysts’ judgment of the urgency of the report and the medical specialty to which the event pertained. Also added was a device "procode", based on the manufacturer and model number information in the report. (See Section 8.1.2 for a description of the process of assigning procodes.) Neither the analysts nor the contractor knew the identity of the facilities whose reports they reviewed.
6.3 Receipt and Data Entry
In this section we describe the procedures used to receive and process the event reports submitted by the DEVICENET facilities. We begin with a discussion of the data security procedures and then describe the data entry activities.
6.3.1 Implementation of the Data Security Plan
As promised during recruitment, CODA developed data security procedures for the project to prevent disclosure of facility-related information. The procedures were as follows:
- A room at CODA was designated as the DEVICENET office in order to provide a secure facility for receiving data. The Manager of Data Operations was stationed in the DEVICENET office. When she or another DEVICENET staff member was not present in the office, the office was locked.
- Two toll-free telephone lines were installed in the DEVICENET room, one for telephone calls and one for a dedicated fax machine.
- CODA sent each facility a supply of self-addressed business reply envelopes to be used when mailing reports to the project. These envelopes were prominently marked "CONFIDENTIAL INFORMATION – TO BE OPENED ONLY BY DEVICENET STAFF."
- A computer dedicated to the project was placed in the DEVICENET room. All DEVICENET data was kept on this computer and access to it -- and hence to the database -- was password-protected. The password was known only to the Manager of Data Operations and two other project staff members.
- A secure e-mail box was created on CODA’s network. Access to the DEVICENET e-mail mailbox was on the DEVICENET dedicated computer, and the mailbox was password-protected.
- At all times, information reported by facilities that was in printed or written form was kept in the DEVICENET project room. When not in use, it was kept in a filing case to which a high security combination lock had been attached. Only the Manager of Data Operations and one other staff person knew the combination for the filing case lock.
Facilities could report to DEVICENET in four different ways: by telephone, fax, e-mail or by mail. Mail was by far the most commonly chosen method. Once an event report was received, CODA project staff followed the procedures below:
- As soon as a case was received, the name of the facility was removed and a Facility ID was temporarily assigned. The case was then assigned a Case ID, and this ID and the date of receipt were noted on the report form.
- For purposes of analysis, facilities had been assigned to groups. The 17 participating hospitals were grouped into three categories, based on size; the six participating nursing homes formed a separate group. The code indicating the group classification was also noted on the form.
- Patient age was masked; i.e., we entered it in 5-year ranges, rather than using an exact age or date of birth. Date of incident was also masked; we kept only month and year of incident on the form, rather than an exact date. We did the same for all dates on the reports.
- We then blocked out anything on the hard copy that could identify the facility or the Study Coordinator, copied the form and shredded the original.
- The report information was keyed into the database on the computer dedicated to the DEVICENET project.
- After 30 days the Facility ID was removed so that it was no longer possible to link the report to the facility. The 30-day time period was chosen so that we would be able to link the original report with any follow-up reports and so that we would have time to discuss any questions about the report with the Study Coordinator.
If a facility wished to report an incident anonymously, it could do so in one of two ways:
- Call in the report and ask that we not put the Facility ID on the case, even temporarily, or
- Mail the report in with no identification on it.
By the end of the study year, we had received a total of 315 reports, of which 14 were anonymous reports.
6.3.2 Data Entry
Data entry was done by CODA staff familiar with the MEDWATCH mandatory and voluntary forms, using a Microsoft ACCESS database customized for use with either form type. The reports typically were entered into the database within one week of receipt.
In addition to the database used for the report contents, CODA kept a separate, secure computer file to record the number of reports received from each facility. This file could not be linked with the particular cases in the report file (there were no dates or report numbers on the file) and was designed only for analysis of the range of reporting behaviors among the various facilities. The results of this analysis are presented in Section 8.
6.4 Study Coordinator Debriefing
In November 1998 CODA conducted a debriefing session for the Study Coordinators from participating hospitals. Ten hospital representatives participated, and approximately 15 CDRH staff participated in the session as well. At this session, Study Coordinators were provided with the preliminary results of the project, a presentation concerning the uses of the MDR data, and information about the policy implications of the project. Study Coordinators were asked to discuss their experiences with the project as a whole and about opinions of the usefulness of individual aspects of the project such as the Memorandum of Understanding, the orientation session, the training video, the newsletters, the assistance with reporting and coding, and the safety notices. These Study Coordinators were generally quite positive about their experience participating in DEVICENET. They indicated that they found the project useful in their work as MDR reporters, and that they found the feedback such as the safety notices and the project newsletter very informative. They also indicated various ways that they had used the project to assist them in quality improvement activities within their facilities.
6.5 Overall Results of the Data Collection Effort
The DEVICENET project received a total of 315 reports during the period from October 1997 to November 1998 for events that occurred from Oct. 1, 1997, through Sept. 30, 1998. All of these came from hospitals; no reports were submitted by any of the six participating nursing homes. Section 8 describes the range of reports received per facility and the distribution of reports by size of facility. As shown in that section, overall the hospitals submitted an average of 18.5 reports, with the average for those that actually submitted reports equal to 22.5.
Most of the reports (286) came to the DEVICENET office by mail in special envelopes provided to Study Coordinators at the orientation sessions. Three reports were sent by fax, and in 26 cases, Study Coordinators called the DEVICENET office to request that their reports be completed by project staff. We completed the 26 reports and followed up by faxing a draft copy of each report to the Study Coordinator for review. Study Coordinators usually faxed back the edited copy or e-mailed their comments on the draft, after which we made the required changes and sent the Study Coordinator a copy for their use. In no case did we forward any reports to manufacturers or to FDA; although encouraged, the task of submitting reports was left strictly to the discretion of the Study Coordinator.
There is reason to believe that the level of DEVICENET reporting activity was far above the average for hospitals in the MEDWATCH system. CDRH estimates that the number of health care facilities required to submit reports is about 60,000, of which approximately 5500 are hospitals. If the FDA received the same average number of reports (including voluntary reports) from hospitals as DEVICENET did, the agency would be receiving more than 100,000 reports per year from hospitals alone. In fact, according to current CDRH estimates from the Manufacturer and User Facility Device Experience (MAUDE) database, health care facilities submitted approximately 5,000 adverse medical device event reports in FY 1998 and only slightly more than that in FY 1997.
7. RESULTS OF THE QUALITATIVE ANALYSIS
Several major points emerged from the qualitative analysis of the DEVICENET project results and from CODA’s concurrent work with patient safety experts. These points concern: a) the importance of providing confidentiality for reporting facilities in encouraging reporting, b) the need for feedback to reporting facilities, c) the benefits of ongoing communication between staff and designated reporters, d) the advantages of having reports submitted to an independent organization, and e) the desirability of changing the reporting rules to make them clearer and to reduce unnecessary reporting burden. These points are discussed below.
7.1 Confidentiality for Reporting Facilities
As the project team found in the focus groups and in the recruitment phase of the project, a number of barriers to reporting exist that have to do with the confidentiality and uses of the information provided. Individual reporters (eyewitnesses to adverse events) and institutional reporters charged with filtering eyewitness reports for reporting outside their institution’s walls (e.g., risk managers and attorneys for health care institutions) often have serious concerns about creating or increasing vulnerability for themselves and/or their institutions if they are reporting an adverse event. Sometimes these concerns are quite well-defined, and at other times they take the form of rather vague, ill-defined fears of submitting information that could be traced back and used against the reporter or the facility. At the eyewitness level, there is apprehension about what could happen to one’s job security, advancement, and/or reputation if one reports an adverse event one has witnessed or in which one has been directly involved. At the institutional level, concerns about discoverability of information as a result of documenting and/or submitting a report about an adverse event constitute important barriers to candor in reporting. In addition, there are worries about the reputation of the health care institution in the eyes of regulators, accreditation organizations, or business partners/purchasers. Such concerns are particularly acute when "use error" may have been involved. They pertain not only to actual adverse events, but also to "near miss" incidents or other situations indicating potential for harm since such information might be used to show the establishment of a pattern or to indicate previous knowledge of a problem in interpreting a subsequent adverse event.
For these reasons, we found that reporting institutions were extremely interested in the confidentiality procedures that CODA proposed to protect the information from disclosure and in the ways that the information provided would and would not be used. The Memorandum of Understanding, outlining the confidentiality to be provided under the project, proved to be a very useful document in conversations with facility representatives during the recruitment process. The specific procedures for ensuring confidentiality, such as removing identifying information from reports, were discussed in detail at the Study Coordinator orientation session. The fact that we allowed anonymous reports, although they were not encouraged, and the fact that CODA was willing to serve as a go-between when Study Coordinators had questions for CDRH seemed to give most of the Study Coordinators assurance that they could participate as fully as possible in the project without having their facilities identified with any individual event. In our view, the specific steps that we implemented to ensure confidentiality were extremely important to the success of the project in encouraging reports on actual harm and potential for harm.
7.2 Feedback and Ongoing Communication
Feedback and ongoing communication were key elements of the DEVICENET project. It is CODA’s impression that much of the success of the project in encouraging reporting was a result of these two elements.
Feedback took several forms: the newsletter, the faxes of CDRH safety notices, and responses to questions presented by Study Coordinators. Individual reports received by the project were summarized, stripped of identifying information, and included in the bimonthly newsletters for Study Coordinators; the ways in which the reports were used were communicated to the participating facilities throughout the project. A draft guidance document CDRH staff wrote to clarify reporting requirements for implanted devices was distributed to the Study Coordinators for their review. Most of the Study Coordinators seemed to appreciate the access to such information that the project afforded.
At the project debriefing session mentioned earlier, there was a consensus that the Study Coordinators found the newsletter very useful. Some reported that sharing the DEVICENET newsletter with staff members was starting to generate more reporting within their facilities as the study year went on. They felt that if the project were to continue beyond one year, the full effects of such feedback might become apparent.
The channel of communication described in Section 6.1 was very important in keeping the Study Coordinators involved and interested in the project. This aspect of the project undoubtedly made an important contribution to the project’s success.
7.3 Advantages of an Independent Reporting Organization
There were a number of advantages in having an independent organization receive the DEVICENET reports. First, reported to us in the exploratory phase of the project, facilities’ representatives are frequently hesitant to submit reports to a government agency, because of lack of trust regarding the security of a database that is directly under the federal government’s control (especially when the agency has regulatory authority). They often fear that information they provide may be released (e.g., to the press, to plaintiffs’ attorneys, or to potential purchasers) and lead them to regret having reported the information in the first place. The DEVICENET arrangement offered Study Coordinators the freedom to submit a report to the project with confidence that the data would not become part of a government database identified with their facility.
Second, Study Coordinators could contact CODA in a variety of ways to ask questions they did not want to raise directly with FDA. For example, they could ask whether specific events needed to be reported to the manufacturer or FDA. They knew that CODA would be able to obtain answers to their questions from FDA without having to divulge the facility’s identity. Thus Study Coordinators were free to ask questions about events without committing themselves to filing a report. Facilities could ask CODA to explore whether CDRH had any information regarding certain types of devices without revealing their reasons for wanting the information. CODA obtained information from the Internet or directly from CDRH and transmitted it to the Study Coordinators for their review, with no questions asked. This type of communication and feedback encouraged greater participation for some facilities in our opinion.
7.4 Use of the DEVICENET Reportability Guidelines
DEVICENET was a voluntary reporting program, with directives concerning reporting that were different from the MDR regulations. As mentioned in Section 3, after the exploratory phase, decisions were made that broadened the kinds of situations that should be reported under DEVICENET compared with MDR reporting. Facilities were encouraged to report to the project all events that were being reported to manufacturers and/or FDA, plus any additional events or incidents that seemed to indicate potential for harm from medical devices. Examples of such reports were provided during the orientation sessions and in the training video. Facilities did not seem to have difficulty in understanding what kinds of reports we were requesting for the project, although we continued to receive questions (which we referred to FDA specialists) about MDR reportability for particular events.
8. QUANTITATIVE ANALYSIS OF STUDY RESULTS
Twenty-three facilities -- 17 hospitals and six nursing homes -- participated in the year-long DEVICENET Study. As can be seen in Table 8.1, between October 1997 and November 1998, CODA received a total of 315 medical device event reports from 14 of the 17 participating hospitals, an average of about 22.5 reports for the 14 facilities that submitted them and 18.5 reports each for the 17 original participating hospitals. More than one-half of all reports came from one large hospital, and a second large hospital contributed another 15% of the reports.
None of the nursing homes enrolled in DEVICENET made reports to CODA during the data collection period. Some of the possible reasons for this lack of reporting are discussed in Section 8.4.
Table 8-1. Reports by Size of Participating Facility
|PARTICIPATING FACILITIES||# FACILITIES||# REPORTS||RANGE||% OF ALL REPORTS|
|TOTAL ALL FACILITIES||23||315||--||100%|
8.1 Event Reporting
This section describes the flow of reports during the reporting period, the length of time between event occurrence and receipt of the reports, the types of reports received, and how the reports fell into the MDR reportability categories of mandatory and voluntary reports.
8.1.1 Event Occurrences and Report Receipt
Event occurrences were fairly evenly distributed over the year with a range of 21 to 33 events per month (mean 27.2 per month), with the exception of November 1997 when there were only 13 events (dropping the mean to 26 events per month). Submission of reports by the facilities was less regular, with a range of 1 to 71 events being reported to CODA in a single month.
There was often considerable lag time between event occurrence and receipt of the report at CODA. Even for those reports that CODA judged to be within the mandatory FDA classification, over half of the events reported (26 of the 44) occurred in the calendar month falling two or more months before the receipt month.
8.1.2 Subjects of Reports
At the time that new medical devices are approved by FDA, the agency assigns them a "procode", which is used for FDA listings of types of devices. Manufacturers are required to use this system for identifying devices on all MDR reports they send to FDA (including reports they forward from user facilities).
Because DEVICENET reports came directly from health facilities, no procodes were added before CODA received the reports. We arranged to have CDRH’s contractor for receipt and processing of MEDWATCH reports review all DEVICENET reports and code them using the FDA procode system.
Seven of the procodes accounted for 30% (98) of the devices reported (see Table 8-2 below). However, there was a wide distribution of procodes represented in the DEVICENET reports, and more than 60 devices received unique procodes.
Table 8-2. Most Frequently Assigned Procodes
|PROCODE||DEFINITION||# OF DEVICES|
|DQO||Catheter, Intravascular, Diagnostic||25|
|FHW||Device, Impotence, Mechanical/Hydraulic||15|
|BTR||Tube, Tracheal (w/wo connector)||12|
|LJT||Port and Catheter, Implanted, Subcutaneous, Intravascular||11|
|FWM||Prosthesis, Breast, Inflatable, Internal, Saline||10|
|DTB||Electrode, Pacemaker, Permanent||10|
Another analysis of the type of devices involved in reported events was performed by two senior nurse-analysts from CDRH, who categorized the medical devices involved by medical specialty. Cardiovascular devices were the subject of the largest proportion (29%) of reports, followed by general hospital devices (17%), general and plastic surgery devices (13%) and those used for gastrointestinal and genito-urinary applications (12%).
8.1.3 Mandatory and Voluntary Reports Received
The FDA Medical Device Reporting system requires facilities to report medical device events in which serious injury or death has occurred. In addition, DEVICENET facilities were encouraged to report events in which the potential for such outcomes existed. The Study Coordinators were encouraged to use the mandatory form (MEDWATCH Form 3500A) even for voluntary reports, because only the mandatory form elicits codes for patient outcomes and device problems. It also has fields for the age of the device and the location where the event occurred. CODA analysts determined that only 14% percent of all the reports described events that appeared to require mandatory reporting. Most reports received by DEVICENET (56%) described events that would fall under voluntary reporting guidelines; however, another 30% of the events fell into a gray area in that it was not clear whether they were mandatory or voluntary. It is likely that many of these reports would not normally have been submitted to FDA.
Similarly, for 30% of the reports, the determination of serious patient injury according to FDA’s definition was difficult to make. Of the 96 reports for which the determination of patient injury was not clear, 60% were submitted on voluntary forms. One-fourth of the reports clearly documenting serious patient injury also were submitted on voluntary forms. Again, many of these events probably would not be reported to FDA under the current system.
The reports also were analyzed along another dimension. Two senior nurse-analysts from CDRH agreed to review all reports and classify them with respect to urgency using a scale of: very urgent, urgent, routine monitoring or well-known problem, or not important. (Note that although a report may be judged "not important", it may be a required report for which FDA had already taken action.) They classified approximately one-third of the reports (113) as very urgent or somewhat urgent; only 19 of these were clearly mandatory reports. The remaining two-thirds either required routine monitoring or were classified as not important to CDRH, usually because events connected to a device were well-documented and FDA had already taken action.
A comparison of the urgency scores given to event reports by CDRH nurse analysts with the evaluation of whether an event appeared to fall under mandatory or voluntary reporting requirements shows some interesting contrasts. About half of the events that appeared to require mandatory reporting (51%) needed only routine monitoring, according to their urgency ratings. An example of this kind of case would be a problem with a catheter in which there was medical intervention, but for which FDA already had taken action, so that additional reports would not make a very valuable contribution to the agency.
On the other hand, 30% of the 175 reports that appeared to fall under voluntary reporting rules were rated as very urgent (2) or somewhat urgent (50). Similarly, 44% of the 95 reports where the mandatory/voluntary distinction was difficult to make fell into the very urgent (2) or somewhat urgent (40) classification. Thus many (94 of 270 reports, or 35%) of the reports that were not necessarily mandatory under MDR regulations actually provided very important (i.e., "urgent") information to CDRH analysts.
8.2 Evaluation of Patient and Event Descriptions
CODA staff rated the reports in terms of whether they clearly communicated what happened during an event and what the effect was on the patient and whether the reports were internally consistent.
Facilities did better at describing what, if anything, happened to patients as a result of the incident than they did at describing the events themselves. Nearly 85% of the patient descriptions were judged to describe the effect on the patient fairly clearly while only 59% of the event descriptions were judged to be complete. (An additional 36% of the event descriptions were found to be somewhat complete.)
CODA also devised an assessment of the reports’ internal consistency and reasonableness based on the descriptions of the event and the patient impact. Overall, 86% of the reports were found to be internally consistent and reasonable, and another 14% were judged somewhat so.
8.3 Evaluation of Device Event and Patient Outcome Codes
When a report on Form 3500A arrived at the DEVICENET office, CODA staff forwarded a copy without identifying information to our nurse consultant and to our Medical Informatics consultant. With them we reviewed the coding and discussed any questions we had about the report. Based on the contact with the consultants, we looked at each patient code and device code assigned by the facility to determine whether it was correct.
Approximately two-thirds of the 154 patient codes assigned by DEVICENET facilities appeared correct, while 26% were incorrect (see Table 8-3 below). Almost three-quarters of the device codes assigned by facilities were judged to appear correct. Device codes were more likely than patient codes to be rated as "appearing correct, but too general" (14% vs. 1%) and less likely than patient codes to be rated as incorrect (5% vs. 26%).
Table 8-3. Evaluation of Patient and Device Codes
|Appears OK, too general||1%||14%|
|Insufficient to evaluate||3%||3%|
In reviewing the coding, project staff always looked for additional codes that would enhance the codes provided by facilities. Of the patient codes that CODA assigned on mandatory forms, 20 (56%) were added to provide information critical to an understanding of the event while another 16 (44%) provided additional information that simply enhanced the information provided by the other codes. The 80 new device codes CODA supplied were evenly split between those that provided critical information and those that provided additional information.
8.4 Discussion of Results of Quantitative Analysis
In this section we discuss what we see as the most important quantitative findings from the study.
Lack of nursing home reports. From our first contact with nursing home staff during the exploratory phase, it was clear that an almost universal belief that "It doesn’t happen here" existed. Adverse event reporting from nursing homes and other long-term care facilities is known to be relatively rare in the FDA reporting system as well. So it perhaps should come as no surprise that the six nursing homes enrolled in DEVICENET did not make a single report to the project. Clearly, our assurances of confidentiality were not enough to overcome the many barriers to reporting in these facilities.
One of the largest barriers is the fact that the nursing home sector is -- and has been for some time -- the most highly regulated segment of the health care provider industry. There is constant concern in nursing home management about any negative information that might come to the attention of authorities. At the time DEVICENET began, the nursing home sector had recently come under the purview of the Joint Commission on Hospital Accreditation, in addition to the many other State and Federal level regulations to which it was already subject.
Nursing home staff also differ from those in hospitals in ways that affect the likelihood of their reporting adverse device events. In general, the staff who deal directly with patients in nursing homes have less clinical training compared with their counterparts in hospitals; they are probably less likely to recognize an adverse event, to know that it should be reported, and to know when or how to report it. There may also be a higher level of concern regarding the impact on job security if an adverse event comes to light. It is also true that the persons charged with reporting adverse events to the MDR system in nursing homes are frequently not safety specialists, but Directors of Nursing or persons in other positions for whom this responsibility is only one part of their overall job description.
Time lapse between event date and report receipt. During the exploratory phase, many facility staff commented on the unrealistic aspect of the MDR regulations concerning elapsed time between awareness1 of a serious adverse medical device event and submission of the report to the manufacturer, and to FDA in the case of deaths. Under the current regulations, which reflect the language of the SMDA law and are not subject to CDRH discretion, this period is specified as no more than 10 days. A repeated complaint was that investigating an event almost always takes longer than the allotted 10 days, and it is often not clear a device was involved until after the investigation is complete. For example, it takes longer than that to get an autopsy report or to investigate an event by other means (e.g., attempts by clinical engineers to test the device or replicate the device problem).
1Although the date the facility became aware of the event is asked for on Form 3500A (for mandatory reports), it is not required on Form 3500 (for voluntary reports), which was used for 67% of the reports submitted to DEVICENET.
In DEVICENET, we did not ask facilities to submit reports within any particular period of time. The results show that the time period between event date and receipt date was often considerably more than one month. In some cases, reports were filled out in batches; that is, the reporter would periodically review a batch of incident reports looking for evidence of reportable events.
On the other hand, in some cases DEVICENET facilities called to request immediate assistance from project staff so that they could submit their adverse event reports within the allotted time period, and we tried to accommodate these requests promptly. These reports were usually about events that fell under the mandatory FDA guidelines. While we do not have comparable figures on the lapse between event and receipt date for the FDA reporting system, we understand that a large percentage of the reports in the FDA system do not meet the requirements of the 10-day rule.
In discussing this problem with the facilities during the exploratory phase, we tried to understand what their objections were to submitting a preliminary report quickly and then filling in the details later. The primary responses were, first, that two submissions were more time-consuming than one and, second, that if it turned out that the event did not fall under reporting requirements, the facilities had no assurance that the original report would be removed from the system once it had gone in.
Differences in ratings of patient descriptions vs. event descriptions. The MEDWATCH form requires the reporter to provide a narrative description of "the event or problem." In addition, other items on the form (e.g., the codes for the patient outcome and the device problem) provide additional information about the event. CODA analyzed how well these items on the reports described the overall event, what happened to the patient, and how the medical device was involved in the event. The information about the patient outcome, that is, what happened to the patient as a result of the event, was judged by CODA analysts to be better (at least 85% fairly clear) than the information about the overall description of the event (59% rated complete for event descriptions).
This difference does not seem to be associated with reporter education or training in how to report. Rather it seems to be related to a) a reluctance, in some cases, to document details of an event that might possibly reflect badly on the facility, and b) the fact that adverse events are often the result of complex chains of events, and reporters often do not have the full information that would be needed to produce a clear event description. In particular, they often cannot determine how a device caused or contributed to an event without an extensive investigation, which may be beyond the capabilities of individual facilities.
Coding. The problems with facilities’ coding of DEVICENET reports were consistent with other reports of problems with the use of the MEDWATCH coding system, both from our discussions with facility staff during the exploratory phase of the project and from our discussions with CDRH staff. Such problems include lack of coding (estimated at 50% of incoming reports), incorrect coding, and use of codes that are too general to be useful (e.g., "device malfunction").
Under a separate contract, CODA evaluated the patient and device coding systems and recommended that CDRH could benefit from merging the MEDWATCH codes with the coding systems and hierarchical structures available within the National Library of Medicine’s Unified Medical Language System (with the addition of a hierarchical coding system for device problem coding). The report also recommended that CDRH consider centralized coding of event reports.
Mandatory vs. voluntary reports. CODA staff attempted to apply the MDR reporting regulations to classify all the reports we received as either voluntary or mandatory under the MDR system. We judged that 14% percent of the reports described events that required mandatory reporting; 56% described events that would fall under voluntary reporting guidelines, and 30% of the events fell into a gray area, which could be interpreted as mandatory by some and voluntary by others. Of the 175 reports that were classified by CODA as voluntary, CDRH nurse analysts judged 30% to be either "somewhat urgent" or (for two reports) "very urgent". Of the reports we were unable to classify, 44% were evaluated as somewhat urgent or, in two cases, very urgent.
The purpose of this study was to evaluate the feasibility and effectiveness of a sentinel system for reporting adverse events connected with the use of medical devices. To be effective, the system had to achieve several goals:
- Increased reporting by user facilities of adverse events involving medical devices, and
- Higher quality of event reports submitted.
In order to achieve these primary goals, it was thought to be necessary to accomplish the following secondary goals:
- Increased awareness of user facility staff about the importance of reporting such events,
- Increased level of knowledge among user facilities about what events are reportable,
- Decreased burden on user facilities, and
- Better communication between facilities and the office to which they report, FDA’s Center for Devices and Radiological Health.
The orientation sessions held for Study Coordinators at the beginning of the project were designed with these goals in mind. Presentations at the sessions covered such topics as the purpose and importance of a sentinel system, the project background and goals, and the rules and regulations surrounding MEDWATCH reporting. Nursing home Study Coordinators received special training in Medical Device Reporting requirements, and time was allowed so that both hospital and nursing home Coordinators could relate their problems with reporting and ask questions of CODA and CDRH project staff.
A training video was produced for clinical staff at participating facilities to increase their awareness of the importance of reporting adverse medical device events. Entitled "The Role of Health Care Professionals in Detecting Medical Device Problems," the video described the potential for harm from medical devices and emphasized the importance of reports from staff members about adverse events. Staff members who saw the video were given a one-page summary, developed by project staff, of their own facility’s internal reporting procedures.
Another education strategy for clinical staff was the creation of a poster that DEVICENET staff sent to all the Study Coordinators. The poster, which reminded readers of the importance of reporting problems with medical equipment or products, was used throughout the facilities in places such as staff lounges, offices, and conference rooms.
Discussions between CODA project staff and Study Coordinators also were used to train Coordinators in what was reportable and how to report it. Study Coordinators could ask about the reportability of particular events, the correct coding for devices or patient outcomes, and the specifics of certain regulations. For their part, project staff could discuss questions about submitted DEVICENET reports with Coordinators and return reports to the facilities with additional coding.
This process also served to increase communication between the facilities and CDRH, with CODA as a go-between. CODA would relay questions from Study Coordinators to FDA without identifying the facility involved. Feedback from CODA and FDA gave Study Coordinators information about what was being reported to the project and how FDA was using the data. The DEVICENET newsletter was a major vehicle for providing this feedback to the facilities.
Progress in Meeting Study Goals
From Study Coordinators’ comments at the end-of-project debriefing, we received the impression that not only the Coordinators but also other members of participating facilities’ staffs had greater awareness of the importance of reporting adverse medical device events. There seemed to be a learning curve involved, with staff members becoming increasingly aware of their reporting obligations as the project newsletter was distributed, posters were hung and Study Coordinators made a point of following up on incidents over the course of the study year.
We also know by the number of reports submitted that the feasibility study was increasing the amount of reporting the facilities normally would have done. CDRH estimates approximately 5500 hospitals are required to submit MEDWATCH reports. If the FDA received the same average number of reports (including voluntary reports) from hospitals as DEVICENET did, the agency would be receiving more than 100,000 reports per year from hospitals alone. In fact, all types of health care facilities combined submitted only about 5,000 MEDWATCH adverse medical device event reports in FY 1998. However, while the goal of increasing the number of reports appears to have been achieved with participating hospitals, there was less success with nursing homes, possibly because of the differences in the category of staff members who were in a position to report events or because of the strict regulatory climate under which nursing homes operate.
Another goal of the study was to increase knowledge about what kinds of events are reportable. Through the Coordinators’ orientation sessions and regular communication with the facilities both personally and through the project newsletter, we attempted to keep facility staff informed about the kinds of events that needed to be reported. In addition to the mandatory reports required by MDR regulations, we encouraged facilities participating in DEVICENET to report events in which the potential for serious injury or death existed. Only 14% of the reports submitted described events that appeared to require mandatory reporting, while more than half (56%) described events falling under voluntary guidelines. In addition, nearly a third of the reports (30%) could have been interpreted as either mandatory or voluntary.
Study Coordinators used the technical assistance offered by the DEVICENET staff in a variety of ways. For many of them, it seemed to serve as a continuation of the training provided at the orientation program. A number of them used the technical assistance to determine which kinds of reports should and should not be submitted and to complete reports more quickly and proficiently, which helped to minimize their reporting burden. Overall, the number of report forms we were asked to complete using information provided by the facility represented only a small percentage of the total reports received. However, about one-third of the hospital Study Coordinators asked us either to complete an entire report form or to assist with patient or event coding. Some who reported only a few events during the study year asked for help in completing all of the reports they submitted. Others, particularly in the larger hospitals, used the knowledge gained in the orientation session and from their previous knowledge of MDR procedures to complete reports themselves, with very little assistance from project staff.
Some participating hospitals reported that, as a result of their participation in the project, they increased the level of effort devoted to the overall reporting process, such as by reviewing internal reports for evidence of situations that should be reported to the MDR and/or the project. However, they did not seem to perceive this as an increase in reporting burden so much as an opportunity to improve this aspect of their overall patient safety program.
Study Coordinators expressed their appreciation for the project’s progress toward the fourth secondary goal: increased communication between user facilities and CDRH. They took advantage of project staff’s position as intermediary to obtain information about medical devices and adverse events that they would not have been comfortable asking directly of CDRH personnel. They also liked the type of information they received in the project newsletter: the kinds of problems other facilities were having, what FDA was doing about device problems, articles of interest about patient safety, etc. The Coordinators interacted with project staff in the process of completing their DEVICENET reports, asking and answering questions about the event to be reported and the coding to use on the report.
The goal of making the reports more complete and of higher quality met with mixed results. Nearly 85% of the patient descriptions on the reports were judged to describe the effect on the patient fairly clearly, while only 59% of the event descriptions were judged to be complete. Additionally, for 30% of the reports, the determination of whether there was any serious patient injury associated with medical device use fell into a gray area, in which some who understood the reporting rules might say "yes" and others "no."
A theme running through all difficulties in getting facilities to make reports and to then give full descriptions of adverse events when they did submit reports was the issue of confidentiality. It is our impression, notably in the case of nursing homes, that reporting is suppressed by fears about report information falling into the wrong hands and the resulting liability of facilities and individuals. Even when reports are completed, facilities will often not give full descriptions of the events, making it difficult to determine the extent of patient injury, whether or not the event falls under reporting guidelines, and which codes are the correct ones to apply to the case.
Concerns about confidentiality can operate at the individual level, when the person who witnesses an event fails to report it because of fears about job security, and the organizational level, when legal departments will not allow certain reports outside the institution’s walls or will not permit sufficient detail in reports that are released. During every stage of the feasibility project, we had to address facilities’ concerns about confidentiality, and this will be a major issue for a permanent sentinel system.
Once a facility feels it can trust the reporting process, a channel of communication will need to be maintained so that facility staff can ask questions about reporting and the organization staff reading the reports can delve for further detail when necessary. These lines of communication should also convey information from FDA about the ways in which reports are used, new findings about device safety, and updates to the reporting process.
A final consideration for a future sentinel system is to expand the MDR guidelines for reporting to include situations indicating potential for harm. The discrepancy between events that fell under mandatory or voluntary guidelines and the urgency ratings given to them by CDRH nurse-analysts demonstrates that many voluntary reports reveal device problems that are of vital interest to FDA. By expanding reports being solicited to include those on events that show the potential for harm, FDA will increase the likelihood that it will detect problems with devices before they cause serious injury or death.