Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management
Document issued on July 18, 2000
This document replaces the draft guidance document of August 3, 1999, entitled
Device Use Safety: Incorporating Human Factors in Risk Management.
U.S. Department of Health and Human Services
Food and Drug Administration
Center for Devices and Radiological Health
Division of Device User Programs and Systems Analysis
Office of Communication, Education, and Radiation Programs (OCER)
Comments and suggestions may be submitted at any time for Agency consideration to: Ron Kaye or Jay Crowley at 10903 New Hampshire Avenue, Silver Spring, MD 20993. Comments may not be acted on by the Agency until the document is revised or updated. For questions regarding the use or interpretation of this guidance contact Ron Kaye 301-796-6289 or by email Ron.Kaye@fda.hhs.gov.
Additional copies can be obtained from the Internet. You may also send an e-mail request to CDRH-Guidance@fda.hhs.gov to receive a copy of the guidance. Please use document number 1497 to identify the guidance you are requesting.
Medical Device Use-Safety: Incorporating Human Factors Engineering
into Risk Management
Identifying, Understanding, and Addressing Use-Related Hazards
Center for Devices and Radiological Health
Office of Communication, Education, and Radiation Programs (OCER)
Division of Device User Programs and Systems Analysis
Table of Contents
Medical Device Use-Safety:
Incorporating Human Factors
Engineering into Risk Management
This guidance1 describes how hazards related to medical device use should be addressed during device development as part of the risk management process. Potential use-related hazards are best identified and addressed using human factors engineering (HFE)2. The process of incorporating these approaches in the risk management processes is explained. Documenting these efforts can demonstrate that the device manufacturer has undertaken efforts to control use-related hazards. The goal is to minimize use-related hazards, assure that intended users are able to use medical devices safely and effectively throughout the product life cycle, and to facilitate review of new device submissions and design control documentation.
Addressing use-related hazards should be undertaken within the context of a thorough understanding of how a device will be used. Essential components of this understanding include:
- Device users, (e.g., patient, family member, physician, nurse, professional caregiver)
- Typical and atypical device use,
- Device characteristics,
- Characteristics of the environments in which the device will be used, and
- The interaction between users, devices, and use environments.
Following a thorough understanding of device use, specific ways that devices could be used that are likely to result in hazards should be identified and investigated through analysis and testing. In addition to investigating known or suspected problems with device use, testing prototype devices with users can identify ways of using devices that could be hazard-related that were not anticipated. This is important because it is extremely difficult to identify all significant device use problems in advance.
1This document is intended to provide guidance. It represents the Agency's current thinking on this topic. It does not create or confer any rights for or on any person and does not operate to bind FDA or the public. An alternative approach may be used if such approach satisfies the requirements of the applicable statute, regulations, or both.
2The term Human Factors Engineering and its acronym (HFE) are used extensively in this document. On occasion the term Human Factors (HF) is also used. The two terms distinguish between application (HFE), and the scientific principles and academic research that provides the basis for it (HF).
After use-related hazards are understood, the hazards are mitigated or controlled by modifying the device user interface (e.g., control or display characteristics, logic of operation, labeling) or the abilities of users to use the device (e.g., training, limiting use to qualified users). The field of human factors provides a variety of useful approaches to help identify, understand, and address use-related problems.
This guidance does not focus on any specific kind of medical device, but applies to all medical devices and accessories that involve interaction with users (e.g., thought, perception, decision-making, and manipulation with hands). It is intended for medical device manufacturers, the Food and Drug Administration (FDA)’s Center for Devices and Radiological Health (CDRH) reviewers of pre-market submissions and design controls, and as a general reference for post-market surveillance activities associated with use-related hazards. It is assumed that readers have some understanding of design controls, risk management and HFE. Some readers might find it helpful to review references 10, 16, 19, 28 and 33 listed in Section 7.0.
A hazard is a potential source of harm. Hazards arise in the use of medical devices due to the inherent risk of medical treatment, from device failures (or malfunctions), and from device use. Hazards resulting from medical devices impact patients, family members, and professional healthcare providers. This document addresses hazards resulting from interactions between users and devices. It does not focus on hazards inherent to medical treatment or caused by device failure.
Hazards associated with device use are a common and serious problem3. Evidence from researchers (Cooper, Leape, and others) suggests that the frequency and consequence of hazards resulting from medical device use might far exceed those arising from device failures. Therefore, it is essential to ensure safe and effective device use if all hazards are to be controlled effectively. An Institute of Medicine report (reference 19, section 7.0) released in November 1999 estimates that as many as 98,000 people die in any given year from medical errors that occur in hospitals, which is more than the number who die from motor vehicle accidents, breast cancer, or AIDS. Though many of these errors are not related directly to the use of medical devices, some are, and the importance of incorporating HFE principles into device design to reduce device related medical errors was highlighted.
Medical device designers are interested in developing highly reliable devices. To do this, they consider the possibilities of hazards arising from failures of the device and its components. These kinds of failures can be identified through conventional reliability analyses. Designers need a more complete and accurate understanding of device use and approaches to include consideration of unique limitations and failure modes of device users as critical components of the device-user system. Relatively few user actions that can cause the device to fail other than the most apparent (e.g., fire or explosion), or well-known instances of use problems are considered by designers. This limitation during device design increases the likelihood of unexpected use scenarios (see Section 1.2) and use-related hazards for users and patients.
3Incidents of death or serious injury resulting from how a device is used, "user error" are reportable events under FDA's Medical Device Reporting (MDR) program.
Hazards typically considered in risk analysis include:
- Chemical hazards (e.g., toxic chemicals),
- Mechanical hazards (e.g., kinetic or potential energy from a moving object),
- Thermal hazards (e.g., high temperature components),
- Electrical hazards (e.g., electrical shock, electromagnetic interference (EMI)),
- Radiation hazards (e.g., ionizing and non-ionizing), and
- Biological hazards (e.g., allergic reactions, bio-incompatibility, and infection).
These hazards most often result from instances of device or component failure that are not dependent on how the user interacts with the device, unless the way a device is used causes it to fail. In addition to the hazards mentioned above, there are certain kinds of hazards for medical devices that often result from device use. Hazards caused specifically by how a device is used are referred to in this document as use-related hazards (Figure 1). These include misdiagnoses (e.g., failure to identify disease or measure physiological parameters accurately), failure to recognize and act on information from monitoring devices, and improper treatment (e.g., ineffective or dangerous therapy).
Figure 1. Device Failure Hazards and Use-Related Hazards
Use-related hazards occur for one or more of the following reasons:
- Devices are used in ways that were not anticipated,
- Devices are used in ways that were anticipated, but inadequately controlled for,
- Device use requires physical, perceptual, or cognitive abilities that exceed those of the user,
- Device use is inconsistent with user’s expectations or intuition about device operation,
- The use environment (see Section 3.2.1) effects device operation and this effect is not understood by the user, or
- The user’s physical, perceptual, or cognitive capacities are exceeded when using the device in a particular environment.
Use-related hazards often occur as a result of a sequence or chain of events involving device use. For instance, a user might not understand the calibration procedure required for a home-testing device. The user could calibrate it incorrectly, use the device and then act on the results it provides. Although the user’s technique following the calibration might be appropriate, inaccurate results are obtained because the device was not calibrated accurately. Decisions involving a patient’s health could be made on these inaccurate results. This use scenario involves the failure to understand the procedure, the incorrect calibration and the subsequent use of the device and the inaccurate results. In this example, a hazard occurred due to how the device was used. In this document, the concept of "use scenarios that result in hazards" refers to problematic use of the device in its entirety. An alternate terminology could be "failures" or "faults" in which scenarios resulting in hazards can be viewed essentially as failures or faults that occur during the interaction between the user and the device.
2.0 Risk Management
The guidance presented in this document describes how HFE approaches can be integrated into Risk Management to help identify, understand, control, and prevent failures that can result in hazards when people use medical devices. Risk management is a systematic application of policies, procedures, and practices to the analysis, evaluation, and control of risks. It is a key component of quality management systems, and is a central requirement of the implementation of Design Controls in the Quality Systems Regulation. Risk management involves the identification and description of hazards and how they could occur, their expected consequences, and estimations or assessments of their relative likelihood. The estimation of risk for a given hazard is a function of the relative likelihood of its occurrence and the severity of harm resulting from its consequences. Following the estimations of risk, risk management focuses on controlling or mitigating the risks.
Estimates of the risk of use-related hazards can be difficult to make. They can also be misleading. Problems with device use that could result in hazards are often difficult to anticipate due to the many ways and conditions under which users interact with devices. This causes some use-related hazards to not be included in risk management. Also, when they are anticipated, their true likelihood is difficult to estimate analytically. Even use-studies can be misleading measures of likelihood because the rate of use-related hazards observed is likely to be less than in actual use (Section 5.3.1). From a perspective of human factors engineering (HFE) in medical device use, the risk associated with low-likelihood use-related hazards could be misleading because users are often less able to react appropriately to situations that occur infrequently. Therefore, it is important to carefully consider the severity of harm in the management of device use risks.
Thorough consideration of use-related hazards in risk management processes should include the following tasks:
- Identify and describe use-related hazards through analysis of existing information (see Section 5.3),
- Apply empirical approaches (see Section 5.5), using representative device users, to identify and describe hazards that do not lend themselves to identification or understanding through analytic approaches,
- Estimate the risk of each use-related hazard scenario,
- Develop strategies and controls to reduce the likelihood or mitigate the consequences of use-related hazard scenarios,
- Select and implement control strategies,
- Ensure controls are appropriate and effective in reducing risk,
- Determine if new hazards have been introduced as a result of implementing control strategies,
- Verify that functional and operational requirements are met, and
- Validate safe and effective device use.
This process will be discussed in Section 5 in conjunction with HFE approaches.
3.0 Human Factors
To understand use-related hazards, it is necessary to have an accurate and complete understanding of how a device will be used. Understanding and optimizing how people use and interact with technology is the subject of human factors engineering (HFE). HFE considerations important to the development of medical devices include device technology, the users, environment in which the technology will be used, how dangerous device use is, and how critical the device is for patient care. An introduction to human factors (HF) considerations for medical devices can be found in the FDA document, Do It By Design.
Several general HFE concepts should be considered before proceeding with a discussion of HFE approaches in the context of risk management.
3.1.1 User Preference does not Necessarily Indicate Safety and Effectiveness A focus solely on user preference in the development of a design does not assure that safety and effectiveness have been adequately considered. Users generally prefer devices that are easy and satisfying to use and are aesthetically pleasing. Too often, device manufacturers and users emphasize these device characteristics at the expense of safety and effectiveness.
Although design features that assure safety and effectiveness could decrease user preference in some instances, they are necessary nevertheless. For instance, safety-related user interface design features such as shields over critical controls, mechanical or software-based interlocks, or verification requirements could slow down the use of a device or effect its aesthetics.
3.1.2 Use Scenarios with a Low Frequency of Occurrence that Result in Hazards Require Careful Consideration
Rare or unusual use scenarios resulting in hazards with serious consequences often prove to be the greatest threat to safe and effective medical device use after a device becomes available for general use. Users are often not prepared for infrequent, unexpected use scenarios because they are often not dealt with adequately in device design, training, or operating instructions. Infrequent but dangerous use scenarios are often difficult to identify, which underscores the necessity for careful application of the analytic and empirical approaches (Section 5.5.) early in, and throughout the design process.
3.1.3 Direct Inspection or Paper-based Analyses of a Device Might not Identify all Hazards
Use-related hazards involve interactions among aspects of the use environment, user, and the device (see Figure 2). Many hazards involving unsafe or ineffective device use can be identified through careful inspection and analyses of existing information pertaining to the use of similar devices. Some use scenarios are rare. Some involve unusual or unexpected ways of interacting with a device, or involve use in unusual circumstances. Use scenarios of this kind are difficult to identify by using only analytic approaches (see Section 5.4). Therefore, it is important to obtain information from the intended user population and test devices under actual or simulated use conditions (see Section 5.5).
Safe, effective, or unsafe, ineffective use of medical devices is determined by the following major components of the device-user system: (1) Use Environments, (2) User Characteristics, and (3) Device User Interface Characteristics. This interaction and its possible results is depicted graphically in Figure 2.
3.2.2 Medical Device Users
A device that is easy for one person to use safely and effectively might present problems for another person. Similarly, a device that is easy for a certain group of users to use safely and effectively could be difficult for another group. Users need devices that they can use safely and effectively. To assure that these needs are met, it is necessary to understand abilities and limitations of the intended users.
It is convenient to refer to the group of users who use a given device as its user population. It is then helpful to describe the user population with respect to the abilities and limitations of its members. For any device, the abilities and limitations of the user population might be relatively uniform. On the other hand, the user population might contain sub-components that have significantly different abilities. Examples are young and old users, or home users and professional healthcare providers. Fatigue, stress, medication, or other temporary mental or physical conditions can temporarily effect ability levels of device users.
Important characteristics of user populations include:
- General health and mental state (stressed, relaxed, rested, tired, affected by medication or disease) when using the device,
- Physical size and strength,
- Sensory capabilities (vision, hearing, touch),
- Coordination (manual dexterity),
- Cognitive ability and memory,
- Knowledge about device operation and the associated medical condition,
- Previous experience with devices (particularly similar devices or user interfaces),
- Expectations about how a device will operate,
- Motivation, and
- Ability to adapt to adverse circumstances.
For example, older users might have difficulty remembering specific sequences for operation, using their hands to do tasks that require fine manipulation, or sensing device outputs such as auditory alarm sounds or information displayed visually. Highly trained and motivated users (i.e., developers, sales personnel, participants in previous use-studies, expert users) are often much more capable of operating complex devices than typical users. They are also likely to adapt better to unexpected or variable circumstances. Motivated and adaptable users are more likely to take actions to compensate for problems with the design of a device. But, if the same device is placed in the hands of more typical users, unexpected use scenarios possibly resulting in hazards could occur.
With proper application of HFE, the design of a device can often be made to compensate for limitations in user ability. For example, diabetics often suffer from some degree of retinopathy (degenerative disease of the retina) resulting in impaired eyesight. These users have difficulty reading the results of blood glucose test kits when the meter displays are very small. Blood glucose meters with small displays were not a good design for this user population. After this problem was understood, subsequent models with larger displays mitigated this hazard.
User experience and expectations are important considerations. Users will expect devices and device components to operate in ways that are consistent with their experience with other similar devices or device interface components. For example, users are likely to expect that the flow-rate of a given substance (such as a gas or liquid flow) will increase by turning a control knob counter-clockwise. Hazards result when an electronically driven device control operates in the opposite direction.
3.2.3 Medical Device User Interfaces
HFE considerations relate directly to the device user interface and responses of the device to user actions. A well-designed user interface will facilitate correct actions and will prevent or discourage actions that could result in hazards.
The user interface includes all components of a device with which users interact while using it, preparing it for use (e.g., calibration, set-up, unpacking), or performing maintenance (e.g., repairing, cleaning). It includes hardware features that control device operation such as switches, buttons, and knobs and device features that provide information to the user such as indicator lights, displays, auditory, and visual alarms. The user interface also includes the logic that directs how the system responds to user actions including how, when, and in what form information (feedback) is provided to the user. An important aspect of the user interface is the extent to which the logic of information display and control actions is consistent with users’ abilities, expectations, and likely behaviors.
Increasingly, user interfaces for new medical devices are computer-based. In these cases, interface characteristics include: the manner in which data is organized and presented, control and monitoring screens, screen components, prompts, navigation logic, alerting mechanisms, data entry requirements, help functions, keyboards, mouses, and pointers. The size and configuration of the device are important parts of the user interface, particularly for hand-held devices. Device labeling, packaging, training materials, operating instructions, and other reference materials are also considered part of the user interface.
An important concept pertaining to user interface use-safety is error tolerance. Error tolerance is the quality of a user interface that prevents or mitigates dangerous or disastrous consequences when an error occurs. Humans make errors. Some kinds of error can be anticipated and are essentially unavoidable – such as inadvertently pressing an adjacent key on a keypad, or bumping the keypad inadvertently while doing other tasks. The application of HFE approaches to device design will increase the likelihood that the design is tolerant of errors that are likely to be made by users. There are many ways to do this; one example is the placement of a shield over the button that initiates a beam of radiation to prevent inadvertent activation. The logic of device operation can also determine its degree of error tolerance. For example, some devices include "interlocks," or mechanisms that prevent a critical process from being initiated without users verifying their intent to initiate it or necessitating extra control steps to be performed before proceeding. In other cases, devices can be designed to do tasks that users do not do well, such as timing certain steps in home-testing procedures, remembering set-up parameters, or test dates, or performing calculations. For complex procedures, devices can prompt users to perform the appropriate action at critical points in the procedure.
The advantages of addressing use-related hazards through application of human factors engineering (HFE) in risk management extend beyond improved safety. Device manufacturers have found competitive advantages from the application of HFE in the design of their products. Also, these efforts reduce the necessity for modifications during implementation and reduce costly updates. When HFE approaches are used in the design of devices, particularly if the perspective of users is obtained, the overall ease of use and aesthetics of a device can be improved with the same effort. Users appreciate medical devices that are easy to use, if they also know the devices are safe. With increased safety, the likelihood of incurring expenses associated with recalls or liability is also reduced. For the process to be integrated well, personnel conducting HFE efforts should be integrated into the design and risk management team.
The type and extent of HFE in design and risk management efforts necessary to control risk associated with device use will vary. Effort applied to identification, description, and mitigation of use-related hazards scenarios should be determined by reasonable assessment of the potential harm of each scenario. In general, the set of scenarios to be considered should be kept manageable, although care should be taken not to dismiss scenarios involving atypical, unexpected, or unusual device use that could result in serious consequences.
The central question to be answered in use-related hazard identification and control efforts is: "Can the intended users use the device safely and effectively?" The processes necessary to answer this question are described in the following section. For some devices, relatively small efforts could be adequate to answer the central question while others will require more effort. The extent of effort required for a given device is often difficult to estimate accurately prior to beginning the process of incorporating HFE into risk management. The variability in approach and the amount of effort required results from the unique characteristics of devices, their expected use, characteristics of the population of users, and the risks of use-related hazards.
This section provides an overview of how HFE considerations and approaches can be incorporated into the design and risk management processes. These four central steps are essential:
- Identify anticipated (derived analytically) and unanticipated (derived empirically) use-related hazards,
- Describe how hazardous use scenarios occur,
- Develop and apply strategies to control use-related hazards, and
- Demonstrate safe and effective device use (validation).
HFE efforts are used to identify, describe, and mitigate use scenarios that result in hazards. Figure 3 shows the structure of a use scenario resulting in a hazard for a medical device. The figure shows how the use of a device is influenced by human factors characteristics that can be separated into the three broad human factors areas: 1) Use Environment, 2) User Characteristics, and 3) Device User Interface Characteristics. When identified, these influences can be described as causes or contributing factors to the use scenario. HFE approaches are used to identify these use scenarios, to understand the causes and contributing factors, and to develop mitigation strategies.
Figure 3. Use Scenario Resulting in a Hazard
Figure 4 depicts the risk management process for addressing use-related hazards. Certain HFE approaches should be applied to allow this process to work effectively. Table 1 provides a cross-reference between risk management activities and the sections in which corresponding HFE approaches are discussed.
Figure 4. Addressing Use-Related Hazards in Risk Management
Risk Management Activities and Associated HFE Approaches
|Risk Management Activity||Applicable HFE Approaches|
|1. Identify and describe use scenarios resulting in hazards.||Section 5.1: Device use Description|
Section 5.2: Standards and Guidelines
Section 5.3 – 5.5 Analytic and Empirical Approaches
|2. Prioritize and assess use-related hazards.||Section 5.3 – 5.5 Analytic and Empirical Approaches|
Section 5.6: Prioritization and Assessment
|3. Develop and implement mitigation and control strategies for use-related hazards.||Section 5.2: Standards and Guidelines|
Section 5.3 – 5.5 Analytic and Empirical Approaches
Section 5.7: Mitigate and Control Use-related Hazards
|4. Verify mitigation and control strategies.||Section 5.3 – 5.5 Analytic and Empirical Approaches|
Section 5.8: Verification and Validation
|5. Determine if the risks related to device use are acceptable.||Section 5.2: Standards and Guidelines|
Section 5.3: Analytic Approaches
|6. Determine if new hazards have been introduced.||Section 5.2: Standards and Guidelines|
Section 5.3 – 5.5 Analytic and Empirical Approaches
|7. Validate safe and effective device use.||Section 5.5: Empirical Approaches|
Section 5.8: Verification and Validation
Describing the intended use of a device is an essential initial step for understanding device use accurately and completely. This description should include the following information:
- Overall device operation,
- General use scenarios that describe how the device will be used,
- Needs of users for safe and effective device use and how they are met by the device,
- Design (or preliminary design) of the user interface,
- Characteristics of the intended user population (particularly that which could affect device use), and
- Expected use environments.
The device use description can be developed from documents on device operation that do not necessarily focus on user interaction as long as they describe the intended use of the device. Input from design team personnel can be very useful at this stage; however, input from intended users should also be obtained. The level of detail contained in the device use description should be sufficient to explain interactions between the user and the device user interface.
Some use-related hazards are evident from the device use description. It also provides a basis for analytic approaches and is necessary for creating valid test scenarios for usability testing (see Section 5.5.2). For example, if a device is intended to be used on emergency vehicles including helicopters, potential use-related hazards could involve failure to hear audible alarms, or inability to perform device connections or manipulations if they require significant time, attention, or manual dexterity. These potential use-related hazards will guide subsequent HFE activities. For instance, when developing scenarios for usability testing, the possible impact on device use caused by noise and motion of a helicopter environment should be simulated, or an actual helicopter should be used for the testing.
The development of the device interface should include review and incorporation of relevant standards and guidelines that are applicable to the design. To facilitate pre-market review and assist manufacturers, FDA has published device-specific and general guidances, some of which contain specific recommendations for device user interface characteristics. FDA has also officially recognized device-specific and general standards published by standards bodies such as Association for the Advancement of Medical Instrumentation (AAMI) and International Electro-technical Commission (IEC). FDA general and specific guidances as well as standards recognized by FDA are listed on FDA’s home page.
Some device-specific standards contain information for developing specific user interface features such as auditory alarms (preferred loudness and pitch), visual displays (size or brightness), printed or displayed text (size, color, and contrast), as well as the overall layout of the user interface. Some general standards also contain considerations applicable to the design of the user interfaces.
It is difficult for standards and guidelines to stay current with changes in technology that influence interface design. Designers should carefully evaluate the applicability and appropriateness of existing standards and guidance to any new device user interface design.
This document describes two broad classifications of approaches for identifying, understanding and evaluating use-related hazards for medical devices: analytic and empirical. These approaches are discussed separately; however, they are interdependent and should be used together.
Analytic approaches (see Section 5.4) involve the description and systematic decomposition and analysis of device use. They are based on the expected use of new devices and on existing information about the use of similar devices. They should be used to identify use-related hazards early in development of the user interface and operating logic of a device. The application of analytic approaches is particularly useful for identifying and resolving use-related hazards that occur infrequently, are too dangerous to "force" in an evaluation involving actual users, or might be too difficult to simulate.
The unpredictability of human behavior, the complexity of medical device user interfaces, and the variability of use environments produce use-related hazards that can be difficult or impossible to identify or understand analytically. Empirical approaches (see Section 5.5) derive information from actual or simulated use of devices. Because empirical approaches evaluate actual or simulated device use they allow for previously unanticipated use scenarios resulting in hazards to be identified and described and for identified use-related hazards to be understood.
Some use-related hazards can be understood through the application of analytical approaches only, while others might not be detected or could require empirical approaches to be sufficiently understood. Both kinds of approaches are used to identify and understand safety-critical user actions, use scenarios resulting in hazards, and the contexts in which these occur. The results of analytic approaches are useful to guide the development and application of empirical approaches.
5.3.1 Analytical and Empirical Approaches in Clinical Evaluation Research
Medical device manufacturers conduct clinical evaluations in a variety of ways for devices (or prototypes). These evaluations can be done prior to approval or after a device is on the market. Regardless of when clinical evaluation is undertaken, the main goal is to evaluate or demonstrate the safety and effectiveness of the device. For any device that requires user-interaction, the use of the device by intended users is a vital part of its overall safety and effectiveness. Therefore, clinical evaluations should include a means to demonstrate that use-related hazards do not occur or that the strategies for their control or mitigation are adequate.
To the extent that appropriate HFE approaches are used in clinical evaluations, their results can validate that device use is safe and effective in the hands of the intended users. To clearly demonstrate that intended users can use a device safely and effectively, the design of clinical evaluations should include user-based approaches. Measures of overall safety and effectiveness can be painstakingly planned, recorded and analyzed, while issues corresponding to use, which could be simultaneously collected with relative ease, are omitted. Of particular importance, occasional problems or anomalies experienced by users are usually not captured, described, or investigated.
Certain characteristics of clinical evaluation research should be carefully considered when the intent is to demonstrate safety and effectiveness of device use:
- Device users involved in manufacturer-sponsored studies might be biased,
- Device user-participants might not accurately represent the population of intended device users. (They are often more capable, motivated, or informed than intended users in general. Some users could have been involved in the development of the device.),
- Personnel who collect data might overtly or inadvertently help users use the device,
- The training received by users participating in evaluations could be more recent or more extensive than what would be reasonably expected for actual users.
The approaches used in clinical evaluation research should include provisions for collecting information specific to device use. Users should be encouraged to identify and describe problems they experience when using the device: close calls, confusion, or repeated difficulties of any kind. Use problems identified should be investigated. If use problems can be dealt with easily, they can be resolved and reevaluated during the study. If not, strategies for mitigation and review should be developed.
There are a variety of analytic approaches that are used by human factors and systems engineers. Analytic approaches used for HFE investigations including function and task analysis, heuristic analysis, and expert reviews. These approaches can be applied within more comprehensive approaches such as Operational Analysis, Analysis of Similar Systems, Failure Modes Effects Analysis (FMEA), Fault Tree Analysis (FTA), Critical Incident Technique, Hazard and Operability Studies (HAZOP), and others. Regardless of the choice of technique, the first step should be to identify and describe use scenarios that result in hazards from the information developed in the device use description. Subsequent analyses will evaluate these scenarios further.
5.4.1 Identify and Describe Use Scenarios Resulting in Hazards
Two perspectives are necessary to identify and describe use scenarios that could result in hazards. The "top-down" perspective identifies possible hazards first, then the analyst determines all the possible use scenarios that could lead to that hazard. The "bottom-up" perspective begins with known, likely, or suspected use scenarios that involve difficulty using a device prototype, similar devices or similar components, and then determines the hazards that can result from these problems analytically.
The best source of information on use-related hazards associated with similar devices (known hazards) is likely to be complaint and customer feedback files. Other sources of information on known hazards are discussion (focus groups) with device users, journal articles, proceedings of professional meetings, newsletters, and relevant internet sites, such as:
- FDA’s Medical Device Reporting data files,
- FDA’s Manufacturer and User Facility Device Experience Database,
- CDRH Safety Alerts, Public Health Advisories, and Notices,
- FDA Enforcement Reports – recalls and legal actions,
- ECRI’s Medical Device Safety Reports
- The Institute of Safe Medical Practices (ISMP's) Medication Safety Alert (http://www.ismp.org/MSAarticles/Calendar/calendar.html), and
- Joint Commission on Accreditation of Healthcare Organizations (JCAHO’s) Sentinel Events.
The device use description (see Section 5.1) and task analyses provide information to help the analyst identify and describe use-related hazards. With respect to the overall HFE process, use scenarios identified from this analysis can be thought of as anticipated use scenarios. Unanticipated use scenarios that result in hazards are identified and described through the application of empirical approaches such as usability testing (see Section 5.5).
Answering the following questions can help identify and describe potential scenarios that could result in hazards (Note: This list is not exhaustive):
- Why have problems occurred with the use of other similar products?
- What are the critical steps in setting-up and operating the device? Can they be performed adequately by the expected users? How might the user set the device up incorrectly and what effects would this have?
- Is the user likely to operate the device differently than the instructions indicate?
- Is the user or use environment likely to be different than that originally intended?
- How might the physical and mental capabilities of users affect their use of the device?
- Are users likely to be affected by clinical or age-related conditions that impact their physical or mental abilities and could affect their ability to use the device?
- How might safety-critical tasks be performed incorrectly and what effects would this have?
- How important is user training, and will users be able to operate the device safely and effectively if they don’t have it?
- How important are storage and maintenance recommendations for proper device function, and what might happen if they are not followed?
- Do any aspects of device use seem complex, and how can the operator become "confused" when using the device?
- Are the auditory and visual warnings effective for all users and use environments?
- To what extent will the user depend on device output or displayed instructions for adjusting medication or taking other health-related actions?
- What will happen if necessary device accessories are expired, damaged, missing, or otherwise different than recommended?
- Is device operation reasonably resistant to everyday handling?
- Can touching or handling the device harm the user or patient?
- If the device fails, does it "fail safe" or give the user sufficient indication of the failure?
- Could device use be affected if power is lost or disconnected (inadvertently or purposefully), or if its battery is damaged, missing or discharged?
5.4.2 Function and Task Analyses
Descriptions of exactly what functions and tasks are vary among function and task analysis techniques available. These differences are not critical; the important contribution of applying function or task analysis techniques is the systematic breakdown of the device-use process into discrete steps or sequences for the purposes of description and further analysis. With respect to safety, function and task analyses can contribute by:
- Identifying critical aspects of device use potentially resulting in hazards to users and patients,
- Providing a basis for the analysis of use-related hazards, and
- Evaluating known incidents or accidents to understand what led to the problem.
A simplistic example of a task analysis component for a hand-held blood glucose meter includes the following tasks:
- Patient’s finger is lanced with automatic lancing device (device +user)
- Blood sample is placed on test strip (user)
- Test strip is placed in device (user)
- The sample is allowed to react with reagents in the test strip for a specific time (device+user)
- Blood glucose level in the sample is measured (device)
- The resulting value is displayed (device)
- The displayed value is read, interpreted, and acted upon (user)
This set of tasks includes examples that are performed by the "user" by the "device" or by a combination of the user and the device ("user+device").
After functions and tasks have been identified, they are analyzed to determine if, and how HF considerations apply. For instance, in Task 2 above, the user places a sample of blood on a test strip. There are five fundamental questions that should be investigated:
- Are any use-related hazards scenarios possible?
- How might they occur?
- How likely are they?
- What are the possible consequences of each?
- How might they be prevented?
To begin to address these, the analyst should pose more specific questions, such as:
- How difficult is it for users to use the device components and accessories to do this task correctly?
- How much effort is required by the user to apply a sample correctly?
- What characteristics of the user population might cause some users to have difficulty with this task?
- Where will the testing be done, and could ambient conditions effect the test results or the users ability to perform the task?
- Is the proper use of test strips evident to the user?
- Will certain user interactions with the device degrade the accuracy, safety and effectiveness of the devices’ subsequent operations (and if so, what are these interactions and how are device operations affected)?
In early glucose monitors, the user had to perform Task #4 manually (the sample is allowed to react with reagents in the test strip for a specific time). Users had difficulty doing this task well, and the accuracy of the results too often suffered from the users’ failure to time the process accurately. In subsequent models, this task was done automatically by the device. Modification in device design and operation removed that use scenario and the resulting hazard.
Analyzing functions and tasks in this way will allow identification of possible hazards associated with device use. Function and task analyses can provide a foundation for subsequent HFE efforts. For instance, test scenarios (see Section 5.5) should be developed to address use scenarios that involve tasks identified as critical or error-prone.
5.4.3 Heuristic Analysis
Heuristic analysis is an analytic process in which the device’s user-interface is formally evaluated from the perspective of users. The object is to identify possible use-related hazards with a focus on the interaction of the user with the user interface and operating logic of the device. Design team members often perform heuristic evaluations, but they are more effective if they involve clinical and HFE personnel. This technique is particularly useful for early identification of difficult or counter-intuitive aspects of the device user interface. Another application is the evaluation of candidate interface design alternatives. The output of heuristic analysis is limited because evaluators typically do not represent real users, use scenarios considered might not be comprehensive, and the evaluation environment is not representative of actual use.
Heuristic analyses should include careful consideration of accepted concepts for design and operation of the user interface, sometimes known as "de-facto" standards or "population stereotypes" which are essentially social and cultural norms and constraints for the use of device components. A simple example is a light switch oriented in a vertical direction being "on" when it is in the "up" position and "off" when in the "down position". For medical devices, general de-facto standards are applicable at times, while others are unique for certain kinds, or types, of medical devices.
5.4.4 Expert Review
Expert reviews rely on clinical and HF experts to analyze device use, identify problems, and make recommendations for addressing them. The process is quite similar to the heuristic analyses. The difference is that expert review relies more heavily on the assessment of individuals with expertise in a specific area. The success of the expert review depends on the expert’s knowledge of the device technology, its use, clinical perspectives, characteristics of the intended users, as well as the expert’s ability to predict actual device use. This kind of review can provide very useful information, particularly early in the design process, but might not be comprehensive since it does not involve actual device use and might not include the perspective of actual users.
Use studies are applicable to several risk management activities. They can be used early in the design process to identify unanticipated use-related hazards. They can also be used to clarify suspected or known problems with device use, demonstrate that use-related hazards have been addressed, evaluate candidate design alternatives, and to validate safe and effective use by intended users. Beyond application to the safety and effectiveness of device use, use studies provide a powerful means for creating effective labeling (including directions for use), and device designs that are user friendly, satisfying to use, and desirable to users. For the consideration of device use-related risks to be complete, empirical methodologies should include efforts that focus on identification and analysis of unanticipated use-related hazards and the incorporation of the results into the overall risk management process. Use Studies can identify problems that were noticed by test participants but did not manifest themselves as errors during use.
Use studies will provide accurate results to the extent that test participants represent actual device users, the test conditions represent realistic use environments, and the test is well run. Members of the team who are developing the device should not participate as users since their knowledge of how the device operates (or should operate) will influence how they use it. If the intended users have certain limitations in their abilities, one focus of the testing should be to establish whether these limitations affect device use. If so, further effort is required to determine whether potential use problems associated with user limitations can be mitigated by modifying the design of the device interface or the operation of the device.
Although user studies are effective in identifying and understanding device use, care should be taken not to underestimate the frequency of problems based on the experiences of test participants. Participants could be (despite good efforts of test coordinators) unrealistically well trained, capable, or careful. Also, when people are observed they often try to "do their best" and often do not use the device long enough to experience infrequent problems.
When applied to medical devices, empirical approaches should support identification, understanding, and mitigation of hazards resulting from device use. Demonstrating how well users like using a device is not sufficient to do this. However, both use-safety and user preference can be addressed through proper application of empirical approaches.
A simple kind of study involving users is the walk-through. It is less time-consuming and less formal than Usability Testing. In a walk-through, a user or small group of users are "walked-through" the process of using a device. During the walk-through, participants are questioned and encouraged to provide feedback on difficulties they notice while using the device. Evaluators can also collect subjective information from participants about thought processes, mental models, and perceived workload when using the device. The walk-through technique can provide valuable information but is limited by a lack of realism. The walk-through technique is most useful early in the development process, and for developing and evaluating usability testing scenarios.
5.5.2 Usability Testing
Usability testing (also called user testing) is a powerful technique used to assess user’s interaction with a product. This technique can also be used to identify and understand previously unanticipated or poorly understood use scenarios resulting in hazards if care is taken to focus on the safety and effectiveness perspectives. The central advantage of usability testing is that device use is realistic and the results of the process are more representative of actual use than results obtained through analytic approaches. If usability testing is employed early in the development process, it can identify potential use-related hazards so that they can be addressed early in the design life cycle.
Usability testing involves systematic collection of data from users (participants) using a device (or device component) in realistic situations. Data are obtained in a variety of ways, including user feedback, manual and automated measures of user performance, and observation. Often, the most convenient data collection methods focus on subjective user feedback. User feedback includes descriptions by test participants of difficulties encountered, good and bad aspects of the device user interface characteristics, including the logic of device operation, and suggested changes. Careful collection of subjective assessment of device use can identify problems that were noticed by test participants ("concerns," or "close calls") but did not manifest themselves as errors during use and not identified in objective performance measures.
Objective user performance measures include the type and number of errors, time required to do tasks, requests for help, accuracy, and the success or failure on individual tasks and overall performance. The application of specific, objective user performance measures enhances and focuses subjective user feedback. Performance measures are particularly useful for complex devices, where users might not be aware of (and therefore unable to evaluate) potentially hazardous use scenarios. These measures are also important for home-use devices where users are often not aware that they are inadvertently effecting the performance or accuracy of the device in some way. Outlier data from performance measures is often informative and should be investigated to determine the nature and pattern of the use scenarios associated with them.
Usability testing can be done in a variety of ways in various degrees of complexity and formality. However it is done, it should include the following:
- An overall goal of improving the usability, including safe and effective device use,
- Test participants represent intended users,
- Test participants do real tasks, particularly tasks that will indicate whether safe and effective use is achieved,
- A focus on high risk use scenarios,
- Testers who observe and record important aspects of what test participants do and say (participants can also respond to questionnaires, or be interviewed following the use of the device), and
- Data collected to support the identification of potential use-related hazards and the development of specific recommendations to address them.
The validity of use testing depends on the extent to which realistic or simulated environments are used during the testing. For example, in clinical settings users must perform multiple tasks simultaneously. These tasks involve individual devices, multiple devices, and duties unrelated to device use. Users must constantly trade-off accuracy for speed. In home environments, users might be distracted or have medical conditions that affect their abilities to interact with the device. Home users can also drop devices or expose them to various temperatures and humidity levels in various parts of the home. Clinical and home users might try to cut costs. There are many aspects of the use environment that can affect device use. By the time use testing is undertaken, anticipated use environments should be understood (device use description).
Use-related hazards identified by analytic and empirical approaches should be assessed to determine their priority for subsequent risk control efforts. This process can involve obtaining and combining input from several individuals who provide perspective from a variety of areas of expertise. In addition, it should also incorporate valid and useful information about likelihood and consequences (i.e., risk) of use-related hazards for similar devices when available.
Important perspectives include those from:
- Clinical experts,
- Expert users,
- Engineers involved with design and operation, and
- HFE or usability specialists.
These individuals should then assess the likelihood of these hazards and their consequences to estimate the risk for each. Within the general process described in this guidance, assessing preliminary results through group consensus is most useful for:
- Identifying hazards for which mitigation is necessary,
- Identifying hazards that have been successfully addressed,
- Developing strategies and controls to eliminate, reduce the likelihood of, or mitigate the consequences of use-related hazards, and
- Verifying that controls are effective in reducing or eliminating hazards.
Identifying and addressing use-related hazards early in the design process reduces the time and expense necessary to correct them. The most effective strategies to address use-related hazards focus on improvements to the design of the device user interface. The user interface should convey the concept for correct operation through its appearance and operation ("look and feel") so that safe and effective use is intuitive. Addressing use-related hazards through this kind of modification is preferred because it reduces or eliminates the need for users to rely on instructions, labeling, or training "patches."
Use-related hazards often require a combination of mitigation and control strategies. The following list presents the order of overall priority for applying strategies to control or mitigate risks of use-related hazards:
- Modify device design to remove hazard or reduce its consequences: Making the interface intuitive and ensuring that critical information is effectively communicated to the user can reduce the likelihood or eliminate certain use-related hazards. If hazards cannot be eliminated, the design should act to mitigate the consequences.
- Make user interface, including operating logic, error tolerant (safety features): When users can make errors using the device, such as pressing an adjacent key on a keypad, the device should act to preclude a hazardous outcome from occurring. Safety mechanisms such as physical safety guards, shielded controls, or software or hardware interlocks will make the design more tolerant of errors that users occasionally make.
- Alert users to the hazard: When neither design nor safety features will effectively eliminate a use-related hazard or mitigate the consequences, the device should detect the condition and provide an adequate warning signal to alert users.
- Develop written procedures and training for safe operation: Where it is impossible to eliminate hazards through any of the previous strategies, or to enhance other control or mitigation strategies, written procedures, labeling enhancements, and training for safe operation should be used.
Instructions, labeling, and training can influence users to use devices safely and effectively and are critical HFE considerations for safe device use. But because they rely on the user to consistently use the device as directed, these approaches are less effective than modifications to the design of the user interface. Therefore, mitigation of use-related hazards should not focus on instruction, labeling, or training fixes in isolation, since these "patches" might not be adequate. Often, a combination of these strategies is the best solution. Regardless of the strategy used, subsequent testing should be done to ensure that the use-related hazards have been successfully controlled.
Verification confirms that the specific functional and operational requirements for the design of a device user interface have been met. The process for verifying individual user interface requirements will likely require focused effort for each. For instance, if a device will be used by a user population of elderly users with hearing ability ranging from normal to moderate impairment, a specification should be developed to assure that the device’s alarm volume can be adjustable to a sufficient level to accommodate these users. The verification process would involve testing the device alarm to ensure that the volume adjustment capability (and any other specifications developed to assist users) has been implemented successfully.
Validation establishes that the device meets the needs of the intended users. The primary need of medical device users is the ability to use the devices safely and effectively under the actual use conditions. Applying usability testing approaches can directly validate a user interface design. For the purpose of validation, it is particularly important to use a production version of the device, representative device users, actual or simulated use environments, and to address all aspects of intended use. If small-scale iterative testing of interface components is done adequately as the device was developed, it might not be necessary for validation efforts to be extensive at the end of the design process. However, some degree of testing of the entire system under realistic conditions with representative users is warranted. In the alarm volume example above, determining whether users with moderate hearing loss can hear the alarm well enough to allow them to use the device safely and effectively is the essential component of validation of this user interface requirement.
Documenting the incorporation of human factors engineering (HFE) in risk management can help demonstrate that a manufacturer has adequately addressed the needs of the intended users. Submitting this documentation can streamline and facilitate that part of the pre-market review process concerned with safe and effective device use.
When information pertaining to device use safety is extensive, it is helpful to provide it in summary form that highlights the most important issues, considerations, resolutions, and conclusions. When portions of this information are contained in various parts of a submission a comprehensive cross-reference should be provided.
The level of detail of device use documentation submitted should be consistent with the level of concern of use-related hazards for the device. The information that should be included with the device use documentation is described below.
6.1 Device Overall
- The purpose and operation of the device,
- The patient populations on whom the device will be used,
- The physical device, e.g., size, shape, weight, important components, and how it is powered,
- A comparison of device use with other devices currently in use that operate similarly or perform similar tasks, and
- A description of how the device addresses the needs of intended users.
- The physical characteristics of the user interface,
- The operating logic of the user interface, and
- Existing or anticipated labeling materials that will be provided to the user with the device, e.g., labels on the device itself, packaging, operating instructions, and training materials.
6.3 Device Use
- How the user interacts with the device user interface,
- How the device is set up and maintained, and
- The primary tasks that the user is expected to perform.
- The intended population of device users,
- The characteristics of device user population that were considered during the design,
- The training and information tools that the user population will require to operate the device safely and effectively, and
- The population of users for which the device is not intended to be used.
- Environments for which the device is intended to be used (e.g., home, hospital, medevac vehicles),and
- Environments for which the device is unsuited, or which can be expected to affect device performance.
- The use-related hazards that have occurred with similar, already marketed, devices,
- The processes used to identify and prioritize use-related hazards,
- The use-related hazards that have either been identified during development or have occurred with this device during early testing,
- How significant use-related hazards were mitigated or controlled during design and development, and
- Why strategies used to address use-related hazards are appropriate.
- Testing and evaluation processes and results associated with determining whether device use design considerations have been achieved, and
- Testing and evaluation processes and results associated with determining whether intended device users can use the device safely and effectively in actual or simulated conditions.
- AAMI/FDA Conference Report: Human factors in medical devices: design, regulation, and patient safety, Association for the Advancement of Medical Instrumentation, Arlington, VA, 1996.
- AAMI HE48, Human factors engineering guidelines and preferred practices for the design of medical devices. Association for the Advancement of Medical Instrumentation, Arlington, VA, 1993.
- Bahr, N., System safety engineering and risk assessment: A practical approach. Washington DC: Taylor and Francis; 1997
- Chapanis, A., The business case for human factors in informatics. In: Shackel and Richardson (eds.). Human Factors for Informatics Usability. New York: John Wiley & Sons; 1994.
- Chapanis, A., Human factors in systems engineering. New York: John Wiley & Sons; 1996.
- Cooper, J., Preventable anesthesia mishaps: A study of human factors. Anesthesiology 1978, 49:399-406.
- Cooper, J., An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection. Anesthesiology 1984, 60:34-42.
- Dumas, J., and Redish, J., A practical guide to usability testing. Ablex, New Jersey, 1993.
- Ehrlich, K. and Rohn, J., Cost justification of usability engineering: A vendor’s perspective. In: Bias and Mayhew (eds). Cost justifying usability. New York: Academic Press; 1994, 73-110.
- FDA, Design Control Guidance for Medical Device Manufacturers. March 11, 1997.
- FDA, Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices. May 11, 2005.
- FDA, Quality system regulation (current good manufacturing practice (CGMP) final rule). October 7, 1996.
- Goodstein, L., Anderson, H. and Olson, S., Tasks, errors, and mental models. New York: Taylor and Francis; 1988.
- IEC TR 513, Fundamental aspects of safety standards for medical electrical equipment. International Electrotechnical Commission; 1994. Geneva, Switzerland.
- ISO Guide 51, Guidelines for the inclusion of safety aspects in standards, first edition. International Organization for Standardization, 1990. Geneva, Switzerland.
- ISO 14971-1, Medical Devices – Risk Management – Part 1: Application of Risk Analysis, first edition. International Organization for Standardization, 1998. Geneva, Switzerland.
- Kirwan, B., and Ainsworth, L.K., A guide to task analysis. London: Taylor & Francis Ltd; 1992.
- Klemmer, E. (ed.), Ergonomics: Harness the power of human factors in your business. Norwood, NJ: Ablex, 1989.
- Kohn, L., and Corrigan, J., (eds.), Building a Safer Health System. Institute of Medicine (IOM), Committee on Quality of Health Care in America. Washington, D.C.: National Academy Press; 1999.
- Leape, L., Error in Medicine. Journal of American Medical Association, 1994 Dec; 21(3) 272.
- Leveson, N., Safeware: system safety and computers, New York: Addison-Wesley; 1997.
- Meister, D., Human factors testing and evaluation. Amsterdam: Elsevier; 1986.
- MIL-STD 1472D, Human engineering guidelines for military systems, equipment and facilities. Washington, D.C.: Department of Defense, January, 1996.
- MIL-STD 882C, System safety program requirements. Washington,DC: Department of Defense. January, 1993.
- Modarres, M., What every engineer should know about reliability and risk analysis. New York: Marcel Dekker, Inc; 1993.
- Mosenkis, R., in Grutting, C., Medical devices: international perspectives on health and safety. Amsterdam: Elsevier; 1994.
- Nielsen, J. and Mack, R., Usability Inspection Methods. New York: John Wiley & Sons; 1994.
- Norman, D., The Design of Everyday Things. New York: Doubleday; 1988.
- O’Brien, T., and Charlton, S., Handbook of Human Factors Testing and Evaluation, New Jersey: Lawrence Erlbaum Assoc; 1996.
- Reason, J., Human Error. Cambridge, Mass: Cambridge University Press; 1992
- Rubin, J., Handbook of Usability Testing. New York: John Wiley and Sons, Inc; 1994.
- Salvendy, G. (ed.), Handbook of Human Factors and Ergonomics. New York: John Wiley and Sons, Inc; 1997.
- Sanders, M., and McCormick E., Human Factors in Engineering and Design. New York: McGraw Hill; 1993.
- Sawyer, C., Do It By Design: An Introduction to Human Factors in Medical Devices. FDA; 1997.
- Schneiderman, B., Designing the User Interface: Strategies for Effective Human-Computer Interaction. New York: Addison-Wesley; 1993.
- Trautman, K., The FDA and Worldwide Quality Systems Requirements Guidebook for Medical Devices. ASQC Press; 1997.
- Wickens, C. D., Gordon, S. E., and Liu, Y., An Introduction to Human Factors Engineering. Addison-Wesley (Longman Imprint): New York, NY; 1998.