2004N-0181 - Critical Path Initiative; Establishment of Docket
FDA Comment Number : EC11
Submitter : Dr. Michael Hufford Date & Time: 08/02/2004 06:08:53
Organization : invivodata, inc.
Private Industry
Category :
Issue Areas/Comments
Comment Issues
6. For each solution identified, please indicate which could be accomplished quickly, in less than 24 months, and which require a long-term approach?
Electronic diaries are in use today and the effect of their widespread adoption in clinical trials could be accomplished quickly. The lack of a clear stance by FDA is one of the roadblocks to adoption of this improved methodology.
1. Hurdle Identification. Please describe the product development issue, the nature of the evaluation tool that is out-of-date or absent, how this problem hinders product development, and how a solution would improve the product development process.
Many clinical trials rely on Patient Reported Outcomes (PRO) data for primary or secondary outcomes. However, collection of PRO data too often relies on outdated paper diary methods whose validity and scientific properties have been seriously challenged. Electronic diary methods that yield higher-quality data could make clinical trials more efficient and reliable, but uncertainty about the FDA's regulatory view of the newer methods has impeded their adoption, and thus led to continuing conduct or larger, slower, and less-reliable clinical trials in many therapeutic categories.

As outlined in detail below, the use of electronic diaries in clinical trials have been shown to: (1) improve data quality by eliminating illegible and out of range responses, (2) prevent back- and forward-filling of diary entries (e.g., Stone et al., 2002, 2003), and (3) enable more sensitive tests of clinical trial endpoints by reducing error variance, allowing sponsors to run trials with up to half the number of subjects while retaining their trial power (e.g., McKenzie et al., 2004; Pearson, 2004).

Like all data captured in clinical trials, PRO data needs to meet established regulatory standards. The FDA requires that PRO data be accurate, reliable, valid, and sensitive measures of endpoints in clinical trials to help evaluate the safety and efficacy of new drugs and medical devices. For example, the FDA Guidance (USFDA, April, 1999) states: 'FDA's acceptance of data from clinical trials for decision-making purposes is dependent upon its ability to verify the quality and integrity of such data during onsite inspections and audits. To be acceptable the data should meet certain fundamental elements of quality whether collected or recorded electronically or on paper. Data should be attributable, original, accurate, contemporaneous, legible.' These latter requirements are also known as the ALCOA standards for data quality and integrity.

Two fundamental problems prevent paper diaries from being an effective way to capture reliable and valid PRO data in clinical trials: poor data quality and poor or falsified compliance with the protocol (Hufford & Shiffman, 2003). First, as much as 80% of paper diary cards contain illegible or out-of-range entries (Quinn et al., 2000). As a result, many paper diary cards fail to provide data that can be used to help establish the efficacy and safety of new treatments. From a regulatory perspective, paper diary data often fail to meet the legibility criterion of the ALCOA standards for data quality and integrity.

Second, many paper diary cards are completed in violation of the protocol. A recent study published in the British Medical Journal examined both apparent and actual compliance rates using paper diaries (Stone et al., 2002): 90% of paper diary cards were marked to indicate that they had been completed in a timely way. However, instrumentation in the paper diary revealed that only 11% could have been completed within the protocol-
prescribed windows. Thus, 79% of paper diary cards were falsified. Finally, the data indicated that not just the time and date information, but also the data themselves had been falsified: almost half the patients had engaged in 'forward-filling' their diaries; that is, making an entry today in tomorrow's diary.

A second group of patients had been assigned to use an electronic diary for the same protocol. In contrast to the poor validity and poor protocol compliance in the paper diary condition, the electronic diary yielded actual adherence of 94%. These compliance results have been replicated in chronic pain patients (Broderick et al., 2003) as well as in a separate, pharmaceutical company-sponsored study on elderly patients with chronic obstructive lung disease.

Back- and forward-filled paper diary data are a violation of the protocol and can undermine reliable testing of a new treatment's safety and efficacy.
4. For each problem identified, if a solution would facilitate the development of drugs, biologics, and/or devices for a particular disease or categories of disease, please indicate which diseases would be affected?
Patient Reported Outcomes (PRO) data are broadly defined as self-reports based on symptoms (e.g., pain, fatigue), self-observations of objective events (e.g., voiding, coughing), and quality of life reports regarding global or disease-specific well being (Hufford & Shiffman, 2002). It has been estimated that 78% of Phase I-IV trials capture PRO data, and that 25% of Phase II-IV clinical trials specifically rely on diaries to capture primary or secondary endpoint data (DataEdge, 1998-1999). Thus, PRO data constitute a sizable and critical part of clinical development and regulatory submissions.

According to a recent review of 194 approved labels between 1997 and 2001, the FDA approved approximately 31% of New Drug Applications using Patient Reported Outcome (PRO) data as part of primary or secondary claims of efficacy (Willke, Burke, & Erickson, 2002). Thus, PRO data broadly and diary data in particular are central to the development of new treatments across a wide variety of therapeutic areas, including but not limited to CNS disorders, rheumatology, oncology, anti-infectives, dermatological, pulmonary, allergy, genitourinary, analgesia, anesthetics, and endocrine disorders.
5. Nature of the Solution. For each problem identified, please describe the evaluation tool that would solve the problem and the work necessary to create and implement the tool/solution. For example, would a solution come from scientific research to
The adoption of electronic diary methods in appropriate protocols would enhance the integrity and efficiency of clinical trials. For more than 18 years, academia and the pharmaceutical industry have been using electronic diaries to study clinical endpoints based on PRO data (see Hufford & Shields 2002, for a review). A robust empirical literature has shown that properly designed electronic diaries are easy to use (e.g., Hufford et al., 2002) and preferred to their paper counterparts (Drummond et al., 1995; Rabin et al., 1996). Electronic diaries have been used to capture primary endpoint data in Phase III registration trials as part of approved new drug applications, and are currently in use in more than 50 trials involving tens of thousands of patients. Multiple vendors provide their electronic diary services to the pharmaceutical industry at the present time.
3. For each problem identified, please indicate the type of drug, biologic, or device to which the hurdle applies.
The need for reliable, valid, PRO data that meets the FDA's standards for data quality and integrity is present across drugs, biologics, and new medical devices.
7. For each problem identified, what role should FDA play and what role should be played by others?
Uncertainty about the FDA's posture towards electronic diaries is a key barrier to adoption of this enabling technology, and has encouraged sponsors to continue to use less effective methods of data collection. The FDA should implement and announce consistent, universal application of existing standards for data quality and integrity (e.g., ALCOA standards), and make clear how these apply to both paper and electronic diaries. The FDA should encourage industry to critically review, and provide supportive data and documentation for the validity, reliability, accuracy, and sensitivity of their PRO instruments, both paper- and electronic, providing a 'level playing field' for both kinds of diary methods. To facilitate adoption of the optimal methods, the FDA can expeditiously provide a clear regulatory framework for use and submission of diary data, working
with industry and other parties to establish appropriate standards and requirements for data that meet the highest standards of scientific integrity.
The use of Patient Reported Outcome (PRO) data that fail to meet the FDA's standards of data quality and integrity undermines the efficiency and reliability of clinical trials. Such data are subject to recall biases, which introduce noise and bias, and inflate error variance, potentially obscuring a reliable test of study endpoints (e.g., Bradburn et al., 1987; Stone et al., 2000). Trials using such flawed data can fail to accurately detect treatment effects, or may require larger studies to detect treatment effects. Several studies have demonstrated that use of electronic diaries reduces error variance compared to paper diary methods (Pearson, 2004). For example, a recent study by McKenzie and colleagues (2004) compared error variance in two studies that used diaries to assess the efficacy of a medication for overactive bladder. One study used paper diaries, and asked patients to record micturitions three times daily. The second study, conducted on a similar patient population, used electronic diaries and asked patients to record all micturitions at the time of occurrence (up to 50 times per day in some subjects) for a period of 14 weeks. Compared to the paper diary study, the electronic diary study reduced variability by 33%. This reduction in error variance translates into a substantial increase in the trial's statistical power (the probability of finding a drug effect, if it is actually present), from 80% (expected based on paper diary performance) to 98% (actual power based on electronic diary performance). The reduced noise and increased power translates into more efficient clinical research. The sponsor anticipates that future OAB electronic diary trials can be conducted with up to 50% fewer subjects resulting in 45% lower costs per trial. Similar results have been seen for studies of sleep quality (Pearson, 2004) and of the prophylactic treatment of migraines. Thus, it appears that using electronic diary methods to gather PRO data in real time can make clinical trials more efficient and more sensitive.

The observed power savings outlined above will vary across trials and endpoints depending, among other factors, on the vulnerability of the endpoint data to recall bias. The net effect of this reduction in error variance is that some clinical trials could be run with less subjects, resulting in fewer patients exposed to untested treatments and addressing one of the major bottlenecks in drug development timelines -- patient recruitment. Moreover, some indications, like rhinitis and major depressive disorder, that are notorious for poor outcomes resulting in many trials being run to produce positive results, could find that fewer total trials need to be run to decide whether to advance or stop a compound earlier in development.

Further, in many areas, the enhanced data quality and data integrity offered by electronic diary methods means that trials are more likely to detect the real benefits a test drug can offer. That is, broad use of these methods could enable industry and the FDA to reduce the probability that a promising candidate drug is rejected due to a false negative finding.