From: Bob Gray [gray@jimmy.harvard.edu] Sent: Wednesday, February 13, 2002 1:10 PM To: fdadockets@oc.fda.gov Subject: comments for Docket No. 01D-0489 I am submitting the following comments on Docket No. 01D-0489, Draft "Guidance for Clinical Trial Sponsors on the Establishment and Operation of Clinical Trial Data Monitoring Committees". Separately, the Group Statisticians from the NCI-funded Cooperative Groups have submitted concerns about a number of points in the draft guidance. I fully endorse those comments. Here I wish to make additional points regarding the need for interim analyses to be performed by independent statisticians, and on the role of the DMC in protocol amendments. A basic premise in the draft guidance is that a study becomes uninterpretable if any changes to the design of a study are made by individuals with any knowledge of the interim data. This is the primary reason given for requiring that an independent statistician prepare and present the interim analyses, while not allowing any members of a steering committee (or equivalent group), including the study statistician, to see interim results. This also underlies the suggestion that DMCs not be allowed to propose modifications to study designs other than those driven by safety considerations (Section 4.4.1.4). If the study design turns out to be inappropriate for unanticipated reasons, a complete separation between those with knowledge of the interim data and those responsible for design and modification of studies could prevent necessary modifications from being made, and ultimately could result in the study being meaningless. As long as the reasons for changes are clearly documented and the modifications are reviewed and approved by an independent body, then studies often can be modified in ways that preserve the overall type I error rates. There is statistical methodology available to address some specific types of modifications, and more could be developed. For example, Rebecca Betensky (Biometrics, 1998, vol 54, p. 1061-1071) has shown how group sequential boundaries can be adjusted to preserve the overall type I error rate if one decides to perform more frequent interim analyses when interim results are close to a boundary. There has also been a considerable body of statistical methodology developed for sample size re-estimation when assumptions made in the original design turn out to be false. Even when such methodology is not available, simulations could often be performed to give some guidance on the impact of interim based redesigns on the operating characteristics of the test procedures. It is simply not true that all such changes render studies uninterpretable. It also seems inconsistent to propose that the DMC be allowed violate the stopping criteria in the protocol monitoring plan (Section 4.3.2), based on full knowledge of the interim data and using whatever arbitrary criteria they think appropriate, but is not allowed to suggest formally modifying the design on the basis of interim efficacy data. The former would seem to have substantially greater risk of affecting the interpretability of the results. However, the draft is correct that such flexibility in the DMC monitoring is needed. The protocol design team is not omniscient, and it is virtually impossible to anticipate all the combinations of events that can and do occur during the course of clinical trials. The DMC has to be able to use their judgment on whether the protocol criteria are still appropriate when unanticipated events occur. Ideally the DMC would use the protocol criteria as a guide on the intent of the monitoring plan to adapt the criteria to unanticipated circumstances. The DMC is also not omniscient, though, and there are situations where they should refer such issues back to the study team for a formal redesign of the study. Only the study design team has the ultimate perspective on their intent in the original design. As discussed above, such redesigns should not render studies uninterpretable, as long as the information leading to the recommendations for modifications and the process of revising the study is fully documented. It certainly is possible for deliberate manipulation of a study design based on interim data to bias the results and make them uninterpretable, especially if the reasons for the changes are falsely represented. Thus there needs to be some process to ensure design changes are appropriate and that the reasons and interim data leading to modifications are correctly represented. However, the proposed remedy of complete separation between those involved in the design and those in the interim monitoring goes too far. This is especially true in government sponsored studies, where the limited resources available would make it very difficult to implement the recommendations in the draft guidance. While it might be thought the FDA guidance would have little impact on such studies, in fact many government sponsored studies now have industry involvement, and could potentially be used in regulatory submissions. Even in studies conducted directly by pharmaceutical companies, though, there are other ways to provide adequate safeguards to protect the integrity of the study than the complete separation advocated in the draft guidance. The studies I am most familiar with are those conducted by the NCI-funded cancer cooperative groups (I head the statistical center for the Eastern Cooperative Oncology Group). For these studies, the study statistician generally conducts the interim analysis, but other members of the study team (or steering committee) do not have access to the interim data (except toxicity). The interim data, the minutes of the DMC meetings, and the DMC recommendations all become part of the public record of the study following its completion. Changes to the study can be recommended either by the study team (including the statistician) or by the DMC. The changes would be written by the study team, but must also be approved by the DMC and by the program staff at the NCI's Cancer Therapy Evaluation Program. This process is sufficiently well documented with sufficient checks in place that there should be little risk of a study becoming totally uninterpretable from changes made on the basis of interim data. It therefore seems unnecessary to require that interim analyses be conducted by independent statisticians in such settings, or to restrict the DMC's ability to suggest or review such changes. While it may be reasonable to suggest complete separation between the design and monitoring teams as a possible solution, the guidance should acknowledge that there are other ways to address these concerns, such as the system used by the cancer cooperative groups. Another reason given in the draft for advocating that interim analyses be performed by independent statisticians is that a study statistician with knowledge of interim results might inadvertently reveal those results to other members of the study team, and potentially harm the study. In our experience in the cooperative group setting, the risk of this is small. Any statistician in my group who violated the confidentiality of interim results would be removed from a study, whether the violation was intentional or not. In roughly 10 years of monitoring studies in our system, there have been no problems. Again the guidance proposes a complex and expensive remedy for something that has not been a significant problem. Robert Gray, Ph.D. Group Statisitician Eastern Cooperative Oncology Group Senior Lecturer Dana-Farber Cancer Institute and Harvard School of Public Health Mailing Address: Department of Biostatistical Science Dana-Farber Cancer Institute 44 Binney St. Boston, MA 02115 email: gray@jimmy.harvard.edu tel: (617) 632-2446 fax: (617) 632-2444