This guidance was written prior to the February 27, 1997 implementation of FDA's Good Guidance Practices, GGP's. It does not create or confer rights for or on any person and does not operate to bind FDA or the public. An alternative approach may be used if such approach satisfies the requirements of the applicable statute, regulations, or both. This guidance will be updated in the next revision to include the standard elements of GGP's.
May 12, 1988
This guidance document may contain references to addresses and telephone numbers that are now obsolete. The following contact information is to be used instead:
- While this guidance document represents a final document, comments and suggestions may be submitted at any time for Agency consideration to the Restorative Devices Branch, 9200 Corporate Blvd., HFZ-410, Rockville, MD 20850.
- For questions regarding the use or interpretation of this guidance, contact the Restorative Devices Branch at 301-594-1296.
- Additional copies are available from the Internet. You may also send an e-mail to CDRH-Guidance@fda.hhs.gov to receive a copy of the guidance. Please use the document number to identify the guidance number 267 you are requesting.
U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES
Food and Drug Administration
Center for Devices and Radiological Health
Rockville, MD 20850
Twinbrook Building #4
Harold Stevens, M.D., Ph.D., Chair
Wen H. Ko, Ph.D.
Howard H. Kaufman, M.D.
Mary K. Gumerlock, M.D.
Edith Irby Jones, M.D. (Consumer Representative)
Marvin L. Sussman, Ph.D. (industry Representative)
Ruth B. Loewenson, Ph.D.
Wilbert Fordyce, Ph.D.
Michael H. Weisman, M.D.
Mitchell Max, M.D., National Institutes of Health (NIH)
Lawrence Friedman, M.D. (NIH)
George C. Murray, Ph.D., Director, Division of Anesthesiology, Neurology and Radiology Devices (DANRD), Office of Device Evaluation (ODE), Center for Devices and Radiological Health (CDRH).
Carolyn Deffer, Deputy Director, DANRD, ODE, CDRH
Richard Chiacchierini, Ph.D., Director, Division of Biometric Sciences, Office of Science and Technology, CDRH
Robert F. Munzner, Ph.D., Executive Secretary, Neurological Devices Panel, DANRD, ODE, CDRH
John G. Harter, M.D., Center for Drug Evaluation and Research
John Dawson, DANRD, ODE, CDRH
Doyle Gantt, DANRD, ODE, CDRH
Julie Hollander, DANRD, ODE, CDRH
Kelvyn H. Cullimore, Dynatronics
William Beaver, M.D., Dynatronics
Kenneth Nudleman, M.D., Dynatronics
John Burkart, Ph.D., Dynatronics
Kelvyn W. Cullimore, Jr., Dynatronics
Kay Cullimore, Dynatronics
Steve Woodhead, Dynatronics
Lawrence Altshuller, M.D., Dynatronics
Edward Loeser, M.D., Dynatronics
Jeffrey Basford, M.D., Dynatronics
David Harris, Ph.D., Dynatronics
J. William Dirksen, Ph.D., Dynatronics
David Conger, M.D., Dynatronics
Jeff Booth ' Dynatronics
Duane Hall, Dynatronics
Keith Hansen, M.D., Dynatronics
Jess Stribling, Esq., King & Spalding
Lenore L. Stribling
Nancy Taylor, Senator Hatch's Office
Kent R. 14offman, Murray
Thome Gould, Murray
Glenn Rahmoeller, Biometric Research Institute, Inc.
Peggy Kearney, Biometric Research Institute, Inc.
Bruce Johnson, Nontoxic & Drug Research, Inc.
Naser Salman, BMP
T.Q. Garvey 111, M.D., Garvey Assoc. Inc.
I.M. Brandt, Lilly
Milton Allen, Temple University
G. Killeen, Arter Hadden
S.K. Davi, Physiotechnology, Inc.
Matthew Weasky, WCG
M.S. Wenley, WCG
C. Harrington, The Gray Sheet
Walter Solomon, Litechnica
Kshitij Mohan, Ph.D., ODE, CDRH
Stephen Hinckley, DANRD, ODE, CDRH
Kathleen Frost, DANRD, ODE, CDRH
Levering Keely, DANRD, ODE, CDRH
Robert Eccleston, CDRH
Richard Chamberlin, FDA
Sharon Findley, DANRD, ODE, CDRH
Marcia Withiam-Wilson, DANRD, ODE, CDRH
James Dillard, DANRD, ODE, CDRH
Louis Hlavinka, PMA, ODE, CDRH
Donald Watchko, DCO, OC, CDRH
Judith Kalson, DPS, OC, CDRH
Sheila Hemeon-Heyer, OST, CDRH
Anita Rayner, OST, CDRH
Barbara Loyd, OLA, FDA
Nirmal Mishra, DSRD, CDRH
Peggy Hamilton, OST, CDRH
Jerome Dennis, DRP, OC, CDRH
Marie Schroeder, DSRD, CDRH
Mary K. Barrick, DBS, OST, CDRH
Richard P. Felten, DLS, OST, CDRH
Carolyn Neuland, DGGD, CDRH
Michael J. Blackwell, DSRD, ODE, CDRH
A. Torres-Cabassa, DSRD, ODE, CDRH
Robert H. James, DPS, OST, CDRH
Steve Babcock, MDR, OC, CDRH
Gideon Kantor, DPS, OST, CDRH
George Koustenis, OC, CDRH
Tim Wells, Exec. Secretary's Office, FDA
Joyce Rollings, DCP, OC, CDRH
Carol Lee, DCA, OTA, CDRH
OPEN PANEL DISCUSSION
Design of Pain Studies. FDA requested advice from the panel on the development of guidance for clinical studies that are used to obtain data to assure the safety and effectiveness of pain relief devices. Dr. Max and Dr. Friedman of the NIH discussion. FDA staff presented a summary of the design which they believe represent a consensus are engaged in the design of clinical trials to measure of pain therapies. (See attachments A and B).
Protocol. The panel encouraged FDA to examine the design of each clinical study before allowing it to begin. They also suggested that FDA consider the credentials of persons who intend to undertake a clinical pain study before approving it. it was noted that pain, in general, fulfills a necessary role in protecting man from disease and injury.
Study Population. Panel members recommended that special attention be given to obtaining a homogeneous population with regard to severity of pain and etiology of disease.
Evaluation Methods. The methods of pain measurement were discussed in general terms, the panel consensus being that valid measurements are difficult to construct because pain is a subjective, emotional experience. It was noted that the distinction between acute pain and chronic pain is essential in constructing the evaluation method; i.e., it is necessary to know the natural time course of the underlying disease process, and to consider the time course in the design of the study.
Sample Size. It was suggested that there be enough subjects to assure that the probability of an error in rejecting any null hypothesis be less than 0.05.
Randomization. It was noted that it may not be possible to achieve randomization in some studies because there is no appropriate alternative therapy, e.g., the study of deep brain stimulators. Although randomization is a valuable technique to help reduce the uncertainty of subjective responses, it cannot reduce the uncertainty introduced by undetermined prior experience factors.
Placebo. The panel recommended that this topic be generalized to include the various methods of control. The panel suggested that consideration be given to the following:
(1) Positive controls can be used when a placebo in not available; however, the appropriate analysis of data obtained using a positive control is different fr6m that used for studies that employ a placebo. The use of the "Phyler's Theorem" for studies having a positive control is favored.
(2) It is necessary to distinguish between a pain therapy that depends upon an effect on the disease process and a pain therapy that has a strictly analgesic effect. The time course of these two types of therapy differ greatly; therefore, the methods used for control and analysis of the data also differ.
(3) Placebo control is essential when subjective measures are used because there is immense variation in the way persons perceive and report pains. These variations are linked to various factors such as cultural differences, prior experience and secondary gain. Use of MMPI screening can be useful for limiting. patient-variability.
Blinding. The consensus of the panel was that blinding of the patients, physicians, investigators, other persons caring for the subject, and the evaluators should be employed to the greatest extent possible; however, it was recognized that attempts can fail unexpectedly. It was also recognized that good blinding is sometimes not possible.
A closed session was initiated to discuss the trade secret data contained in the laboratory testing part of approval application file P860013; however, after a brief discussion of the power output measurement accuracy, it was determined that there was no need for discussion of any trade secret data. Thereupon, the meeting was immediately re-opened.
OPEN PANEL DISCUSSION
FDA staff summarized the premarket approval application P860013 for a low powered helium-neon laser that is intended for the relief of the pain of arthritis in finger joints and briefly reviewed the preclinical and clinical testing provided in the premarket approval application .
Laboratory Testing. Laboratory testing used to establish proper manufacturing controls was discussed briefly. One problem noted concerned a discrepancy in the stated output of the laser.
Preclinical Studies. Animal studies commissioned by the firm to support the safety of the device were briefly discussed. Although no adverse effects were reported in these studies, FDA staff noted that none of the studies provided data that would assure the long-term safety of persons being treated for the pain associated with rheumatoid arthritis. No scientific data was provided to explain the means by which the laser might be effective in treating hand pain associated with rheumatoid arthritis nor was any physiological process shown to be connected to the pain relief property of the laser. FDA staff indicated they did not believe it was necessary to explain the mechanism of action; however, the burden of providing valid scientific evidence of effectiveness must be supported entirely by the results of the clinical study conducted by the applicant.
Clinical Studies. FDA reviewers noted that several published clinical studies of laser pain therapy were included in the application; however, it was difficult to relate these studies to the applicant's device because they employed lasers of different wavelengths and different dosages. Also, some of these studies pertain to treatment of pain with an etiology other than rheumatoid arthritis, and others were not presented in sufficient detail to allow any conclusion to be drawn from them. The reviewers concluded that none of the published studies presented in the PMA could be used to establish the firm's claims of effectiveness. Data from a pilot study conducted by Dr. Judith Walker were included and the results were used to develop the main study protocol.
The main study was intended to be parallel, randomized, double-blinded. All subjects were to be diagnosed as having classical or definite rheumatoid arthritis as classified by the American Rheumatism Association (ARA). Seven of the eleven ARA criteria had to be met in order to enter the study.
The protocol required three treatments each week for three weeks administered to the metacarpal and proximal interphalangeal joints of the dominant hand. These were for 60 seconds of exposure at 0.95 mW. Using Steinbroker's functional classification, only subjects in class two or class three were to be selected. All subjects were to have active disease as measured by active synovitis, pain and tenderness in at least a few joints of the hand. AU subjects were to be maintained on stable medication doses throughout the therapy including aspirin, nonsteroids, gold, oral corticosteroids and cytotoxics.
Five investigators conducted seven trials in a total of 152 patients. Two of these investigators also studied a second series of patients. FDA reviewers expressed the following concerns about the clinical data obtained:
- There was marked variation between investigators in the results obtained.
- Blinding could easily have been compromised.
- Many subjects participating in the study may not have had rheumatoid arthritis that was active. (According to the applicant, 38 of the 152 subjects were taking no medication.)
- Concomitant medications were not well controlled as required by the protocol. More than 50 different drugs were used by the subjects and many took medications inconsistently, or took them on an "as needed" basis. No documentation was provided to show that each subject's medication had been stabilized in the six months prior to entering the study.
- Follow-up information was available for less than two-thirds of the subjects. No justification or explanation for the missing subjects was provided.
- Of the various objective measurements of outcome that were made, the applicant could conclude that only one i.e., grip strength, improved. No data was provided by the firm to support their hypothesis that a systemic effect caused the observed improvement in both. the treated and the untreated hand.
- The pain evaluation methods did not clearly define whether the finger pain or generalized pain was being evaluated.
- In analyzing pain relief data, the applicant defined improvement as at least a single category favorable change in perceived pain relief on a five category scale. If a stricter definition of improvement is used, i.e., at least a two-category improvement in perceived pain relief, no statistically significant ]5enefit is demonstrated. FDA consultants expressed concern that a single category criteria improvement would not be sufficient to establish significant benefit in these-subjects.
- The data presented by the applicant show that treatment effects occurred at the end of the three-week treatment period, but no data was provided to show that continued treatment is either safe or effective. Concerned was expressed that the data indicates three weeks are needed to produce an effect, but no data was obtained to show its duration. FDA staff also expressed concern that no data was provided to support the safety of repeated treatment.
- FDA reviewers estimated that over 250 statistical tests were performed by the applicant but relatively few tests favored laser therapy. Of the 15 evaluation criteria studied, the applicant concludes from an analysis of the results that six of these criteria were found to be statistically significant in favor of laser therapy with a confidence level based on acceptable error of p = 0.05 or less. These measures included: present pain, greatest 24 hour pain, grip strength, the investigators overall perception of treatment response, and the subject's overall perception of pain relief and change in activity. The panel's statistical reviewer expressed the opinion that a much high6r confidence level should be set when multiple tests are performed.
OPEN PUBLIC HEARING
Representatives of Dynatronics Inc. presented a brief history of the firm and outline the history of this PMA application. The manufacturer's verbally related much of the data presented in the PMA including a description of the device, of the investigation protocol, and of the clinical results.
They pointed out that labeling for this product indicates it is intended for adjunct therapy to temporary relieve joint pain in the hands of persons being treated for rheumatoid arthritis.
The firm gave a general overview of present treatments of pain which included non-steroids, drugs, antidepresents, anesthetic blocking agents, the use of heat and cold, TENS devices, and physical therapy with hot wax. Other treatment modalities such as counseling, biofeedback, hypnosis, and ablative procedures; i.e., rhizotomy and cordotomy were also mentioned . It was their opinion that the mechanism of action for these treatment modalities is no better known than it is for the laser. Suspected mechanisms of action of the laser with rheumatoid arthritis patients were discussed.
It was emphasized that this laser treatment is intended as an adjunct to other treatments, and if it is found to be beneficial with a specific patient, the frequency of the treatments might be cut back, stopped, or repeated for another course as would be done with drug intervention.
OPEN COMMITEE DISCUSSION - RECOMMENDATIONS
Panel members and consultants expressed the following opinions concerning approval of this application:
Measures used in the primary study, such as range of motion. ring size, and grip strength appear to be measures of osteoarthritis rather than measures of rheumatoid arthritis. More appropriate measures, such as pain on motion, tenderness and swelling of the finger joints, were not included. There were no measures of rheumatoid arthritis activity such as sedimentation rates or joint examinations to show that the disease was active.
Although the protocol specified that no changes in medication were to be made, the use of concomitant analgesics during the course of the study was paradoxical. The protocol was also violated by the use of drugs not approved by FDA for rheumatoid arthritis, such as methotrexate and sulfasalazine.
Clinical trials of a medical treatment for arthritis pain include experimental dose adjustment as part of the testing. A dose response relationship is needed in studying treatments that are hard to blind.
A major pitfall of this study was that patients were entered who did not have substantial manifestation of disease. The protocol required that the patients be stable on drugs for six months. It is suspected that patients meeting this criteria probably did not have active rheumatoid arthritis. Experience indicates that it is difficult to obtain compliance with rheumatoid arthritis patients who are in protocol that requires a stable dose of medication for more than one month because of fluctuations in disease activity.
An important aspect of designing any rheumatoid arthritis protocol is to establish limits on the amount of steroids to be allowed. The steroid dose must be also held constant for a given period of time. The primary study failed to do this.
The variability in the disease activity of rheumatoid arthritis patients, with activity waxing and waning, makes the duration of the observations important. The study period must be long enough to allow the fluctuations to damp out.
Concern was expressed regarding the medical credentials of some of the investigators. One investigator reported that none of his subjects were used any antirheumatic medication, although he had classified his patients as having significant functional impairment.
Treatment of only pain without concomitant relief of the other disease symptoms raises concern about the physical process that may be occurring within the joints treated by laser because denervated joints are known to be at great risk. Studies of the long range effect on the joints is needed.
The use of a pilot study in which different patients were treated in different joints without standardization made it impossible to interpret the data.
The envelope system of randomization is subject to abuse. It was suggested that other methods of randomization be used, such as central on-fine assignment by code.
The panel unanimous voted to recommend that the PMA for the laser pain therapy device not be approved because it does not provide sufficient data to reasonably assure the safety and effectiveness of this device for the intended patient population.
Recommendations to Applicant for Future Studies to obtain PMA Approval
In addition to correcting the problems mentioned above, the panel recommended that any future studies include the following:
- Multiple baseline evaluations should be used to evaluate the variability among patients without any treatment.
- Each subject in the study should maintain a daily diary to record pain, stiffness and medications.
- The goal of the study (i.e. pain treatment or rheumatoid arthritis treatment) must be well defined before attempting to design the study. The outcome measurements must be clearly specified.
- Any adjunctive therapy that is to be allowed must be clearly defined.
- Documentation of the state of the disease and the state of the pain need to be established independently so that they may be dissociated. (This may require pilot studies.)
- Dose response curves should be obtained to establish maximum effect.
- Consultation and review by rheumatologists, pharmacologists and statisticians is needed in planning.
- Establishment of an independent external peer review committee is needed to objectively evaluate the data.
- Persons known for having been successful in gaining peer review grants should be identified and involved throughout the studies. These consultants include persons with a background in the specific disease process and patient population.
- In studying a laser to be used for the relief of pain associated with rheumatoid arthritis, the investigator should be familiar with the treatment of rheumatoid arthritis using drugs, physical therapy and other modalities.
- The firm might consider the study of another type of disease process having associated pain to make it possible to separate the disease effect from the pain effect.
- Every effort must be made to assure blinding.
- The study should be long to enough to show an increasing effect.
- Additional functional measurements are needed.
I approve the minutes of the meeting as recorded in this summary.
Harold Stevens, Ph.D., M.D., (date)
I certify that I attended this meeting of the Neurological Devices Panel on May 12, 1988, and that these minutes accurately reflect what transpired.
Robert F. Munzner, Ph.D. (date)
Neurological Devices Panel
GENERAL CONSIDERATIONS IN THE DESIGN OF CLINICAL STUDIES
FOR PAIN-ALLEVIATING DEVICES
PROTOCOL. A protocol for a study should be developed before the study commences and should remain essentially unchanged throughout the study. The protocol should clearly state and define the overall objective of the study as well as the primary question and any subsidiary questions involved in the study, The design of the study should clearly describe the study population, the procedures for enrollment of subjects, treatment intervention, schedule for follow-up visits, and methods used for the ascertainment of response variables. The organization of the study should be stated, including the anticipated number of sites and methods used for data monitoring.
STUDY POPULATION. The study population should be defined in advance and provide clearly defined inclusion and exclusion criteria so that an appropriate homogeneous population can be obtained. This is particularly important to demonstrate that a medical device is effective in the treatment of a specific disease or disorder. The study population should be selected so that their disorder is of a sufficient level of severity that their exists a reasonable chance of demonstrating improvement either with the standard treatment or the study device.
EVALUATION METHODS. The evaluation methods used for the monitoring of the patient's pain must be validated for the particular patient population under study. The patient evaluation protocol and the data collection procedures should be described in detail.
SAMPLE SIZE. it is important to determine early in the planning stage of the study the size of the study. This process is not an isolated event but is dependent on the basic study design. Factors such as the types of measurements used to determine outcome results and their clinical significance must be incorporated into the process of determining the study size. The investigator should identify what is the smallest difference that is of such clinical value that it would be very undesirable to fail to detect it. Therefore, the protocol should include the calculation of a sample size sufficient to provide for adequate levels of significance and power for the proper analysis of the data.
RANDOMIZATION. Patients should be randomized during allocation to treatments. Methods used for the randomization process are important in terms of validation of the results. Randomization tends to ensure comparability between groups with respect to known and unknown risk factors, remove investigator bias in the allocation of subjects, and guarantee that statistical tests will have valid significance.
PLACEBOS. Where a placebo can be ethically used in a study, its use should be incorporated. It is important that the patient be told that there exists a possibility that he or she may receive a placebo. The use of a placebo is useful in cases where the objective is to see whether a new intervention plus standard care is better or worse than a placebo plus standard care. Where a placebo group is to be used, it should be reasonably demonstrated that the persons in this group are unaware of which treatment they are receiving. Investigators should be aware that placebo therapy has at times been as effective as standard therapy.
CONCOMITANT THERAPY. Consideration must be given early on in the study design to the effects of patients on concomitant treatment. Every effort must be made not to introduce bias into the study results due tot concomitant treatment. Concomitant treatment may take many forms and is not necessarily confined to concomitant drug therapy. For example, various forms of physical therapy prior to patient evaluation may affect patient results. Concomitant treatment may influence the response variable. If only a portion of the treatment population is on concomitant treatment, a larger sample size may be required. When concomitant treatment is an issue in a study, those patients receiving concomitant treatment must be randomized between study groups.
BLINDING. The protocol should include adequate provision for assuring that bias is not introduced into the study results. During the conduct of a study, many opportunities arise for systematic error to be either knowingly or unknowingly introduced into the data. This error will result in a difference between the actual true value and the value obtained from the collected data. This type of error, which is generally introduced by means other than sampling variability, must be controlled to the extent possible. Therefore, it is recommended that blinding be used wherever possible in the study design. The blinding method (if possible) must be carefully planned and defined to assure that blinding is maintained.
FOLLOW-UP. After patients are taken off of treatment, an adequate follow-up period should be incorporated into the design of the study. Many aspects of the basic study design will dictate the appropriateness of the follow-up period.
MULTIPLE SITE TRIALS. In general, multiple site studies are desirable. For studies of devices for which no physiological basis has been established for the mechanism by which the device causes an effect, and, therefore, the clinical trial provide the sole basis for evaluating the safety and effectiveness of the treatment, independent studies must be performed to confirm the reported results.
SUBGROUP ANALYSIS is an important aspect of many studies. When subgroup analysis is to be done, the investigator should determine the statistical techniques to be used and the subgroups to be analyzed prior to the start of the study in order to avoid possible introduction of bias in the later analysis of the study results. The investigator may be tempted to subgroup the patient populations with regard to treatment results based on post hoc analysis of the data. This type of analysis may be suggested by the data themselves. As a result, many comparisons are theoretically possible; however, the tests for significance of the resulting data or data dredging as it is sometimes called is best left to the realm of generating hypotheses for future studies.
- Friedman, L.M., Furberg, C.D., and DeMets, D.L.: Fundamentals of Clinical Trials. 2nd Ed., PSG Publishing Company, Inc., Littleton, Massachusetts, 1985.
- Pocock, S.J.: Clinical Trials. John Wiley & Sons, New York, 1983.
- Melzack, R.: Pain Measurement and Assessment. Raven Press, New York, 1983.
CONSIDERATIONS IN THE DESIGN OF CLINICAL STUDIES
FOR PAIN-ALLEVIATING DEVICES
POINTS FOR DISCUSSION
- Despite its clinical significance, pain has proved difficult to measure. It has, therefore remained only an approximate index of the severity of certain types of pathology and an imprecise indicator of the effectiveness of treatment. Numerous methods have been developed in an attempt to arrive at some quantitation of this important symptom, some of which rely upon directly observable conditions associated with pain (i.e., objective measurements which may be associated with the underlying pathology), others which attempt to measure the patient's perception of the pain itself such as visual analog scales, numerical rating scales, box scales, verbal rating scales, and the more comprehensive McGill Pain Questionnaire.
(a) What, in your opinion, are good measurements of pain?
(b) Under what circumstances should multiple assessments be used?
(c) Under what circumstances can objective measures be used to measure pain?
- Many authors have discussed specific potential sources of bias encountered in studies where outcome must be measured subjectively, e.g., placebo effect, patient/observer bias, and variability between and among patients.
(a) Under what circumstances should placebos be used?
(b) Under what circumstances should pain studies be double- or triple-blinded?
(c) Under what circumstances should a cross-over study be used'?
(d) Under what circumstances should repetitive measurements be made?
(e) When should pain measurements be made, e.g., upon entry into the study, immediately before and after treatment, between treatments?
(f) Under what circumstances can the results of studies which are not blinded be used to demonstrate alleviation of pain?
(g) Under what circumstances can the results of studies which do not include randomized, concurrent controls be used to demonstrate alleviation of pain?
- What criteria should be established to determine clinical benefit?
- Are there any other special techniques which need to be considered in the study of pain, e.g., in the selection of subjects or in the control of concomitant therapies?
- At least one author has reported that patients with pain in anatomically differing sites tend to more extensively describe their pain than patients with pain in a single body region or part. They, also state that the data suggest that the over-inclusive verbal reporting style of these patients is consistent across classes of descriptors and that sensory descriptors may also be confounded.
(a) In your experience, is this a phenomenon which should be taken into consideration in designing and conducting pain studies?
(b) If so, what steps should an investigator take to assure the reliability of subjective assessments of pain in such patients?
- Under what circumstances can you use a demonstration of alleviation of pain in one anatomical site to predict alleviation in another site? i.e., given the demonstration of a particular localized effect in one site, what information would be needed to provide reasonable assurance that the same effect is likely to occur as a result of similar treatment of another part of the body?
- What criteria should be established to determine reproducibility of results by independent investigators? What should be done when multiple investigators obtain conflicting results?