How You Can Prevent Poor Data
The FDA is focused on improving the conduct and quality of clinical trials, and here is how you can help.
The views and opinions presented in this article are those of the authors and do not necessarily reflect those of the US FDA, the US Department of Health and Human Services, or the Public Health Service.
The importance of a thoughtfully crafted and thoroughly defined protocol is paramount to the reviewers at the FDA. In many ways, a clinical protocol is a roadmap that outlines the depth and quality of data that are expected to be generated from a clinical investigation. Resources are allocated within the FDA in a manner that ensures adequate input on clinical studies that are FDA-regulated. The FDA draws expertise from a variety of specialists depending on the nature of the drug, product, or device, and may include statisticians, epidemiologists, physicians, engineers, pharmacists, veterinarians, and pathologists—to name a few.
The back-and-forth discussions between these FDA reviewers and the sponsors of clinical studies can sometimes be intense and challenging. However, there is little doubt within the FDA that the rigor and time put into the review of these protocols is not spent in vain. In fact, it is any reviewer’s hope that a well-written, scrupulously defined, and strictly obeyed protocol can serve only to improve the quality of the data generated by a study. In its essence, a well-designed protocol ensures that the subjects of the trial, who volunteer their bodies (and possibly their lives) for medical research, have not placed themselves at risk for naught. Even if the data ultimately demonstrate that the treatment they received was ineffective, at least their altruism led to data that can be used to further medical science.
An important factor for any review of clinical results is the robustness of the data. What a reviewer ultimately wants to know, before making any decision about the reported outcomes and conclusions, is the level of adherence to the protocol. To put it another way: How robust are the data in support of the actual outcome?
One way that the FDA encourages the generation of quality data by investigators is by making Good Clinical Practice guidelines a ubiquitous part of every FDA-regulated clinical trial. In fact, the FDA defines Good Clinical Practice as, “a standard for the design, conduct, performance, monitoring, auditing, recording, analysis, and reporting of clinical trials.” More information on Good Clinical Practice guidelines that should be read by all clinical investigators can be found at www.fda.gov/oc/gcp/default.htm.
However, to advance the discussion of the means by which a reviewer determines how well the data support an outcome (or endpoint), some reflection on examples by which data can be “corrupted” is in order. Hopefully, the following discussion will highlight how important the investigator’s role is in ensuring the quality of data that they and their subjects generate.
Mistake #1: Bias Was Not Controlled
The presence of bias in a clinical study can be a big signal that the data are of suboptimal quality. The word bias generally refers to the influence of an untoward variable on the outcome of a study. There are many ways that bias can enter a study, and sources of bias are extensively discussed throughout the published literature. Some of the main sources of bias for trials that seek a regulatory objective include selection bias, investigator (or referral) bias, and measurement or information bias. Selection bias can occur whenever subjects of distinct demographic or clinical subgroups are distributed in a nonrandom fashion between study arms. Another form of selection bias can occur whenever the distribution of patients, who themselves are subject to regional standards of care, are distributed unevenly across study arms. Information (or measurement) bias occurs during the evaluation of an effect, in which errors in measurement or the assessment scale itself influences the outcome. Unblinding, discussed further below, can lead to a variety of biases, including information bias.
An example by which investigator bias may enter into a study could occur if the protocol leaves the use of adjunctive therapies “to the discretion of the investigator.” The possibility of an investigator introducing his or her bias into a study can occur without intention. Randomization can control investigator bias to some degree in large clinical studies, but randomization is ultimately based in chance. Most opportunities for bias should not be left to randomization alone and should be prospectively controlled. A protocol will need to ensure that any adjunctive therapy cannot introduce bias into the study. The FDA spends a fair amount of the review process in determining both the size (how significantly the bias may affect the scientific soundness of a study) and the direction (which outcome it favors) of any potential bias.
Mistake #2: Data Simply Disappeared
As a general rule, regulatory agencies do not like things to just disappear—whether it’s plutonium or clinical data—it should always be handled with care and be accounted for. However, almost inevitably, data go missing for a variety of reasons. Usually, it is because subjects have been lost to adequate follow-up in the course of a study. Who is at fault for this only becomes a concern if foul play is suspected, which is rare. However, any protocol should anticipate and clearly specify how missing data will be managed. Every investigator should carefully document attempts to maintain protocol-mandated follow-up visits. All subjects are sources of precious data and the whereabouts of these data need to be every investigator’s priority. If you do happen to lose the data, you must make an effort to find it. You must also have an idea of where it went and expect to explain how and why it was lost in the first place.
The FDA tends to cast a jaundiced eye toward missing data, but fortunately, we are open to ways by which it can be accounted for. One method is to perform a type of worst-case scenario in which the lost subjects are assumed to either have had or not had a prespecified outcome. These assumptions can then be tested against the known data to see if the ultimate outcome of the study would have been affected if the data had not been lost. Additionally, intention-to-treat and per-protocol analyses can be used together to determine the effect lost-to-follow-up subjects, withdrawn subjects, or dropouts can have on the data. It is also helpful to simplify the Case Report Forms used to collect the data, because excessive, superfluous variables may increase the chances of missing data.
Mistake #3: Protocol Deviations
Protocol deviations occur whenever protocol mandated treatments or evaluations either do not occur or occur at a time outside of the protocol-specified window. Although some level of deviation is expected due to real-world contingencies, frequent protocol deviations can occasionally be attributed to sloppy clinical oversight.
Protocol deviations usually raise more doubts about the quality of the research than about the scientific soundness of the study. However, there are times when protocol deviations can add up and affect the outcome of the study. One example is hospital lab draws. When the protocol says that labs will be drawn every 8 hours, such as CKMB for cardiovascular studies, how often are those labs really drawn at exactly every 8 hours? Some hospitals simply will not draw CKMB if the CK is normal, which becomes a protocol deviation if not taken into account before the trial began. This may become a problem if the endpoint is defined by those lab draws, such as non–Q-wave MIs defined by peak CK and/or CKMB levels. It is best to anticipate these possibilities beforehand rather than to revise the protocol and explain what happened to the study sponsor, FDA, or both.
Mistake #4: Blinding was Broken
Studies are blinded to ensure that investigators (and analysts) do not introduce bias into their evaluations with respect to treatment assignment. Unblinding may occur accidentally. One must consider the multiple parties involved in many clinical studies and the information that is passed between them to ensure ongoing safety of the subjects and the quality of the data. This includes communication between groups such as investigators, medical monitors, the sponsor, the Clinical Events Committee, IRBs, and Data Safety Monitoring Boards, all of whom have different roles and can be exposed to different levels of data analyses. Unfortunately, there is little one can do in a retrospective fashion to deal with unblinded data, and the remedy is sometimes repeating the trial.
For most device trials, in contrast to drug trials, it may be impossible to maintain double-blinding, in which both the subject and investigator are unaware of the actual treatment assignment. An example is radiopaque devices that can be detected by the investigator on radiography or angiography. The potential for the investigator to give a biased evaluation due to unblinding can be controlled by employing core laboratories that review the angiograms under strict, predefined criteria, away from the study site. Similarly, clinical endpoint committees, used commonly in cardiovascular studies, can be tasked to make separate, independent assessments of clinical data in a process referred to as adjudication.
Regardless of the depth of oversight, if an investigator feels that she has been unblinded, she should make this known to a study coordinator prior to submitting a potentially biased evaluation. Not taking this action would be a disservice to both the sponsor of the study and the subjects.
In conclusion, the collection of good data begins with the establishment of a well-written protocol, including precise definitions of eligibility criteria, adverse events, study success, treatment plans, and follow-up requirements. Adherence to the prospective plan is equally critical in ensuring appropriate outcome analyses and study conclusions. Although this article presents these issues in the context of FDA review, readers should be mindful of these potential data corruptions when evaluating literature intended to compare treatment options.
Robert P. Fiorentino, MD, MPH, is a Medical Officer with the FDA’s Center for Devices & Radiological Health. He may be reached at (301) 443-8262, ext. 161; firstname.lastname@example.org.
Takahiro Uchida, MD, MSc, is a Clinical Reviewer with the FDA’s Center for Devices & Radiological Health. He may be reached at (301) 443-8609, ext. 169; email@example.com.