A Critique of RCTs: Historical Failure to Strike Gold

Source: Rounding the Earth Author: Mathew Crawford

“History is the best teacher, who has the worst students.” -Indira Gandhi

In our last article, we demonstrated RCTs are neither the highest form of scientific experimentation, nor are guaranteed to achieve their goal of proving superior testing. In this article, we examine the empirical history of RCTs, and compare the data from RCTs with “lesser” forms of experimental data.

In a paper published in the Annals of Medicine in November, 2017, Alexander Krauss analyzed the 10 most commonly cited RCTs in medical literature–each with thousands of citations and substantial impacts on public health policy–and found them replete with unexamined limitations, biases, and failures of randomization.

He concluded that focusing on RCTs as the “only reliable research design” would lead to “neglect” of other issues “that are studied using other methods (e.g., longitudinal observational studies or institutional analysis).” (Emphasis ours.)

In an NEJM publication authored by epidemiologist John Concato and colleagues in 2000, the authors found that of 99 reports on five clinical topics, “the average results of the observational studies were remarkably similar to those of the randomized, controlled trials.”

In fact, the authors noted that results from observational studies on vaccines resulted in tighter “point estimates” (confidence intervals), a measured probability range that helps researchers understand the bounds of an experimental result in a random model. This has at least something to do with the greater cost (in direct economic terms) of obtaining each data point collected in an RCT. And more data points mean greater certainty (tighter confidence intervals).

Even more to the point, aggregating vast amounts of data from numerous observational studies performed on different populations can “balance” both known and unknown confounders far better than can an RCT with a fraction of the number of participants. Remember our practical credo on data:

Regardless of all else, seek ways to make the greatest net value out of all data.

Oftentimes, it is easier to reach confidence with many good, but imperfect data points than it is to reach confidence with few nearly perfect data points. And while the historical data bears that out, we should fully expect it!

Plenty of RCTs have failed in their goal, whereas much great science has been done without them.

This disparity may relate to experimental design, but it might also relate to the analyst. It is the job of the analyst to make the best use of the available data. A superior statistician with an inferior data set is nearly always preferable to an inferior statistician wielding the most pristine data possible (and no respectable statistician would argue that point). Only a foolish statistician would start work with less than a full data set, from all experiments, given the option.

The skeptic: So why then do all the experts say they don’t trust results without an RCT?

They don’t. No matter how much writers in the media tell you what Dr. Fauci thinks, and vaguely refer to his opinions as those of “all” or “the majority of” scientists or statisticians, the only field of science in which people (foolishly) talk that way is in pharmaceutical medicine.

This claim is not backed by history or reality, but by money. Once upon a time statisticians objected to the claim that RCTs were necessary to prove that smoking causes lung cancer. We should have taken that moment to forever put the myth of the need for the “gold standard” RCT to rest. 

Psychiatrist Norman Doidge refers to this attitude that rejects all other forms of evidence as “RCT fundamentalism”. In his article, “Medicine’s Fundamentalists”, Doidge noted that RCTs can confound by design. The criteria by which some patients are excluded from a study may change results in unexpected ways.

Doidge observed that 71.2% of RCTs were found not to be representative of real world patient circumstances, and that 35% of their findings were not replicated when tested a second time. Larger data sets help with this problem.

So, just how did RCT fundamentalism take root?

The very first RCT took place only in 1948 after British statistician Ronald Fisher spent more than two decades paving the way with a statistical framework for measuring and interpreting outcomes while testing hypotheses through observed experimental test results. Gradually, more and more RCTs took off over the years. In preparation for this article, I conversed about the history of RCTs with medicine manufacturer Monica Hughes who noted,

RCTs didn’t exist much before the 1960s Kefauver Harris Amendment to the FDA. It’s not how the vast majority of medical innovation was ever done, and the FDA itself only existed until 1962 to assess safety, not efficacy.

Since that time, some medical treatments, most often in the realm of pharmaceutical therapy, have been accepted into practice because of RCT results. And also many without them. Although Dr. Raoult’s twelve [disease therapies developed] stands out as a colossal achievement, never in his career did he run an RCT. His many successful treatments achieved widespread acceptance without them.

RCTs took root in pharmaceutical and vaccine testing over the past few decades. They were the best way to sell both consumers and regulatory agencies on new medicines. That probably has something to do with the long history of phony cures peddled by pharmaceutical salesmen over the years. But such history is often quickly forgotten, and slogans become psychologically self-reinforcing until they become the new official myth.

Gradually, the methodology became more widely applied, successfully or not, until those who believed in its superiority saw it as the bright line standard by which data are accepted or rejected in analysis (where convenient for the bottom line).

We might ask ourselves: Were RCTs conducted to determine whether lockdowns or masks inhibited the spread or severity of SARS-CoV-2 before scores of nations made those decisions? No.

Ultimately, nearly all practical decisions are made on the best data available, and quite often that data speaks loudly (more so for HCQ than for either masks and lockdowns). Judgment and decision making have always lived, and will likely always live, in a zone somewhere between art and science. Strangely, the RCT fundamentalists never spoke up on these topics. It is likely that political heuristics take precedence in most minds at such moments.

Nobody suggests running RCTs to determine the efficacy of schools, the safety of automotive vehicles, or whether or not higher rates of death among methamphetamine users is due to the drug or some other correlating factor. Nobody decides to teach their children how to safely cross the street by running RCTs. As Doidge points out, it was observational studies that taught us that Sudden Infant Death Syndrome occurred more often in babies put to sleep on their tummies instead of their backs, and that observation saved lives. Common sense dictates that we make the best forms of data and information available as valuable as we can within the best economic framework we can muster.

Look outside of the medical industry, and what do we see? The primary reason for the Information Age is that technology now allows us to make use of more and more data—even mundane data. Our most successful industries—technology and finance—have no such heuristic for rejecting data not passed through the rubric of an RCT. Where economical, these industries perform AB testing and other randomizing sieves. And so too can an adequate statistician perform techniques to achieve better randomization without an RCT. Propensity score matching is one such example.

Why then does the pharmaceutical industry and its promoters insist on conducting RCTs before something is accepted as true? RCTs take substantial amounts of time and significant resources to conduct. This creates a barrier to entryespecially for inexpensive solutions to medical problems. In other words, the myth that RCTs are some necessary “gold standard” is a deception that, along with regulatory agency, prevent any possibility for simpler and less expensive (less profitable) medical solutions to gain traction

Used judiciously, RCTs can certainly have their place in productive science. They may provide value aside from slowing down competing solutions, such as when there is a desire to identify subgroups sensitive to a new medication. But the idea that RCTs uniformly provide substantial value to the results of an experiment, much less fulfills a necessary criterion, is an engineered mythical axiom that runs counter to the general scientific experience. Rejecting RCT fundamentalism should be easy for the practical thinker.




Journal of Medicine Says HCQ + Zinc Reduces COVID Deaths

HCQ Peer-reviewed: Pathophysiological Basis and Rationale for Early Outpatient Treatment of SARS-CoV-2 (COVID-19) Infection

The Chloroquine Wars: Part I

The Chloroquine Wars Part II

The Chloroquine Wars Part III

The Chloroquine Wars Part V – A Closer Look at RCTs Studying Hydroxychloroquine Efficacy

The Chloroquine Wars Part VI – The Simple Logic of the Hydroxychloroquine Hypothesis

The Chloroquine Wars, Part VII – Why Did Dr. Anthony Fauci Leave Hydroxychloroquine Off the Early Pandemic Research Priority List?

The Chloroquine Wars Part VIII – Hydroxychloroquine’s Safety Profile and a Cost-Benefit Analysis

The Chloroquine Wars Part IX – How Research Can be Rigged by Statistically Stacking the Deck (A Simpson’s Paradox Tale)

The Chloroquine Wars Part X – A Discussion of the Insanity of the Chloroquine Wars

The Chloroquine Wars Part XI – See No Good, Hear No Good, Speak No Good

The Chloroquine Wars Part XII – Manufactured Fear During Hydroxychloroquine’s Trump Moment

The Chloroquine Wars Part XIII -A Clockwork Orange Man

Share on facebook
Share on twitter
Share on whatsapp
On Trend

Latest Stories

A Conversation With Dr. Brian Tyson – The Chloroquine Wars Part XLI

The data from the vaccine trials must be recalculated under the lens that some of the serious adverse events (SAEs) should be treated as COVID-19 cases.
There may not be any vaccine efficacy in the data at all if some of the post-vaccination deaths are due to vaccine-induced COVID-19. It could even be the case that the mortality efficacy goes negative. My implied lives saved calculations put this easily in the realm of possibility. The only way to know is for authorities to organize and compile the data. And while a risk analysis is certainly due after more than six months of hundreds of millions of doses delivered, there is little indication authorities have bothered with the process. That’s more than a bit unsettling.

Read More »

Dr. Brian Tyson, USA: Hydroxychloroquine – Data, Strategies and Success Treating over 6000 Covid Patients

We continue to delve extensively into the COVID-19 data with esteemed statistician, Mathew Crawford. In this episode, we take a closer look at the use of early treatment and its success. Mathew has been researching the data from Dr. Brian Tyson, who has successfully treated over 6000 patients who contracted COVID throughout the pandemic. It is my great honor to have Dr. Tyson here on this episode to not only discuss the data, but also the strategies that he has been using throughout the pandemic.

Read More »