James Penston
About Stats.con - How we’ve been fooled by statistics-based research in medicine: Statistics-based research is the method by which the causes of disease and the effectiveness of new treatments are investigated. Epidemiological studies and large-scale randomised controlled trials dominate medical research. Judged by the number of papers published each year, this type of research would appear to be a success. Yet it’s a triumph of appearance over substance. We’ve been cajoled into believing that great advances in medicine have occurred when, in fact, this isn’t the case.Large RCTs are placed at the summit of the hierarchy of evidence and are claimed to be the most reliable means of establishing causal relationships in medical research. They are highly complex structures designed to identify small differences in outcome between the active treatment group and controls. But how do we know that the observed difference is caused by the drug? Proponents of RCTs assert that the method excludes alternative explanations – namely, the unequal distribution of other causal factors, bias in the assessment of the outcome and chance. In other words, they believe that these studies have internal validity. The primary thesis of stats.con is that the grounds for causal inference in statistics-based research are lacking. Firstly, the components of the RCT – including randomisation, allocation concealment, double-blind administration of treatment, the handling of withdrawals and drop-outs, and the statistical tests – don’t guarantee that the conditions for internal validity have been satisfied. Secondly, the frequentist approach to statistics, which continues to be used in almost all medical research studies despite being subjected to serious criticisms in recent years, is unsound. Thirdly, and most importantly, the inference from a small difference in outcome to the presence of a causal relationship is highly questionable.Given these arguments, it’s of some importance to note that neither the results of individual RCTs nor the statistical method in general can be tested independently. This is an inevitable consequence of the subject matter of this type of research which involves heterogeneous samples with unknown mixtures of constituents. The inability to test the results of statistics-based research is of particular concern as fraud is more common than hitherto supposed in medical research.But even if we were to accept the validity of causal inference in this situation and to dismiss concerns about independent testing, we would still face the unpalatable truth that the product of statistics-based research is of little value. The reliability of any generalisation from the results of an individual study to the wider population of patients – that is, the external validity – is always open to question. We can never know whether the results of a RCT apply to either a particular patient or to a specified group. This is an enormous disadvantage in medicine. But that’s not all. The size of the treatment effect in large-scale studies is very small. Indeed, it’s so small that the true size of the effect is deliberately hidden by researchers and others with a vested interest in the outcome of the studies. When we look closely, the product of these studies is of dubious worth and doubtful meaning.The reasons for the widespread acceptance of statistics-based research are to be found in the events of the past fifty years or more. History shows how the advocates have used every means at their disposal to spread a flawed methodology and how their views have infiltrated the thinking of generations of researchers, practicing physicians and others involved in the care of patients. But this doesn’t apply only to medical research. Many other academic disciplines use similar methods. If, as is argued in stats.con, the case against statistics-based research is made, then the implications extend far beyond the field of medicine.
