skip to Main Content

Guest Column: Unreliable science (or scientists)?

Unreliable science (or scientists)?

It’s been a couple of years since we were confronted with one of the most impressive cases of scientific fraud in science: the Stapel-case. We recently read a book on the affair called ‘The Publication Factory’, by Ruud Abma, connected to the faculty of social sciences at Utrecht University. Although we’ve had some exposure to data and knowledge in this particular field, disbelief at the enormous scale at which Stapel committed his fraudulent behaviour struck us. Furthermore, the conclusions of the Lefelt-commision, who thoroughly investigated the Stapel-case gave more reason for doubt. They described in their report ‘a research culture that was focused too much on confirming own thoughts (‘confirmation bias’) with questionable, selective and non-critical data handling’.

It’s been a couple of years since we were confronted with one of the most impressive cases of scientific fraud in science: the Stapel-case. We recently read a book on the affair called ‘The Publication Factory’, by Ruud Abma, connected to the faculty of social sciences at Utrecht University. Although we’ve had some exposure to data and knowledge in this particular field, disbelief at the enormous scale at which Stapel committed his fraudulent behaviour struck us. Furthermore, the conclusions of the Lefelt-commision, who thoroughly investigated the Stapel-case gave more reason for doubt. They described in their report ‘a research culture that was focused too much on confirming own thoughts (‘confirmation bias’) with questionable, selective and non-critical data handling’.

Fraud is arguably the most ‘sexy’ aspect of scientific reliability, but it is probably not the most important one, at least not in quantitative terms. Understanding the reliability of science requires knowledge and understanding at different levels.

The ‘bottom level’, if you will, is data analysis, in particular the interpretation of statistical significance. Many still believe that the p-value is a parameters of the ‘truthness’ of a particular outcome, but this assumption is wrong. In reality, ‘truthness’ is much more a function of prior probability of a particular outcome than of statistical significance testing of the same outcome. If you address a hypothesis that is unlikely to be true from the beginning, it is probably still untrue after obtaining a statistically significant result claiming the opposite. (Nature, February 2014)

A second level of understanding why scientific results may be untrue is bias. If statistical tests inform you about the data themselves, bias informs you on how these data got into the study in the first place, for example which relevant selection processes preceded data collection or analysis reporting. There is a lot of literature on bias, and we feel no urge to educate you in this. It is noteworthy, however, that we may still not fully grasp all existing sources of bias, as previously unrecognised sources of important bias still emerge in modern epidemiological science (e.g. Index Event Bias, JAMA 2011).

Adding up the impact of low prior probabilities and bias, the American epidemiologist John Ioannidis concluded nearly 10 years ago that ‘the majority of (statistically significant!) research findings are false’. Sobering, isn’t it? Ioannidis’ conclusions, however, did not even take into account the potential impact of outright fraud, such as Stapel committed. Hence, the magnitude of fraud in medical sciences is relevant to estimate its impact on the reliability on top of error and bias. How much worse does it get?

In a meta-analysis by Danielle Fanelli, roughly 2% of scientists admitted to having committed fraud themselves, but 14% testified that they had observed fraud being committed by close colleagues. In a recent study performed by us in Belgium, these numbers were even higher.
This is evidence to suggest that medical science is worse than many other scientific fields, but strong evidence is lacking. ‘We’ do, however, have in recent years an exceptionally large cohort of infamous frauds amongst us. Also, biomedical research has a poor reputation in terms of replication of positive study outcomes, and evidence of publication bias (preferential publication of positive study outcomes) is quite strong. Having said that, the discussion should not be whether one field is more fraudulent than the other, but we should focus on why this apparently occurs, and what we can do about it.

A few years ago, we performed an online survey among all medical professors in The Netherlands. A prominent result was that many of them believed that the pressure to publish more and more papers had become excessive, and was responsible for a loss of quality and reliability of medical science as a whole. Many testified that quantitative parameters of output and citation impact have compromised attention for quality and originality of research. In addition, many felt that other than scientific skills, such as good tutoring and patient care, were undervalued in terms of generating career possibilities.

This all sounds pretty depressing, doesn’t it? But there are some reasons for optimism. Attention for this subject is rapidly increasing, both on a national as well as an international level. In academic medical centers, national academic platforms, the Netherland Organization for Medical Research ZonMw and at the Royal Dutch Science Academia KNAW, committees, task forces and scientists themselves are increasingly active in addressing/questionning the reliability of science and scientists. For example, urgent calls to get rid of the Journal Impact Factor and related perverse stimuli are being heard in the international science community (ie the DORA declaration, see http://www.ascb.org/dora-old/files/SFDeclarationFINAL.pdf).
The public is also increasingly worried and involved. After all, it is their and their children’s health that may be at stake.

Not every country has a Stapel-affair to generate inspiration/attention, but the consensus that something needs to change to make science and scientists more reliable is by now widespread. And if we would succeed, we think that performing science will also become much more fun than it is right now. More focus on originality and quality. We believe in a hunt for truth, not for papers. By the time many readers of the Amsterdam Student Journal will seriously start conducting research, we hope at least some of this desire to change will have become reality.

J. Tijdink & Y.M. Smulders

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top