CLINICAL TRIALS ARE VITAL TO SCIENCE, YET RESEARCHERS KEEP MOVING THE GOALPOSTS
This is Science Fictions with Stuart Ritchie, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.
Imagine I said to my editor I was going to write today’s article about, say, antidepressants. “Okay”, he replies, “that’s a good topic. I’ll put you down for one on antidepressants, then.”
And then, when I come to file the article with him, it turns out I actually wrote it about cancer research.
Now, there’s nothing wrong with an article about cancer research (as you’ll hopefully agree once you read this one). But it is at least a little odd that I planned to write about antidepressants but then switched the topic. Did the article about antidepressants not work out, for some reason? Maybe the scientific studies didn’t provide the evidence I needed to make the point I wanted to make?
If this happened in the context of a newspaper article, it would perhaps be puzzling (not to mention very irritating for my editor), but it probably wouldn’t be a massive deal in the grand scheme of things. But in science, planning to do one thing then doing another is rife – and it has big consequences for how we understand important areas of research.
Important areas like cancer research. In comparison to something like social psychology, which is of course interesting but arguably less pressing than the quest for new cancer treatments, you’d think that cancer research would be taken with a great deal more seriousness.
And indeed, clinical trials in general are subject to all sorts of rules that other areas of research – which don’t involve testing experimental medical treatments on human subjects – don’t have to worry about.
One of those rules is trial pre-registration: if you’re doing a clinical trial, you have to register your plans for the trial in a government database.
In that registration, you have to say what the trial is going to be about. Specifically, you have to declare what your primary endpoint will be. That’s the main result, assessed after the trial ends, that essentially tells you whether the trial worked or not.
An obvious primary endpoint in cancer research might be survival: whether the cancer patients taking your new drug survived longer than those on a placebo drug, or an older drug that might be less effective. Another might be the progression of the disease: the extent to which a tumour has changed size by a certain point in time.
You can also have secondary endpoints. These tend to be less important than the big, survival-related ones – they might be subjective, like asking the patients about their quality of life, or could be blood tests for cancer-related chemicals.
The point is that the primary and secondary endpoints are all set in stone before the trial takes place: the researchers have “called their shots”, like an archer saying they’re going to hit exactly this part of the target. That makes it all the more impressive if they go on to actually hit that part – imagine if they only declared which part they were aiming at after they’d shot the arrow (the latter, by the way, is effectively what all non-registered scientific studies are doing).
A new study of endpoint switching
Except… the endpoints often aren’t set in stone at all. A scary proportion of clinical trials “switch” their registered endpoints, just like I switched the topic of this article. The newest evidence on this comes from a study published last week that took 755 cancer clinical trials and compared what they said they’d do to what they actually did (they were all phase-3 trials, so they all involved human patients and a treatment they had good reason to think would work).
They looked at what the studies had declared as their primary endpoint, and checked to see whether by the time of publication they had either:
- swapped the primary endpoint for a secondary one;
- mysteriously not mentioned the primary endpoint at all;
- introduced a new primary endpoint;
- moved the goalposts by redefining their primary endpoint.
Out of 755 trials, 145 – that’s 19.2 per cent – did one of these “outcome switches”.
One in five trials having an outcome switched is a big problem. It implies that more statistical analyses were run than were revealed, which could easily mean a higher rate of untrue, false-positive results.
It implies that the authors of the trial didn’t get the result they “wanted”—a positive result for their treatment of choice—and were willing to bend the rules to make it nonetheless appear that they did. It implies that they haven’t been fully straight with us, and might in some cases be trying to hide something. It’s not, overall, a huge vote of confidence in the quality of their research.
Outcome switching doesn’t always imply something nefarious, of course: there can in some cases be a good reason for it. Maybe another study was published, halfway through your trial, and it showed that a different endpoint was actually a better measure of cancer progression. Maybe – as the authors of the new study suggest, bending over backwards to be charitable – someone simply recorded the primary endpoint wrong on the study registration.
But then, for full transparency, you’d want the authors of the study to tell us why they switched the outcome. Unfortunately, the new study found, fewer than a third of the articles with outcome switches actually explained what they’d done and why they’d done it. Again, that’s not a brilliant signal of integrity on the part of the triallists.
And it gets worse: of the studies that had an outcome switch, 61 per cent showed positive results; of the ones without a change, only 50 per cent were positive. It’s not by any means certain, but this difference could be because the outcome switching caused the positive results in the first place. If your cancer drug doesn’t look like it works according to one outcome, just forget that one and switch to another that looks more promising – even if its “results” might be due to random chance.
There are a couple of issues with this new study. Despite being all about pre-registration, this study wasn’t itself pre-registered – so we don’t know whether they stuck to their plan for assessing endpoints. And despite being all about transparency, the authors didn’t provide a list of the trials they looked at and the ones that had endpoint switches.
Technically you could go back and do their entire analysis again and you’d probably find similar results – but why should you have to?
I don’t think these problems are catastrophic or anything, but they are ironic. If you’re going to do a study where the ultimate recommendation is that studies should be more open and transparent, surely you should be as open and transparent as possible yourself.
And now, the secondary outcome
In any case, this is hardly the only evidence we have of rampant endpoint switching in research. You might remember that I “switched” the topic of this article from antidepressants to cancer research – well, if we look at our secondary outcome of antidepressant trials (which, er, has been the secondary outcome all along – I promise!), we find that they do outcome switching there, too.
A 2018 study found that, not only are antidepressant trials with positive results far more likely to be published, but ones with negative results (that is, ones that say the antidepressant in question doesn’t work in comparison to alternatives) are often retroactively turned into a positive trial using outcome switching.
Lots of other areas of science are starting to do (and require) pre-registrations as well. The problem of outcome-switching is a perfect reminder of how important it is not to become complacent: just because a scientist appears to be being transparent – “I registered my study online before I did it!”- doesn’t mean they’ve actually followed all the rules.
But, while we’re talking about antidepressants, there is a slight glimmer of hope from a study published last year. It looked at antidepressant trials over time, and whether negative results were magically being transformed into positive ones, or whether they were being reported accurately. Although the rates of misreporting were still quite high in the more recent trials, and although studies with negative results were more likely to obfuscate those “disappointing” conclusions, the overall level of transparent, honest reporting had substantially improved in the more recent trials.
Could it be that, after decades of talking about the problems of outcome-switching, the message is beginning to get through?
Other things I’ve written recently
I’m as big a fan as you can get of the new class of weight-loss drugs – in particular semaglutide. But I still wrote a piece in response to the latest wave of social media chat claiming the drug can also improve your willpower. We just don’t have any convincing evidence of this yet, so let’s all… exercise a little bit of self-control when discussing it.
Science link of the week
I had not the slightest clue this was happening, but according to a new ProPublica report, there are fake, pay-to-publish “scientific journals” where your high schooler can publish a paper in order to make their university application look more impressive.
It’s another example of the CV arms race – the CVs required for the same place, position, or job getting more and more inflated over time – and not exactly brilliant for scientists either, who now have their searches for research polluted with papers written by high-school kids.
2023-05-25T18:28:40Z dg43tfdfdgfdThis is Science Fictions with Stuart Ritchie, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.
