The vagaries of scientific experiments

One usually likes to think that scientists know what they’re doing. Here’s something that might shake your confidence. In bio-medical research, scientists often use cell lines. These are in vitro cells, originally taken from a human or animal donor, which can be experimented on to help develop new drugs or treatments. The problem is that, according to one review, in “at least 5 percent” of studies, the scientists have totally mixed up where the cells came from. This means that in at least one in 20 studies that were sent off for peer review the scientists were completely confused about the most basic element of their research. They thought, for instance, they were doing research on lung cells when they actually had pancreatic ones. In more extreme cases, they’ve fooled themselves into thinking they’re looking at human cells when in fact they come from rats. No wonder that when other scientists come along later and try to repeat the experiments, they can’t get the same results.

Major efforts are now going into forcing scientists to prove the provenance of their cell lines – as well as their lab mice, whose genetic background makes a huge difference to results of experiments but whose origin is often just as unclear as the cell lines. These and the many other avoidable screw-ups by biomedical researchers are among the topics of Unreliable, a depressing tell-all book by the high-powered medical scientist Csaba Szabo. He has seen it all – papers sent to him for peer review that contain obvious fraud; his own research plagiarized and mysteriously republished in dodgy Indian journals; grant review sessions where top professors can’t agree which proposed studies are good or bad.

To his credit, he discusses a time when (he suspects) a junior member of his lab faked some laboratory images and sent them to be published in a journal under the lab’s collective names. (The person in question went on to be caught faking images in several papers once he’d left Szabo’s group.) When alerted to the problem by some online data sleuths who make a hobby of checking research papers, Szabo ensured that the paper was retracted from the literature.

Alas, such fraud is far from rare in academia – and a lot of it occurs at the highest level. Szabo mentions the story of Marc Tessier-Lavigne, who rose to be president of Stanford University before it was realized that several studies he’d published when he was a lab researcher contained fake data (he resigned in 2023). One Nobel Prize-winning cancer researcher has now had 13 papers retracted, all for potentially fraudulent results.

Why do many scientists act this way? One reason is that they don’t, in general, earn much money. Someone once said that doing a PhD was “choosing to forego current income in order to forego future income.” Instead, they’ve built an elaborate system of status – which comes from credentials, recognition, grant money, and, perhaps most of all, publication. Conspicuously absent from that list is “ultra-reliable, replicable, solid results.”

The scientific publication system, aside from being a giant money-burning scam, is the root of many of our research problems. It emphasizes “exciting,” “groundbreaking,” “breakthrough” findings instead of staider but more robust research. And the hyper-competition among scientists to publish more and more papers means that many of them are happy to bend or fully break some of the rules to get ahead. At the very least, this race to get published, combined with the peer review system where reviewers work on an entirely voluntary basis (and are themselves busy, distracted researchers), means that errors like the aforementioned misidentification of cell lines are far less likely to be spotted.

Szabo describes these problems and adds more that I hadn’t really considered, including a (perhaps overly long) section on the process scientists need to go through to get a US visa – where a long list of publications gives your application a boost. It’s just another incentive to push the boundaries to extend your scientific CV, even if the publications it describes aren’t much cop. I confess I did find it strange that the book effectively opens with this visa discussion – an odd choice, given the much juicier stories that come later. In general, this isn’t a well-ordered or well-edited volume. In many places it reads like a sloppy first draft – and I found myself irked by the huge overuse of cartoons, especially when they’re often poorly drawn and where the humor levels never rise above lukewarm.

There’s also something unsatisfying about another list of science’s failings. It’s now been something like 15 years since scientists started talking prominently about the “replication crisis,” where people can’t repeat the results of other labs because the original experiments were so unreliable. There have been endless news articles, documentaries and books on the topic, including one, I should say for full disclosure, by me.

Scientists might think they are looking at human cells when in fact they come from rats 

Yes, consciousness-raising is good, and a frightening number of scientists remain seemingly oblivious to the failings in their field. But I couldn’t help feeling that now’s the time for some answers. I’d love to read a bold thesis about what a 21st-century, reliability-maxxing science should look like. Alas, this is not it.

Sure, Szabo includes a brief section on how we might need to do away with the idea of the scientific paper (a living-fossil technology that hasn’t changed all that much since it was invented in 1665). He talks about punishing scientific fraudsters with more than a slap on the wrist. He suggests changing incentives at universities so that they hire a different sort of researcher who’s more interested in rigor and transparency.

But how? Szabo isn’t sure. He hopes that when enough politicians and taxpayers realize that they’re being taken for a disgraceful, wasteful ride by the scientific establishment they’ll rise up and do something about it. But, as I do whenever I encounter someone who hopes for a revolution, I yearned for more concrete policy ideas. Szabo knows this world inside out and has managed to rise to the top of his field. Doesn’t he have more specific suggestions for what would make it better?

Maybe a revolution is coming regardless, with the rise of AI. This could bring unprecedented scientific progress, as presaged by the recent advances made by Google DeepMind. But it could also make science’s unreliability problems a lot worse. We’ve all heard of AIs that allow criminals to steal your voice (or even your appearance) and use a simulacrum of you to bilk money out of your grandmother. AIs will also allow scientific fraudsters to generate perfect false data sets that evade everyone’s detection and fake lab images that bear no hint of Photoshopping. Never mind just proving where cell lines and lab mice have come from. In a world of perfect AI sham science, researchers will have to prove where every single lab result comes from, otherwise nobody will believe a word they say.

How will we deal with this? Will we need a scientific Nineteen Eighty-Four, where labs are constantly monitored by CCTV cameras and each keystroke on a scientist’s computer is logged? That sounds like a scary future. But the thing is, if you accept what Szabo is saying about the sheer unreliability of much of our current scientific research, we’ve needed something like it for a very long time.