The study doesn’t say what the headline claims it says
“Coffee cures cancer.”
Three weeks later – “coffee causes cancer.”
Both headlines were real. Both cited real studies. Neither one gave you the information you’d actually need to change your behavior based on what the science shows.
Science journalism has a structural problem baked into it.
The incentives of journalism – drama, novelty, simplicity, clicks – are almost perfectly misaligned with the realities of how science actually works. Science is slow. Incremental. Full of uncertainty. Constantly revising itself based on new evidence. Headlines are fast, definitive, and engineered to make you feel like something has just been decided once and for all. So the translation from a lab report to a social media post goes through a bunch of steps, and at every step, the signal gets louder and the actual truth gets fuzzier.
Here’s how to read science news without getting played.
First – click through to the actual study.
Not the press release. Not the news article. The study itself. Or at the very least, find a summary that describes the methodology and sample size. Who were the subjects? How many? How were they selected? What did they actually measure? Most headline-driving claims fall apart the second you look at the methodology section of the underlying paper.
Mice are not small humans.
A massive percentage of medical findings that make headlines are based on animal studies. Animal studies are genuinely useful at early stages of research – they help identify which questions are worth asking and which compounds might be worth testing further. But a drug that cures cancer in mice has a very long road ahead before anyone knows if it does anything useful in humans. When a headline says a new treatment “shows promise,” stop and find out what that promise is actually based on. Half the time, it’s based on a handful of rodents.
Sample size matters.
This is the point from Essay 17 on statistics. A study of forty people is not the same as a study of forty thousand people. Small studies produce dramatic, publishable results all the time – and those dramatic results often don’t hold up at all when the research is repeated on larger populations. That’s not a failure of science. That’s just how science is supposed to work. The failure is when a news story treats a forty-person pilot like a settled finding.
Correlation is not causation.
This is the Essay 15.28 post hoc problem showing up in a new outfit. People who eat more vegetables tend to live longer. True. They also tend to have higher incomes, access to better healthcare, more education, less chronic stress, and more consistent sleep. Isolating the variable that actually matters in a causal sense is hard, expensive, and often not done in the studies that end up making headlines. When a story says “people who do X live longer,” the honest follow-up is – compared to who, controlling for what?
Check the funding.
Essay 16 covered why this matters. Who pays for the research is context, not automatic disqualification – but it’s a reason for extra scrutiny. Industry-funded studies have a well-documented pattern of producing more favorable results on the industry’s own products than independent studies of the same products do. That doesn’t mean every industry study is wrong. It means the baseline level of skepticism should go up, and the demand for independent replication should go up with it.
Look for replication.
One study is a starting point, not a conclusion. When independent research teams, working in different countries, using different methodologies, end up with similar results – that’s meaningful. That’s when something actually starts looking like a real finding. When a claim only comes from one lab, or hasn’t been tested in a different population, treat it as preliminary. Interesting. Worth watching. Not proven.
Science is the best tool we have for understanding reality.
Science journalism is often a distorted reflection of that tool. Reading the reflection clearly means knowing where the distortions come from – and staying skeptical of any headline that feels too satisfying, too certain, or too perfectly timed for your current feelings about coffee.