Ben Goldacre’s second TED talk was published this month, and it’s on similar lines to the first (you can find his previous one here). He’s on top form and the whole thing is great, but I want to mention one part of it.
He starts of by discussing the infamous Bem study on precognition from a little while back. This study found that people’s emotional reactions to certain images (like a snake) sometimes occurred before they saw the picture, as though their spidey-sense was tingling.
This caused quite a stir. Believers felt vindicated and skeptics started looking for methodological flaws, and predicting that future replications would fail. However, some believers in the paranormal predicted that a replication would be attempted and fail too — but not because there’s no such thing as psychic powers.
Instead, the reason is that successful results are only possible when the collective consciousness of the people who are aware of them believes that precognition exists. Sort of like the character in Mystery Men who can only turn invisible when no one is watching, or, to complete the analogy, only when people who believe he can turn invisible are watching.
Once a study is made public, the consensus of the consciousnesses that are aware of it changes. Previously, the majority of the minds that knew about the study were believers in the paranormal, but now, the majority believe in the materialistic scientific model, which does not predict psychic powers to exist. Hence, further replications fail.
This explanation is quite a common one, and it’s also used by psychics, dowsers and telepathics who fail tests under controlled conditions — essentially, skeptical vibes interfere with psychic powers. In order to experience something in your life, you have to believe it to be true wholeheartedly; if you don’t believe in ghosts you’ll never see one, or at least, your experience would be easy to explain away.
Reasoning forward, if you’re a skeptic scientist studying the paranormal, you’re destined to the life of a debunker, while if you’re believer scientist studying the paranormal, you’re destined to be debunked by the skeptics. Presumably, paranormal scientist believers who later changed their tack never really believed.
But as Goldacre points out, there’s another explanation for this — publication bias. Science is probabilistic; it’s all about the likelihood of a certain hypothesis being correct, which gets closer and closer to proven with more positive studies, but it can never get there. Along the way, some studies will support a hypothesis and some won’t.
Even the historic discovery of the Higgs Boson earlier this year was like this. They didn’t get a picture on screen and go “Oh shit! There it is!” They continuously calculate a percentage chance that it exists by analaysing the data produced by the particle bashing and when that probability reaches a certain level, they release the findings.
Many of the particle collisions didn’t contribute to that probability, and so it is in human studies — only now we have a larger problem. The collider is continuously doing its work an analysing data, each collision adding to the total pool of data. But in peer-reviewed science, the pool of data tends to be skewed towards positive results. This is publication bias.
Prior to this study, there will have been lots and lots of tests of precognition and psychic powers. But, as Goldacre points out in the talk, try going to a leading journal and saying “Hey, I’ve got a study here saying that students can’t predict the future! Want to publish it?” It’s unlikely to work.
But combine these two things — the probabilistic nature of science and the tendency of journals to only publish positive results, and you get a problem — fluke findings are more likely to make their way into papers.
The Bem study was later repeated by Stuart Richie, Richard Wiseman and Christopher French, who failed to replicate the findings. They had trouble getting this one published, but eventually did. I wonder if they would have gotten their replication published if Bem’s study had never existed.
Still, all this is consistent with the collective consciousness idea, because we don’t know what happened in every single study on precognition ever conducted. If we did (which, ironically would probably require paranormal powers), we’d be able to say the collective consciousness idea is dead wrong because many studies conducted prior to this one were unsuccessful, therefore it’s nothing to do with belief and shared reality and everything to do with good old fashioned publication bias. For now I’ll let you make up your own mind on that one.
Moving to more earthly concerns than the nature of reality, this problem of publication bias isn’t limited to the supernatural. Every study looking at it has noted its prevalence, and the publication of studies in favour of certain drugs while negative results were buried has cost hundreds of thousands of lives. But I’ll let Ben Goldacre tell you all about that: