Skeptophilia (skep-to-fil-i-a) (n.) - the love of logical thought, skepticism, and thinking critically. Being an exploration of the applications of skeptical thinking to the world at large, with periodic excursions into linguistics, music, politics, cryptozoology, and why people keep seeing the face of Jesus on grilled cheese sandwiches.

Tuesday, August 28, 2018

What we've got here is a failure to replicate

I frequently post about new scientific discoveries, and having a fascination for neuroscience and psychology, a good many of them have to do with how the human brain works.  Connecting behavior to the underlying brain structure is not easy -- but with the advent of the fMRI, we've begun to make some forays into trying to elucidate how the brain's architecture is connected to neural function, and how neural function is connected to higher-order phenomena like memory, learning, instinct, language, and socialization.

Whenever I post about science I try my hardest to use sources that are from reputable journals such as Science and Nature -- and flag the ones that aren't as speculative.  The reason those gold-standard journals are considered so reliable is because of a rigorous process of peer review, wherein scientists in the field sift through papers with a fine-toothed comb, demanding revisions on anything questionable -- or sometimes rejecting the paper out of hand if it doesn't meet the benchmark.

[Image is in the Public Domain]

That's why a paper published in -- you guessed it -- Nature had me picking my jaw up off the floor.  A team of psychologists and social scientists, led by Colin Camerer of Caltech, took 21 psychological studies that had been published either in Nature or in Science and didn't just review them carefully, but tried to replicate their results.

Only 13 of them turned out to be replicable.

This is a serious problem.  I know that scientists are fallible just like the rest of us, but this to me doesn't sound like ordinary fallibility, it sounds like outright sloppiness, both on the part of the researchers and on the part of the reviewers.  I mean, if you can't trust Nature and Science, who can you trust?

Anna Dreber, of the Stockholm School of Economics, who co-authored the study, was unequivocal about its import.  "A false positive result can make other researchers, and the original researcher, spend lots of time and energy and money on results that turn out not to hold," she said.  "And that's kind of wasteful for resources and inefficient, so the sooner we find out that a result doesn't hold, the better."

Brian Nosek, of the University of Virginia, was also part of the team that did the study, and he thought that the pattern they found went beyond the "publish-or-perish" attitude that a lot of institutions have.  "Some people have hypothesized that, because they're the most prominent outlets they'd have the highest rigor," Nosek said.  "Others have hypothesized that the most prestigious outlets are also the ones that are most likely to select for very 'sexy' findings, and so may be actually less reproducible."

One heartening thing is that as part of the study, the researchers asked four hundred scientists in the field who were not involved with the study to take a look at the 21 papers in question, and make their best assessment as to whether it would pass replication or not.  And the scientists' guesses were usually correct.

So why, then, did eight flawed, non-replicable studies get past the review boards of the two most prestigious science journals in the world?  "The likelihood that a finding will replicate or not is one part of what a reviewer would consider," Nosek said.  "But other things might influence the decision to publish.  It may be that this finding isn't likely to be true, but if it is true, it is super important, so we do want to publish it because we want to get it into the conversation."

Well, okay, but how often are these questionably-correct but "super important" findings labeled as such?  It's rare to find a paper where there's any degree of doubt expressed for the main gist (although many of them do have sections on the limitations of the research, or questions that are still unanswered).  And it's understandable why.  If I were on a review board, I'd definitely look askance at a paper that made a claim and then admitted the results of the research might well be a fluke.

So this is kind of troubling.  It's encouraging that at least the inquiry is being made; identifying that a process is flawed is the first step toward fixing it.  As for me, I'm going to have to be a little more careful with my immediate trust of psychological research just because it was published in Nature or Science.

"The way to get ahead and get a job and get tenure is to publish lots and lots of papers," said Will Gervais of the University of Kentucky, who was one of the researchers whose study failed replication.  "And it's hard to do that if you are able run fewer studies, but in the end I think that's the way to go — to slow down our science and be more rigorous up front."


This week's Skeptophilia book recommendation is from one of my favorite thinkers -- Irish science historian James Burke.  Burke has made several documentaries, including Connections, The Day the Universe Changed, and After the Warming -- the last-mentioned an absolutely prescient investigation into climate change that came out in 1991 and predicted damn near everything that would happen, climate-wise, in the twenty-seven years since then.

I'm going to go back to Burke's first really popular book, the one that was the genesis of the TV series of the same name -- Connections.  In this book, he looks at how one invention, one happenstance occurrence, one accidental discovery, leads to another, and finally results in something earthshattering.  (One of my favorites is how the technology of hand-weaving led to the invention of the computer.)  It's simply great fun to watch how Burke's mind works -- each of his little filigrees is only a few pages long, but you'll learn some fascinating ins and outs of history as he takes you on these journeys.  It's an absolutely delightful read.

[If you purchase the book from Amazon using the image/link below, part of the proceeds goes to supporting Skeptophilia!]

No comments:

Post a Comment