Doubt, experiment, and reproducibility

Yesterday I got a response on a post I did a little over a year ago about research that suggested fundamental differences in firing patterns in the brains of liberals and conservatives.   The study, headed by Darren Schreiber of the University of Exeter, used fMRI technology to look at functionality in people of different political leanings, and found that liberals have greater responsiveness in parts of the brain associated with risk-seeking, and conservatives in areas connected with anxiety and risk aversion.

The response, however, was as pointed as it was short.  It said, "I'm surprised you weren't more skeptical of this study," and provided a link to a criticism of Schreiber's work by Dan Kahan over at the Cultural Cognition Project.  Kahan is highly doubtful of the partisan-brain study, and says so in no uncertain terms:

Before 2009, many fMRI researchers engaged in analyses equivalent to what Vul [a researcher who is critical of the method Schreiber used] describes.  That is, they searched around within unconstrained regions of the brain for correlations with their outcome measures, formed tight “fitting” regressions to the observations, and then sold the results as proof of the mind-blowingly high “predictive” power of their models—without ever testing the models to see if they could in fact predict anything. 

Schreiber et al. did this, too.  As explained, they selected observations of activating “voxels” in the amygdala of Republican subjects precisely because those voxels—as opposed to others that Schreiber et al. then ignored in “further analysis”—were “activating” in the manner that they were searching for in a large expanse of the brain.  They then reported the resulting high correlation between these observed voxel activations and Republican party self-identification as a test for “predicting” subjects’ party affiliations—one that “significantly out-performs the longstanding parental model, correctly predicting 82.9% of the observed choices of party.” 

This is bogus.  Unless one “use[s] an independent dataset” to validate the predictive power of “the selected . . .voxels” detected in this way, Kriegeskorte et al. explain in their Nature Neuroscience paper, no valid inferences can be drawn.  None.

So it appears that  Schreiber et al. were guilty of what James Burke calls "designing an experiment to find the kind of data you reckon you're going to find."  It would be hard to recognize that from the original paper itself without being a neuroscientist, of course.  I fell for Schreiber's research largely because I'm a generalist, making me unqualified to spot errors in highly specific, technical fields.

Interestingly, this comment came hard on the heels of a paper by Monya Baker that appeared last week in Nature called "1,500 Scientists Lift the Lid on Reproducibility."  Baker writes:

More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments.  Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research... 

Data on how much of the scientific literature is reproducible are rare and generally bleak.  The best-known analyses, from psychology and cancer biology, found rates of around 40% and 10%, respectively.  Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence. 

The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.  “At the current time there is no consensus on what reproducibility is or should be.”

The causes were many and varied.  According to the respondents, the failure to reproduce results derived from issues such as low statistical power to unavailability of method to poor experimental design; worse still, all too often no one bothers even to try to reproduce results because of the pressure to publish one's own work, not check someone else's.  As as result, slipshod research -- and sometimes, outright fraud -- gets into print.

How dire is this?  Two heartening responses described in Baker's paper include the fact that just about all of the scientists polled want more stringent guidelines for reproducibility, and also that work of high visibility is far more likely to be checked and verified prior to publication.  (Sorry, climate change deniers -- you can't use this paper to support your views.)

[image courtesy of the Wikimedia Commons]

What it means, of course, is that science bloggers who aren't scientists themselves -- including, obviously, myself -- have to be careful about cross-checking and verifying what they write, lest they end up spreading around bogus information.  I'm still not completely convinced that Schreiber et al. were as careless as Kahan claims; at the moment, all we have is Kahan's criticism that they were guilty of the multitude of failings described in his article.  But it does reinforce our need to think critically and question what we read -- even if it's in a scientific journal.

And despite all of this, science is still by far our best tool for understanding.  It's not free from error, nor from the completely human failings of duplicity and carelessness.  But compared to other ways of moving toward the truth, it's pretty much the only game there is.

The partisan brain

I tend to avoid politics, both here on Skeptophilia and also in my personal life.  There are two reasons for this: first, I find most political issues such a snarled Gordian knot that I have no idea how anyone could be smart enough to solve them; and second, even on the issues about which I have strong opinions, I've found that arguing with people seldom changes minds on either side.  So entering into an argument that basically is "I think so because I think so," and is unlikely to convince anyone of anything, might be the very definition of the word "pointless."

[image courtesy of the Wikimedia Commons]

I do find it interesting, however, to consider why people so seldom shift their political views, even when presented with facts and data to the contrary.  It's like we're stuck in our worldview, unable to move away from the narrow little window we're looking out of.  And now, some scientific research might have an answer for why that is.

Darren Schreiber et al. of the University of Exeter published a fascinating paper last month called "Red Brain, Blue Brain: Evaluative Processes Differ in Democrats and Republicans" in the online journal PLOS-ONE, which has as its main claim that there are fundamental differences in brain function between liberals and conservatives.  Studies had already shown that there is a brain structure difference; liberals tend to have more gray matter in the anterior cingulate cortex, conservatives in the amygdala.  But suggestive as that is, differences in structure don't always imply differences in function, so it was premature to conclude that these structural differences caused individuals to adopt particular political stances.

In this case, however, it appears that the earlier researchers were on to something.  In particular, there seemed to be a biological underpinning to the well-demonstrated tendency of conservatives to be risk-averse and liberals to be risk-seeking.  The authors write:

(C)onservatives demonstrate stronger attitudinal reactions to situations of threat and conflict. In contrast, liberals tend to be seek out novelty and uncertainty.  Moreover, Democrats, who are well known to be more politically liberal, are more risk accepting than Republicans, who are more politically conservative.  While ideology appears to drive reactions to the environment, environmental cues also influence political attitudes.  For instance, external threats prime more conservative attitudes among liberals, moderates, and conservatives.

Schreiber et al. set out to see if the different attitudes toward risk would show up on a fMRI, which would indicate that there was a functional difference between the brains of liberals and conservatives:

To test a conjecture that ideological differences between partisans reflect distinctive neural processes, we matched publicly available party registration records with the names of participants (35 males, 47 females) who had previously taken part in an experiment designed to examine risk-taking behavior during functional brain imaging...  Individuals completed a simple risk-taking decision-making task during which participants were presented with three numbers in ascending order (20, 40, and 80) for one second each.  While pressing a button during the presentation of the number 20 on the screen always resulted in a gain of 20 cents, waiting to select 40 or 80 was associated with a pre-determined possibility of either gaining or losing 40 or 80 cents.  Therefore, participants chose between a lower “safe” payoff and a higher risky payoff.  The probabilities of losing 40 or 80 cents were calibrated so that there was no expected value advantage to choosing 20, 40 or 80 during the task, i.e. the overall pay-off would have been the same for each pure strategy.

They found that the two groups did, indeed, show different levels of activity in the two parts of the brain that earlier research had shown to differ:

Consistent with the findings of structural differences by Kanai et al, significantly greater activation was observed in the right amygdala for Republicans and in the left posterior insula (near the temporal-parietal junction) in Democrats when making winning risky versus winning safe decisions. No significant differences were observed in the entorhinal cortex or anterior cingulate cortex. All attempts to use behavior to distinguish Republicans from Democrats were unsuccessful, suggesting that different neural mechanisms may underlie apparently similar patterns of behavior.

The authors are clear in their conclusion that they, too, have established correlation, but are yet to show causation; "One might infer that the differing brain structures identified by Kanai et al. suggest genetic foundations for the differences in ideology," they write, in their discussion of results.  "However, recent work has shown that changes in cognitive function can lead to changes in brain structure."  So how much of the difference they and others have shown is genetic in origin, and how much due to remodeling of the brain's circuitry because of the environment, is still uncertain.

It does support, however, the fruitlessness of political argument.  If there is a biological underpinning to political stance, it's to be expected that it's not going to be easy to change.  There are cases, of course, where it's important to try -- in issues of social justice and care for the environment, for example.  But this research shows pretty clearly that such battles aren't going to be easy to win.

And you have to wonder what a fMRI would show for a generally apolitical person like myself.  No brain activity whatsoever?

Maybe I'm better off not knowing.