Doubt, experiment, and reproducibility

Yesterday I got a response on a post I did a little over a year ago about research that suggested fundamental differences in firing patterns in the brains of liberals and conservatives.   The study, headed by Darren Schreiber of the University of Exeter, used fMRI technology to look at functionality in people of different political leanings, and found that liberals have greater responsiveness in parts of the brain associated with risk-seeking, and conservatives in areas connected with anxiety and risk aversion.

The response, however, was as pointed as it was short.  It said, "I'm surprised you weren't more skeptical of this study," and provided a link to a criticism of Schreiber's work by Dan Kahan over at the Cultural Cognition Project.  Kahan is highly doubtful of the partisan-brain study, and says so in no uncertain terms:

Before 2009, many fMRI researchers engaged in analyses equivalent to what Vul [a researcher who is critical of the method Schreiber used] describes.  That is, they searched around within unconstrained regions of the brain for correlations with their outcome measures, formed tight “fitting” regressions to the observations, and then sold the results as proof of the mind-blowingly high “predictive” power of their models—without ever testing the models to see if they could in fact predict anything. 

Schreiber et al. did this, too.  As explained, they selected observations of activating “voxels” in the amygdala of Republican subjects precisely because those voxels—as opposed to others that Schreiber et al. then ignored in “further analysis”—were “activating” in the manner that they were searching for in a large expanse of the brain.  They then reported the resulting high correlation between these observed voxel activations and Republican party self-identification as a test for “predicting” subjects’ party affiliations—one that “significantly out-performs the longstanding parental model, correctly predicting 82.9% of the observed choices of party.” 

This is bogus.  Unless one “use[s] an independent dataset” to validate the predictive power of “the selected . . .voxels” detected in this way, Kriegeskorte et al. explain in their Nature Neuroscience paper, no valid inferences can be drawn.  None.

So it appears that  Schreiber et al. were guilty of what James Burke calls "designing an experiment to find the kind of data you reckon you're going to find."  It would be hard to recognize that from the original paper itself without being a neuroscientist, of course.  I fell for Schreiber's research largely because I'm a generalist, making me unqualified to spot errors in highly specific, technical fields.

Interestingly, this comment came hard on the heels of a paper by Monya Baker that appeared last week in Nature called "1,500 Scientists Lift the Lid on Reproducibility."  Baker writes:

More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments.  Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research... 

Data on how much of the scientific literature is reproducible are rare and generally bleak.  The best-known analyses, from psychology and cancer biology, found rates of around 40% and 10%, respectively.  Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence. 

The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.  “At the current time there is no consensus on what reproducibility is or should be.”

The causes were many and varied.  According to the respondents, the failure to reproduce results derived from issues such as low statistical power to unavailability of method to poor experimental design; worse still, all too often no one bothers even to try to reproduce results because of the pressure to publish one's own work, not check someone else's.  As as result, slipshod research -- and sometimes, outright fraud -- gets into print.

How dire is this?  Two heartening responses described in Baker's paper include the fact that just about all of the scientists polled want more stringent guidelines for reproducibility, and also that work of high visibility is far more likely to be checked and verified prior to publication.  (Sorry, climate change deniers -- you can't use this paper to support your views.)

[image courtesy of the Wikimedia Commons]

What it means, of course, is that science bloggers who aren't scientists themselves -- including, obviously, myself -- have to be careful about cross-checking and verifying what they write, lest they end up spreading around bogus information.  I'm still not completely convinced that Schreiber et al. were as careless as Kahan claims; at the moment, all we have is Kahan's criticism that they were guilty of the multitude of failings described in his article.  But it does reinforce our need to think critically and question what we read -- even if it's in a scientific journal.

And despite all of this, science is still by far our best tool for understanding.  It's not free from error, nor from the completely human failings of duplicity and carelessness.  But compared to other ways of moving toward the truth, it's pretty much the only game there is.

Thought vs. experiment

To a scientist, there's no more fundamental approach to knowledge than experimentation.  You want to find something out?  Design an experiment to see if your idea about how the world works is correct.  Good scientists are always testing, questioning, and trying to find new ways to tweak the system and see how it responds.

What's fascinating from a historical perspective is that this is a fairly new way to approach knowledge.  In general, the pre-Enlightenment attitude was that if you wanted to learn, you simply had to think about stuff.  Thought was considered to be the purest way to gain knowledge; no need to contaminate your brain with dirty, clunky, uncooperative matter.  Even Kepler started out from this standpoint -- when he first started to work on the problem of the shapes of planetary orbits, he began from the assumption that they were circles (because circles are "perfect") and that the relationship between one planet's orbit and the next had something to do with the "Five Perfect Solids" of Greek mathematical theory.  Fortunately, Kepler was (1) working with a rigorous experimentalist, Tycho Brahe, and (2) honest, because he found out pretty quickly that his ideas weren't working -- and was forced to the uncomfortable conclusion that planetary orbits were messy, lopsided ellipses.  Galileo, you might recall, faced persecution for church officials not because of heresy with regards to religious doctrine, per se -- his problems with the Vatican started because of three claims, one famous (his acceptance of the heliocentric model) and the other two less-known (his rejection of Aristotle's claims that an object's falling speed is dependent on its mass, and that objects float or sink in water depending upon their shape).  It's fascinating, and not a little horrifying, that church officials had demonstrated for them experiments supporting Galileo's conclusions -- and they still didn't believe the evidence of their eyes, preferring instead the "pure thought" of Aristotle and Plato, for whom experimentation was somehow intrinsically suspect.

Amazingly, that idea -- that you can arrive at the truth just by thinking about it -- lingers still.  Some of it is relatively innocent, the sort of thing I see in high school science classes -- misconceptions that stem from the thought, "Well, of course it works that way.  That seems logical."  More insidious, though, are the schools of thought that embrace that approach, that deliberately eschew experimentation in favor of contemplation.  And in the last couple of days, I found two excellent examples of just this way of thinking.

The first one was in the online version of Fate magazine, so I suppose I shouldn't be all that surprised, considering the source.  Entitled, "Auric Energy Fields and Their Effects on Electronics," the article in question, written by "noted wisdom teacher" Kala Ambrose, looks at the alleged phenomenon of people whose presence can somehow interfere with electronic devices from computers to DVRs to streetlights.  And she makes the following statement:

As a psychic, I see the aura around people, which is a flexible field of energy around the body with many layers. The level closest to your body, is described as the etheric body and in a sense, it’s the battery of the body, receiving and emitting electrical impulses in and out from your body. You bring energy in and you release energy, all through the auric body. There are many layers extending outward from the etheric body including the mental layer and the emotional layer, both of which are also energy fields where we store and emit energy and we bring this energy into and down into the physical body from these layers... For some people, who also tend to have psi abilities, they release this pent up energy in a wave. I refer to it as an energy blast, which can affect the environment around them. One way that these people begin to notice this effect, is that they will find when walking or driving by street lights, that the lights will go off or turn on when they pass by. If this has happened to you, you are releasing this pent up energy or someone near you is releasing their energy... The over-abundance of energy that you described, can affect lights and other electronics when released in a quick blast. Think of it as an energy surge. Typically this indicates that the person is not aware of the energy they are releasing and so it comes as a surprise when an electronic device is affected. For many people, they emit this energy the strongest when they are agitated, stressed or in a high emotional state (positive or negative).

Now, let's assume for a moment, just for fun, that the phenomenon is real; i.e., that the people who claim to interfere with electronic devices are telling the truth.  What I find the most interesting about Kala Ambrose's claims is that never once does she seem to think, "Hey!  If some guy's body is emitting enough energy to interfere with a computer, that has to be measurable!  Maybe we should build a device to measure, test, and study this 'auric energy field.'"  No, she seems to believe that all you need to do to understand this is to think about it:

The next time this occurs, stop right away and ask yourself, How am I feeling, What’s on my mind right now? Also ask those present what they noticed when it occurred. Gather this information to discern what the triggers are that set off the energy spikes.

An even more striking example of this philosophical approach to science comes from Joseph Farrell's blog Giza Death Star, in which he responds to a press release from the world of physics in a post titled "Space-Time Crystals."  Farrell, to his credit, posts a link to the original press release, and from that press release we learn that Frank Wilczek of MIT and Xiang Zhang and Tongcang Li of UC Berkeley are working on trapping loops of ions inside crystals, creating an rotating charge signal that would "(break) temporal symmetry."  Wilczek is careful to specify that the "space-time crystal" thus created would span only extremely small distances (a tenth of a millimeter) and exist only at phenomenally low temperatures (one-billionth of a degree Kelvin), and that "being in their ground states, such systems could not be employed to produce useful work."

Farrell, on the other hand, begs to differ.

He says that he beat Wilczek, Zhang, and Li to the punch years ago, and did it without ever performing a single experiment:

Way back when, when I began writing my high speculations and sharing them with the public, I began by deciding to “take the plunge” and “high dive” off the deep end, and share my hypothesis that the Great Pyramid may have been a sophisticated kind of phase conjugate mirror manipulating the fabric of the physical medium itself. And at the end of my first book on the subject, I speculated on a kind of crystal that would somehow be able to trap and rotate EM waves. Not knowing what to call such bizarre things, I simply call them “phi” crystals, since they were suggested to me by the constant phi, and by the Fibonacci sequence. My reason for thinking that such crystals would be an integral component of any such machine was simply that there would have to be some sort of coupled oscillator able to interact with the “rotation moment” of the fabric and structure of the local medium, or local space-time.

Now, from my admittedly rather rudimentary understanding of physics, this sounds like a lot of horse waste right from the get-go, but what I find the most interesting about all of Farrell's blathering on about this is that he jumps right past Wilczek's cautions that since space-time crystals are in their ground state, the laws of thermodynamics would render it impossible for them to perform work -- and describes how these curiosities could become "sources of energy" that would "make our largest thermonuclear bombs look like firecrackers."

And how did he arrive at all of this?  Apparently, just by pondering the Fibonacci sequence and other such constructs:

But imagine, for a moment, the possibility that such a technology could be turned into, say, a source of energy...  (T)o my mind anyway, the possibility – long term to be sure – opens up that such things could eventually become sources of energy. We’re a long way from that, to be sure, and even a long way of any such verified understandings of these wildly speculative ideas, but nonetheless, the possibility should be mentioned.

I find it even more curious that Farrell is weighing in on subtle concepts in physics when his own Ph.D. is in patristics.  What is patristics, you might ask?  I had to ask, because I didn't know, and found out that patristics is "the study of early Christian writers, known as the Church Fathers."  Yup, that will certainly prepare you to comprehend abstruse concepts in solid-state physics.

So, anyway, the Platonic ideal of arriving at knowledge just by analyzing it with Pure Thought is with us still, apparently.  And just as it did in the case of Galileo's detractors, without the foundation of data, evidence, and experiment to support it, theoretical musings are just as likely to go wrong as right.  It is exactly this error in approach that science corrects -- even though there are people out there who still don't see why all that silly experimentation should be necessary.