Re-examining the ganzfeld

Today's post asks a question not because I'm trying to lead you toward a particular answer, but because I honestly don't know the answer myself.

In Wednesday's post, I discussed some alleged claims by psychics (which a team in Australia evaluated for accuracy and found seriously wanting) and made the statement, "I'm all for keeping an open mind about things, but at some point you have to conclude that a complete absence of hard evidence means there's nothing there to see."

A friend of mine responded, "What if there's hard evidence out there that you're choosing not to accept because you've already made your mind up?"  He wasn't being combative; like me, he was just asking a question, and it's actually a reasonable thing to ask.  And he sent me a link to a post over at Paranormal Daily News that looks at one of the most famous experimental setups for detecting psychic powers: the ganzfeld experiment.

"Ganzfeld" is German for "complete field," and refers to the fact that the test subjects are placed in near-complete sensory deprivation, in order to keep them from receiving any information accept (allegedly) from telepathy.  Padded goggles are placed over the eyes; earpieces play recordings of white noise.  The subjects lie flat in a place with no drafts or other air movement.  (Some have even had subjects floating in a sensory deprivation tank.)  Then the "sender" -- usually the researcher conducting the experiment -- looks at some kind of pattern, often the famous "Zener cards" (cards with five different geometric patterns in five different colors), and the "receiver" (the test subject) reports what (s)he sees/experiences.

A participant in a ganzfeld experiment [Image is in the Public Domain]

The Wikipedia page for the ganzfeld experiment (linked above) is unequivocal; it says, "[It] is a pseudoscientific technique to detect parapsychological phenomena...  Consistent, independent replication of ganzfeld experiments has not been achieved."  However, the article my friend sent is equally unequivocal in the opposite direction -- that it has generated results that are far outside of what would come out of a random-choice statistical model, and has been done over and over with the same outcome.  The author, Craig Weiler, writes:

This works.  Not perfectly, but certainly well enough for an experiment.  It’s been done more than enough times by more than enough people to rule out any statistical anomalies.  The success rate is typically between 32 and 35%.  That’s pretty normal for a successful, statistics based experiment.  There have been six different meta analyses from skeptics and researchers alike, all showing positive results.  From an objective scientific perspective, this is an ordinary successful scientific experiment.

While I can't say I warm to the sneery tone of the article -- Weiler really needs to learn the difference between a "skeptic" and a "scoffer" -- it does bring up the question of who's right, here.  The critics of the ganzfeld experiment and other such attempts to prove the existence of paranormal abilities claim that no sufficiently-controlled experiment has ever generated positive results; the supporters claim that there are plenty of positive results that all the scientists are ignoring because they can't explain them (or, worse, because those results contradict their own biases).

Weiler is right that there have been meta-analyses done of the ganzfeld results, and that they have changed the minds of neither the pro- nor the anti- factions.  Finding a truly unbiased analysis has turned out to be not to be easy.  A September 2020 article in Frontiers in Psychology by Thomas Rayberon comes the closest of anything I've seen, but unfortunately tries to steer a middle course of "maybe, maybe not" by agreeing with both sides at once even though they're saying opposite things.  Rayberon writes (citations have been removed in the interest of space; go to the original article if you're interested in seeing them):

Psi research can be considered as a subfield of consciousness studies concerned with interactions between individuals and their environment that transcends the ordinary constraints of space-time.  Different lines of research have been developed for more than a century to tackle psi using experimental research, spontaneous cases, clinical cases, selected participants, and applications.  Several meta-analyses of studies conducted under controlled conditions examine precognitive dreams, telepathy, and presentiment and have demonstrated statistically significant effects...

While these results support the existence of consistent anomalous experience/behavior that has been labeled “psi,” there is currently no consensus in the scientific community concerning their interpretation and two main positions have emerged so far.  The “skeptics” suppose that they are the consequences of errors, bias, and different forms of QRPs [questionable research practices].  The “proponents” argue that these results prove the existence of psi beyond reasonable doubt and that new research should move on to the analysis of psi processes rather than yet more attempts to prove its existence.  This absence of consensus is related to the difficulty of drawing firm conclusions from the results of psi research.  Indeed, they represent an anomaly because there is currently no scientific model – based on physical or biology principles – to explain such interactions even if they exist.

Which reminds me from the quote from Lord of the Rings, "Go not to the Elves for advice, for they will say both yes and no."  The last bit -- that there is no current scientific model that could account for psychic phenomena -- is certainly true; but if there are statistically significant effects (which Rayberon says explicitly in the preceding paragraph), then surely there must be some protocol for devising an experiment that meets the minimum criteria of the true skeptics (people who base their understanding on the evidence, regardless of what their preconceived notions might have said).  The fact that there is no current scientific model to explain telepathy is, while correct, entirely irrelevant.  The first thing to do is to determine if the phenomenon itself is real.  There was no scientific model to explain radioactivity when it was discovered by Henri Becquerel, nor the apparent constancy of the speed of light when it was demonstrated by Michelson and Morley, nor the patterns of inheritance uncovered by Gregor Mendel.  The first thing was to determine if what they were seeing was accurate.  Once that happened, the scientists moved on to trying to figure out a model that accounted for it.

Rayberon then goes on to make quite a puzzling statement that implies it might be impossible even to tell if the phenomenon is real.  Science, he says (again, correct most of the time) uses experimental protocols that eliminate any possibility of interference by the experimenter.  That's impossible in psi research (italics are the author's):

Thus, if psi exists, the problem is the following: an advertent or inadvertent “direct” interaction between the researcher and the object of study could be possible.  This destroys the conditions necessary for the convincing scientific demonstration of psi itself.

Rayberon says this "paradox" makes psi research impossible to confirm or disconfirm.  But isn't an interaction between the researcher and the test subject what the psi researchers themselves are trying to demonstrate?  What an honest psi researcher -- well, any honest researcher, really -- needs to do is to isolate the variable (s)he's studying so that, as far as is possible, whatever results come out of the experiment can only be attributable to that variable.  So in a properly-conducted ganzfeld experiment, the researcher has eliminated any possibility of the test subject getting information about the pattern from anywhere except the mind of the "sender."

And from my admittedly layperson's viewpoint, that can't be all that hard to do.  If there have been multiple instances of positive, statistically-significant results from ganzfeld trials -- and Weiler and Rayberon agree that there have been -- then they deserve some explanation other than shrugging and saying, "I don't see how it could work."  If there are "errors, biases, and questionable research practices" generating the results, the "skeptics" (using the word in the sense both Weiler and Rayberon use) need to determine what those are.  If, on the other hand, the results aren't from poor experimental design or outright cheating, then let's have the "skeptics" and "proponents" team up to find a protocol they can both agree to.

Figuring out a model for what's going on can wait until we see if there is anything going on.

So after accusing Rayberon of playing both sides, I'm honestly not doing much better.  My inclination is to doubt the existence of psi abilities because the evidence seems sketchy for such a wild claim.  But that inclination is a bias I'm well aware of, and all it would take is one sufficiently well-designed experiment to convince me I was wrong.  Right now, all that seems to be happening is both sides becoming more entrenched and yelling at each other across no-man's land, which doesn't accomplish much but pissing everyone off.

So come on, folks.  Either psi exists or it doesn't.  If it doesn't, we can go on to studying actual real phenomena.  If it does, it will overturn pretty much everything we know about psychology, and would be one of the most colossal discoveries in the past hundred years.  How about teaming up and settling this question once and for all?

**************************************

Doubt, experiment, and reproducibility

Yesterday I got a response on a post I did a little over a year ago about research that suggested fundamental differences in firing patterns in the brains of liberals and conservatives.   The study, headed by Darren Schreiber of the University of Exeter, used fMRI technology to look at functionality in people of different political leanings, and found that liberals have greater responsiveness in parts of the brain associated with risk-seeking, and conservatives in areas connected with anxiety and risk aversion.

The response, however, was as pointed as it was short.  It said, "I'm surprised you weren't more skeptical of this study," and provided a link to a criticism of Schreiber's work by Dan Kahan over at the Cultural Cognition Project.  Kahan is highly doubtful of the partisan-brain study, and says so in no uncertain terms:

Before 2009, many fMRI researchers engaged in analyses equivalent to what Vul [a researcher who is critical of the method Schreiber used] describes.  That is, they searched around within unconstrained regions of the brain for correlations with their outcome measures, formed tight “fitting” regressions to the observations, and then sold the results as proof of the mind-blowingly high “predictive” power of their models—without ever testing the models to see if they could in fact predict anything. 

Schreiber et al. did this, too.  As explained, they selected observations of activating “voxels” in the amygdala of Republican subjects precisely because those voxels—as opposed to others that Schreiber et al. then ignored in “further analysis”—were “activating” in the manner that they were searching for in a large expanse of the brain.  They then reported the resulting high correlation between these observed voxel activations and Republican party self-identification as a test for “predicting” subjects’ party affiliations—one that “significantly out-performs the longstanding parental model, correctly predicting 82.9% of the observed choices of party.” 

This is bogus.  Unless one “use[s] an independent dataset” to validate the predictive power of “the selected . . .voxels” detected in this way, Kriegeskorte et al. explain in their Nature Neuroscience paper, no valid inferences can be drawn.  None.

So it appears that  Schreiber et al. were guilty of what James Burke calls "designing an experiment to find the kind of data you reckon you're going to find."  It would be hard to recognize that from the original paper itself without being a neuroscientist, of course.  I fell for Schreiber's research largely because I'm a generalist, making me unqualified to spot errors in highly specific, technical fields.

Interestingly, this comment came hard on the heels of a paper by Monya Baker that appeared last week in Nature called "1,500 Scientists Lift the Lid on Reproducibility."  Baker writes:

More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments.  Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research... 

Data on how much of the scientific literature is reproducible are rare and generally bleak.  The best-known analyses, from psychology and cancer biology, found rates of around 40% and 10%, respectively.  Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence. 

The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.  “At the current time there is no consensus on what reproducibility is or should be.”

The causes were many and varied.  According to the respondents, the failure to reproduce results derived from issues such as low statistical power to unavailability of method to poor experimental design; worse still, all too often no one bothers even to try to reproduce results because of the pressure to publish one's own work, not check someone else's.  As as result, slipshod research -- and sometimes, outright fraud -- gets into print.

How dire is this?  Two heartening responses described in Baker's paper include the fact that just about all of the scientists polled want more stringent guidelines for reproducibility, and also that work of high visibility is far more likely to be checked and verified prior to publication.  (Sorry, climate change deniers -- you can't use this paper to support your views.)

[image courtesy of the Wikimedia Commons]

What it means, of course, is that science bloggers who aren't scientists themselves -- including, obviously, myself -- have to be careful about cross-checking and verifying what they write, lest they end up spreading around bogus information.  I'm still not completely convinced that Schreiber et al. were as careless as Kahan claims; at the moment, all we have is Kahan's criticism that they were guilty of the multitude of failings described in his article.  But it does reinforce our need to think critically and question what we read -- even if it's in a scientific journal.

And despite all of this, science is still by far our best tool for understanding.  It's not free from error, nor from the completely human failings of duplicity and carelessness.  But compared to other ways of moving toward the truth, it's pretty much the only game there is.