Statistical fudging

The last thing we need right now is for people to have another reason to lose their trust in scientists.

It's a crucial moment.  On the one hand, we have the Intergovernmental Panel on Climate Change, which just a week and a half ago released a study that we have only twenty or so years left in which we can take action to limit the warming to an average of 1.5-2.0 C by 2050 -- and even that will almost certainly increase the number of major storms, shift patterns of rainfall, cause a drastic rise in sea level, and increase the number of deadly heat waves.  And it bears mention that a lot of climate scientists think that even this is underselling the point, giving politicians the sense that we can wait to take any action at all.  "It’s always five minutes to midnight, and that is highly problematic," said Oliver Geden, social scientist and visiting fellow at the Max Planck Institute for Meteorology in Hamburg, Germany.  "Policymakers get used to it, and they think there’s always a way out."

Then on the other hand we have our resident Stable Genius, Donald Trump, who claimed two days ago that he understands everything he needs to know about climate because he has "a natural instinct for science."  To bolster this claim, he made a statement that apparently sums up the grand total of his expertise in climatology, which is that "climate goes back and forth, back and forth."  He then added, "You have scientists on both sides of it.  My uncle was a great professor at MIT for many years, Dr. John Trump.  And I didn’t talk to him about this particular subject, but... I will say that you have scientists on both sides of the picture."

It bears mention that Dr. John Trump was an electrical engineer, not a climatologist.  And Donald Trump didn't even ask him for an opinion.

So we have scientists trying like hell to get the public to see that scientific results are reliable, and people like Trump and his cronies trying to portray them as engaging in no better than guesswork and speculation (and of having an agenda).  That's why I did a serious facepalm when I read the article sent to me a few days ago by a friend and frequent contributor to Skeptophilia, Andrew Butters, author and blogger over at Potato Chip Math (which you should all check out because it's awesome).

This article, which appeared over at CBC, comes from a different realm of science -- medical research.  It references a paper authored by Min Qi Wang, Alice F. Yan, and Ralph V. Katz that appeared in Annals of Internal Medicine, titled, "Researcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians."

If the title isn't alarming enough by itself, take a look at what Wang et al. found:

Inappropriate analysis and reporting of biomedical research remain a problem despite advances in statistical methods and efforts to educate researchers...  [Among] 522 consulting biostatisticians... (t)he 4 most frequently reported inappropriate requests rated as “most severe” by at least 20% of the respondents were, in order of frequency, removing or altering some data records to better support the research hypothesis; interpreting the statistical findings on the basis of expectation, not actual results; not reporting the presence of key missing data that might bias the results; and ignoring violations of assumptions that would change results from positive to negative.  These requests were reported most often by younger biostatisticians.

The good news is that a lot of the biostatisticians reported refusing the requests to alter the data.  (Of course, given that this is self-reporting, you have to wonder how many would voluntarily say, "Yeah, I do that all the time.")

"I feel like I've been asked to do quite a few of these at least once," said Andrew Althouse, biostatistician at the University of Pittsburgh.  "I do my best to stand my ground and I've never falsified data....  I was once pressured by a surgeon to provide data on 10-year survival rates after a particular surgical intervention.  The problem — the 10-year data didn't exist because the hospital hadn't been using the procedure long enough...  The surgeon argued with me that it was really important and pleaded with me to find some way to do this.  He eventually relented, but it was one of the most jarring examples I've experienced."

[Image is in the Public Domain]

McGill University bioethicist Jonathan Kimmelman is among those who are appalled by this finding.  "If statisticians are saying no, that's great," he said.  "But to me this is still a major concern...  Everyone has had papers that are turned down by journals because your results were not statistically significant.  Getting tenure, getting pay raises, all sorts of things depend on getting into those journals so there is really strong incentives for people to fudge or shape their findings in a way that it makes it more palatable for those journals.  And what that shows is that there are lots of instances where there is threat of adulteration of the evidence that we use."

It's not surprising that, being human, scientists are prone to the same foibles and pitfalls as the rest of us.  However, you'd think that if you go into science, it's because you have a powerful commitment to the truth.  As Kimmelman says, the stakes are high -- not only prestige, but grant money.  Still, one would hope ethics would win over expediency.

And this is a particularly pivotal moment, when we have an administration that is deeply in the pockets of the corporations, and has shown a complete disregard for scientific findings and the opinions of experts.  The last thing we need is to give them more ammunition for claiming that science is unreliable.

But it's still a good thing, really, that Wang et al. have done this study.  You can't fix a problem when you don't know anything about it.  (Which is a truism Trump could learn from.  "Climate goes back and forth, back and forth," my ass.)  It's to be hoped that this will lead to better oversight of statistical analysis and a more stringent criterion during peer review.  Re-establishing the public trust in scientists is absolutely critical.  Our lives, and the long-term habitability of the Earth, could depend on it.

 ***********************************

This week's Skeptophilia book recommendation is something everyone should read.  Jonathan Haidt is an ethicist who has been studying the connections between morality and politics for twenty-five years, and whose contribution to our understanding of our own motives is second to none.  In The Righteous Mind: Why Good People are Divided by Politics, he looks at what motivates liberals and conservatives -- and how good, moral people can look at the same issues and come to opposite conclusions.

His extraordinarily deft touch for asking us to reconsider our own ethical foundations, without either being overtly partisan or accepting truly immoral stances and behaviors, is a needed breath of fresh air in these fractious times.  He is somehow able to walk that line of evaluating our own behavior clearly and dispassionately, and holding a mirror up to some of our most deep-seated drives.

[If you purchase the book from Amazon using the image/link below, part of the proceeds goes to supporting Skeptophilia!]