I know that this comes from the fact that they're frustrated, too -- they worked hard, and didn't do as well as they wanted to. My fear, though, is that they'll decide that the converse must be true -- that if they don't study, they'll do better. I always try to point out to them that they're comparing their performance on two different tests over different material taken at two different times. In other words, that it isn't a controlled experiment, and they shouldn't use this observation of a single data point as a reason not to study.
Not sure how well the argument works, or if at that point it'd be better just to say, "I know, it sucks when you don't do well," and let it go.
It's a problematic way of thinking, though, and one I can sum up as "the plural of anecdote is not data." Just because something worked (or didn't) for you, once, doesn't mean that you've discovered a correlation.
Take, for example, the latest goofy alt-med approach to avoiding skin damage in the sun: "edible sunscreen." I'm not making this up. In fact, there was a piece in The New York Times about it just day before yesterday. The idea is that you consume a flavored drink ("UVO" is one brand) that contains vitamins and minerals and antioxidants and various other kinds of alt-med fairy dust, and it's supposed to prevent sunburn.
[image courtesy of the Wikimedia Commons]
What makes this, and the New York Times article, interesting from a critical thinking perspective isn't the woo-woo magic drink phenomenon. Heaven knows there are enough completely evidence-free alt-med cures and preventatives out there; if I devoted this blog to that topic only, I'd never run out of material (although I might well run out of readers). What is most interesting is that Alyson Krueger, the author of the article, interviewed Scott Kyle, a 53-year-old competitive sailor from California, who is a devotee of UVO. Kyle said that when he uses UVO, he doesn't burn. "Normally I would be sunburned for a couple of days, and now I'm not," Kyle said. "I give it to other sailors, and they notice a difference... I totally don't care what doctors think, because it works on me."
Which is, in spirit if not in detail, exactly what my students are saying when they claim that studying simply doesn't work because they tanked on one quiz. But in the United States, we've been trained to listen to folksy anecdote and somehow give it the same weight (or higher!) than controlled scientific experiment. In fact, we've been taught to ignore scientific experiment altogether; who knows what those pointy-headed lab-coat-wearing guys are up to, and what their real motives are?
Look, it's not that I like sunscreen myself. I'm the guy who wears as little clothing as is legally permissible when the weather is warm, and my wife has been known to sneak up behind me and squirt sunscreen on my shoulders when I'm not looking. But at the same time, I know that sunburn isn't good for you, that sunscreen is a good idea, and that skin cancer sucks. I'm choosing to ignore a risk (when I can get away with it), not claiming that I'm somehow reducing my risk with the equivalent of medical voodoo.
So sorry, Mr. Kyle, but your "data" doesn't really matter. Your report of "not as sunburned," collected on different days under differing conditions, represents an anecdote at best. If UVO is shown to be efficacious -- which, as a guy with a pretty good background in biology, I sort of doubt -- it will come from well-controlled research, not the musings of a guy who is already convinced.
There's this stuff you use on babies to help them when they are teething. My wife and I, at wits end with our second child, tried it and it worked. Now, I'm 100% certain it wasn't the homeopathy that was helping, but something about the solution that did the trick. Maybe it was an "inactive" ingredient, maybe it was the consistency of the solution, maybe it was something in the non-recyclable plastic administering tube, maybe it was the temperature of the liquid, I really couldn't say because we didn't run an experiment. Which is I guess the point (which I'll finally get to). When you're under fire you're not really concerned with the caliber of the bullets, you just want the shooting to stop. In other words, desperate people will try anything, and so long as they're getting what they want in the end they will ask very few questions. Naturally, when it doesn't work out they'll start to wonder what happened and look for any reason that doesn't involve them to explain it (because being wrong doesn't feel as good).
ReplyDelete