The law of small numbers

A few days ago, I had a perfectly dreadful day.

The events varied from the truly tragic (receiving news that a former student had died) to the bad but mundane (losing a ghostwriting job I'd been asked to do because the person I was working for turned out to be a lunatic, and had decided I was part of a conspiracy against him -- the irony of which has not escaped me) to the "I'll-probably-laugh-about-this-later-but-right-now-I'm-not" (my dog, Guinness, recovering from his recent illness, and feeling chipper enough to swipe and destroy my wife's favorite hat) to the completely banal (my computer demanding an operating system update when I was in the middle of working, tying it up for two and a half hours).

All of this brought to mind the idea of streaks of bad (or good) luck -- something that you find people so completely convinced of that it's nearly impossible to get them to break their conviction that it sometimes happens.  We've all had days when everything seems to go wrong -- when we have what my dad used to call "the reverse Midas touch -- everything you touch turns to shit."  There are also, regrettably fewer, days when we seem to have inordinate good fortune.  My question of the day is: is there something to this?

Of course, regular readers of this blog are already anticipating that I'll answer "no."  There are actually three reasons to discount this phenomenon.  Two have already been the subjects of previous blog posts, so I'll only mention them in brief.

One is the fact that the human brain is wired to detect patterns.  We tend to take whatever we perceive and try to fit it into an understandable whole.  So when several things go wrong in a row -- even when, as with my experiences last week, they are entirely unrelated occurrences -- we try to make them into a pattern.

The second is confirmation bias -- the tendency of humans to use insignificant pieces of evidence to support what we already believe to be true, and to ignore much bigger pieces of evidence to the contrary.  I had four bad things, of varying degrees of unpleasantness, occur one day last week.  By mid-day I had already decided, "this is going to be a bad day."  So any further events -- the computer update, for example -- only reinforced my assessment that "this day is going to suck."  Good things -- like the fact that even though our dog is back to getting into trouble, he is recovering; like the the fact that we've been enjoying the International Ceramics Congress workshops this weekend; like the fact that lovely wife brought me a glass of red wine after dinner -- get submerged under the unshakable conviction that the day was a lost cause.

It's the third one I want to consider more carefully.

I call it the Law of Small Numbers.  Simply put: in any sufficiently small data sample, you will find anomalous, and completely meaningless, patterns.

To take a simple model: let's consider flipping a fair coin.  You would expect that if you flip said coin 1000 times, you will find somewhere near 500 heads and 500 tails. On the other hand, what if you look at any particular run of, say, six flips?

[Image licensed under the Creative Commons ICMA Photos, Coin Toss (3635981474), CC BY-SA 2.0]

In any six-flip run, the statisticians tell us, all possible combinations are equally likely; a pattern of HTTHTH has exactly the same likelihood of showing up as does HHHHHH -- namely, 1/64.  The problem is that the second looks like a pattern, and the first doesn't.  And so if the second sequence is the one that actually emerges, we become progressively more amazed as head after head turns up -- because somehow, it doesn't fit our concept of the way statistics should work.  In reality, if the second pattern amazes us, the first should as well -- when the fifth coin comes up tails, we should be shouting, "omigod, this is so weird" -- but of course, the human mind doesn't work that way, so it's only the second run that seems odd.

Another thing is that in the second case, the six-flip run of all heads, when it come to the seventh flip, what will it be?  It's hard for people to shake the conviction that after six heads, the seventh is bound to be tails, or at least that tails is more likely.  In fact, the seventh flip has exactly the same likelihood of turning up heads as all the others -- 1/2.

All of this brings up how surprisingly hard it is for statisticians to model true randomness.  If a sequence of numbers (for example) is actually random, all possible combinations of two numbers, three numbers, four numbers, and so on should be equally likely.  So, if you have a truly random list of (say) ten million one-digit numbers, there is a possibility that somewhere on that list there are ten zeroes in a row.  It would look like a meaningful pattern -- but it isn't.

This is part of what makes it hard to create truly randomized multiple-choice tests.  As a former science teacher, I frequently gave my classes multiple-choice quizzes, and I tried to make sure that the correct answers were placed fairly randomly.  But apparently, there's a tendency for test writers to stick the correct answer in the middle of the list -- thus the high school student's rule of thumb, which is, "if you don't know the answer, guess 'c'."

Randomness, it would seem, is harder to detect (and create) than most people think.  And given our tendency to see patterns where there are none, we should be hesitant to decide that the stars are against us on certain days.  In fact, we should expect days where there are strings of bad (or unusually good) occurrences.  It's bound to happen.  It's just that we notice it when several bad things happen on the same day, and don't tend to notice when they're spread out, because that, somehow, "seems more random" -- when, in reality, both distributions are random.

I keep telling myself that.  But it is hard to quell what my mind keeps responding -- "thank heaven it's a new week - it's bound to be better than last week was."

Well, maybe.  I do agree with another thing my dad used to tell me: "I'd rather be an optimist who is wrong than a pessimist who is right."  I'm just hoping that the statisticians don't show up and burst my bubble.

***********************************

It's astonishing to see what the universe looks like on scales different from those we're used to.  The images of galaxies and quasars and (more recently) black holes are nothing short of awe-inspiring.  However, the microscopic realm is equally breathtaking -- which you'll find out as soon as you open the new book Micro Life: Miracles of the Microscopic World.

Assembled by a team at DK Publishers and the Smithsonian Institution, Micro Life is a compendium of photographs and artwork depicting the world of the very small, from single-celled organisms to individual fungus spores to nerve cells to the facets of a butterfly's eye.  Leafing through it generates a sense of wonder at the complexity of the microscopic, and its incredible beauty.  If you are a biology enthusiast -- or are looking for a gift for a friend who is -- this lovely book is a sure-fire winner.  You'll never look the same way at dust, pollen, algae, and a myriad of other things from the natural world that you thought you knew.

[Note: if you purchase this book using the image/link below, part of the proceeds goes to support Skeptophilia!]