Listen to the hands

I've been interested for a long while in the interplay between our senses -- how all of our sensory organs work together to create our perceptual world.  A simple example -- at least to describe, if not to explain completely -- is proprioception, which is our sense of the position of our bodies.  If you ask your typical biology-teacher-type to explain how we sense what direction is up and what direction is down, (s)he will probably tell you that it's the semicircular canals of the inner ear, that act a bit like a carpenter's level to sense the pull of gravity.  This is the organ that gets fouled up when you spin around and become dizzy, and it's skew signals from the semicircular canals that seem to be at fault in people who get motion sickness.

The truth, of course, is more complicated.  You achieve proprioception not just using your inner ears but at least two other ways -- the tactile sense (i.e. the pressure under the soles of your feet if you're standing, and against your butt if you're sitting), and your visual sense (you can see your surroundings and the position of your body relative to them).  Honestly, you need all three, or your sense of balance gets pretty precarious.  If you doubt this, try a simple experiment -- stand on one leg, close your eyes, and then tip your head backward, thus confounding all of your proprioceptive senses except the feeling of pressure under your foot.  If you can keep your balance more than a few seconds, you're doing well.  (It is strongly suggested that if you try this, you either have a spotter or else surround yourself with pillows.)

This isn't just true of proprioception.  Even more surprising is the McGurk Effect, a bizarre result of the interaction between sight and hearing in determining what someone is saying.  If a person with decent vision and hearing watches a friend speaking and is trying to determine what they're saying, you would think that if the eyes and ears disagree, the ears would win.

But no.  In the McGurk Effect, if you are watching someone say a syllable like "va" but you see them at the same time moving their mouth as if they were saying "ba," you hear "ba."  The eyes overrule the ears.  It's absolutely convincing even if you know what's going on.  (In the video I linked in the previous paragraph, I tried listening to the guy saying "va" and mouthing "ba" over and over, but closed my eyes on every other syllable, I heard "ba va ba va ba va."  Try it.)

A paper last week in the Proceedings of the Royal Society B gives us another curious example of this.  It has to do with "talking with your hands" -- something I do incessantly.  In fact, one of my classes once challenged me to deliver my lecture while sitting on my hands.  I lasted about two minutes before saying, in some dismay, "I can't do this."

[Image licensed under the Creative Commons Luisfi, 071228 human hands, CC BY-SA 3.0]

In "Beat Gestures Influence Which Speech Sounds You Hear," by Hans Rutger Bosker and David Peeters of the Max Planck Institute for Biolinguistics, we read about how you perceive the stress difference in words that have the same sounds but a different stress pattern -- such as "disCOUNT" (the verb) and "DIScount" (the noun), or "preSENT" (verb) and "PREsent" (noun).  With no visual cues, the test subjects were good at discerning what was said; unsurprising, as in English (and many other languages) stress is an important clue to meaning.

But when a video was added showing the speaker using a "beat gesture" -- the kind of strong, up-and-down motion of the arms many speakers use for emphasis -- it created something the researchers call a "manual McGurk Effect."  The speaker may have said "obJECT," but if (s)he used a beat gesture on the first syllable, the listener heard "OBject."

"Listeners listen not only with their ears, but also with their eyes," said study co-author Hans Bosker, in an interview in Science Direct. "These findings are the first to show that beat gestures influence which speech sounds you hear...  Our findings also have the potential to enrich human-computer interaction and improve multimodal speech recognition systems.  It seems clear that such systems should take into account more than just speech."

Communication is subtle and complex, which is one of the reasons why it's so hard to learn a second language fluently.  Words with the same literal definition (denotation) can carry an entirely different emotional or subtextual message (connotation), something it takes a long while for a new speaker to pick up.  (To illustrate this point, my Intro to Linguistics professor said, "Consider how your words would be taken if you told someone 'Have a nice day,' as opposed to 'I hope you manage to enjoy your next 24 hours.'")

But as the Bosker and Peeters paper shows, it's more than just the sounds and how they're interpreted.  Facial movements and gestures of the hands and arms all combine to give us information about not only the emotional content of what's said, but its actual denotative meaning.  This might lie behind why it's so common for people to misinterpret written sentences, as in emails and texts.  If the speaker was standing in front of us, we'd see (s)he was joking, or not; cheerful or angry; speaking literally or being sarcastic.  We need more information than the words alone to understand them properly -- adding another layer to the already complicated affair of perceiving and interpreting our world accurately.

************************************

Science fiction enthusiasts will undoubtedly know the classic 1973 novel by Arthur C. Clarke, Rendezvous with Rama.  In this book, Earth astronomers pick up a rapidly approaching object entering the Solar System, and quickly figure out that it's not a natural object but an alien spacecraft.  They put together a team to fly out to meet it as it zooms past -- and it turns out to be like nothing they've ever experienced.

Clarke was a master at creating alien, but completely consistent and believable, worlds, and here he also creates a mystery -- because just as if we really were to find an alien spacecraft, and had only a limited amount of time to study it as it crosses our path, we'd be left with as many questions as answers.  Rendezvous with Rama reads like a documentary -- in the middle of it, you could easily believe that Clarke was recounting a real rendezvous, not telling a story he'd made up.

In an interesting example of life imitating art, in 2017 astronomers at an observatory in Hawaii discovered an object heading our way fast enough that it has to have originated outside of our Solar System.  Called 'Oumuamua -- Hawaiian for "scout" -- it had an uncanny, if probably only superficial, resemblance to Clarke's Rama.  It is long and cylindrical, left no gas or dust plume (as a comet would), and appeared to be solid rather than a collection of rubble.  The weirdest thing to me was that backtracking its trajectory, it seems to have originated near the star Vega in the constellation Lyra -- the home of the superintelligent race that sent us a message in the fantastic movie Contact.

The strangeness of the object led some to speculate that it was the product of an extraterrestrial intelligence -- although in fairness, a team in 2019 gave their considered opinion that it wasn't, mostly because there was no sign of any kind of internal energy source or radio transmission coming from it.  A noted dissenter, though, is Harvard University Avi Loeb, who has laid out his case for 'Oumuamua's alien technological origin in his new book Extraterrestrial: The First Sign of Intelligent Life Beyond Earth.

His credentials are certainly unimpeachable, but his book is sure to create more controversy surrounding this odd visitor to the Solar System.  I won't say he convinced me -- I still tend to side with the 2019 team's conclusions, if for no other reason Carl Sagan's "Extraordinary Claims Require Extraordinary Evidence" rule-of-thumb -- but he makes a fascinating case for the defense.  If you are interested in astronomy, and especially in the question of whether we're alone in the universe, check out Loeb's book -- and let me know what you think.

[Note: if you purchase this book using the image/link below, part of the proceeds go to support Skeptophilia!]

Talk to the hand

I used to tease my poor, long-suffering mother because of her habit of talking with her hands.

It wasn't even necessarily when the person she was talking to was physically there in the room with her.  She talked with her hands (well, with one hand) while she was on the phone.

"Mom, they can't see you," I told her.

"I know," she said in an exasperated tone.  "I can't help it."

I got my comeuppance when I started teaching, and a student -- more than one, actually -- pointed out my habit of sculpting the air while I was explaining something.  One of them challenged me to deliver a lecture with my hands clasped behind my back.

That lasted approximately two minutes, much to the class's amusement.

So Mom, if you're listening right now, you get the last laugh.  I don't know that I can explain it any better than you did, but if I sit on my hands, I become completely tongue-tied.

[Image licensed under the Creative Commons Wikimania2009 Beatrice Murch, Talking with the hands, CC BY 3.0]

Apparently it's more or less universal, and the reason may be more than just introducing expressive body language into our conversation.  A study from the University of Connecticut released this week in Proceedings of the National Academy of Sciences found that hand gestures change the quality and intonation of our voices -- therefore communicating subtle information even if the listener can't see what our hands are doing.

In "Acoustic Information About Upper Limb Movement in Voicing," by Wim Pouw, Alexandra Paxton, Steven J. Harrison, and James A. Dixon, we read about a simple experiment -- volunteers were instructed to make a continuous vowel tone ("aaaaaaa....") into a microphone, simultaneously making rhythmic motions with the arms, while listeners in another room tried to synchronize their own arm movements with those of the vocalizers.

The ability to do that was nearly universal, even though the changes in the tonal quality were very subtle.

The authors write:

Co-speech gestures, no matter what they depict, further closely coordinate with the melodic aspects of speech known as prosody.  Specifically, gesture’s salient expressions (e.g., sudden increases in acceleration or deceleration) tend to align with moments of emphasis in speech.  Recent computational models trained on associations of gesture and speech acoustics from an individual have succeeded in producing very natural-looking synthetic gestures based on novel speech acoustics from that same individual, suggesting a very tight (but person-specific) relation between prosodic–acoustic information in speech and gestural movement.  Such research dovetails with remarkable findings that speakers in conversation who cannot see and only hear each other tend to synchronize their postural sway (i.e., the slight and nearly imperceptible movement needed to keep a person upright).

Because the entire skeletomuscular system is connected, movement of the arms and hands alters the shape/position of the chest cavity and throat, creating small changes in our vocal quality.  That, apparently, is enough to convey information to our listener, even if they can't see the gestures.

"Some language researchers don't like this idea, because they want language to be all about communicating the contents of your mind, rather than the state of your body," said study co-author James Dixon. "But we think that gestures are allowing the acoustic signal to carry additional information about bodily tension and motion.  It's information of another kind."

So add this to the long list of subtleties affecting our communications.  I still remember my Intro to Linguistics professor telling us about tonal languages like Thai, in which the pitch and/or pitch changes in vocalizing a syllable change its meaning, and he asked if we thought information in English was altered by changes in tone or stress.  One person came up with the rising pitch at the end of a sentence communicating a question, and increasing stress to indicate emphasis, but that was about it.

"Really?" he said.  "Then tell me why the following all mean something different."  And he read us this list of sentences:

• She gave the money to him today?
• She gave the money to him today?
• She gave the money to him today?
• She gave the money to him today?
• She gave the money to him today?

Minor alterations in the way a sentence is uttered can entirely change the meaning of the words.

I guess I shouldn't have picked on my mom for talking with her hands.  Human communication is complex, and what we end up saying to our listeners can depend on a great many things beyond the exact words used.  So be careful how your hand moves when you're talking to a friend on the telephone.

You may be telling them more than you realize.

**********************************

This week's Skeptophilia book of the week is six years old, but more important today than it was when it was written; Richard Alley's The Two-Mile Time Machine: Ice Cores, Abrupt Climate Change, and Our Future.  Alley tackles the subject of proxy records -- indirect ways we can understand things we weren't around to see, such as the climate thousands of years ago.

The one he focuses on is the characteristics of glacial ice, deposited as snow one winter at a time, leaving behind layers much like the rings in tree trunks.  The chemistry of the ice gives us a clear picture of the global average temperature; the presence (or absence) of contaminants like pollen, windblown dust, volcanic ash, and so on tell us what else might have contributed to the climate at the time.  From that, we can develop a remarkably consistent picture of what the Earth was like, year by year, for the past ten thousand years.

What it tells us as well, though, is a little terrifying; that the climate is not immune to sudden changes.  In recent memory things have been relatively benevolent, at least on a planet-wide view, but that hasn't always been the case.  And the effect of our frantic burning of fossil fuels is leading us toward a climate precipice that there may be no way to turn back from.

The Two-Mile Time Machine should be mandatory reading for the people who are setting our climate policy -- but because that's probably a forlorn hope, it should be mandatory reading for voters.  Because the long-term habitability of the planet is what is at stake here, and we cannot afford to make a mistake.

As Richard Branson put it, "There is no Planet B."

[Note: if you purchase this book using the image/link below, part of the proceeds goes to support Skeptophilia!]