Listen to the hands

I've been interested for a long while in the interplay between our senses -- how all of our sensory organs work together to create our perceptual world.  A simple example -- at least to describe, if not to explain completely -- is proprioception, which is our sense of the position of our bodies.  If you ask your typical biology-teacher-type to explain how we sense what direction is up and what direction is down, (s)he will probably tell you that it's the semicircular canals of the inner ear, that act a bit like a carpenter's level to sense the pull of gravity.  This is the organ that gets fouled up when you spin around and become dizzy, and it's skew signals from the semicircular canals that seem to be at fault in people who get motion sickness.

The truth, of course, is more complicated.  You achieve proprioception not just using your inner ears but at least two other ways -- the tactile sense (i.e. the pressure under the soles of your feet if you're standing, and against your butt if you're sitting), and your visual sense (you can see your surroundings and the position of your body relative to them).  Honestly, you need all three, or your sense of balance gets pretty precarious.  If you doubt this, try a simple experiment -- stand on one leg, close your eyes, and then tip your head backward, thus confounding all of your proprioceptive senses except the feeling of pressure under your foot.  If you can keep your balance more than a few seconds, you're doing well.  (It is strongly suggested that if you try this, you either have a spotter or else surround yourself with pillows.)

This isn't just true of proprioception.  Even more surprising is the McGurk Effect, a bizarre result of the interaction between sight and hearing in determining what someone is saying.  If a person with decent vision and hearing watches a friend speaking and is trying to determine what they're saying, you would think that if the eyes and ears disagree, the ears would win.

But no.  In the McGurk Effect, if you are watching someone say a syllable like "va" but you see them at the same time moving their mouth as if they were saying "ba," you hear "ba."  The eyes overrule the ears.  It's absolutely convincing even if you know what's going on.  (In the video I linked in the previous paragraph, I tried listening to the guy saying "va" and mouthing "ba" over and over, but closed my eyes on every other syllable, I heard "ba va ba va ba va."  Try it.)

A paper last week in the Proceedings of the Royal Society B gives us another curious example of this.  It has to do with "talking with your hands" -- something I do incessantly.  In fact, one of my classes once challenged me to deliver my lecture while sitting on my hands.  I lasted about two minutes before saying, in some dismay, "I can't do this."

[Image licensed under the Creative Commons Luisfi, 071228 human hands, CC BY-SA 3.0]

In "Beat Gestures Influence Which Speech Sounds You Hear," by Hans Rutger Bosker and David Peeters of the Max Planck Institute for Biolinguistics, we read about how you perceive the stress difference in words that have the same sounds but a different stress pattern -- such as "disCOUNT" (the verb) and "DIScount" (the noun), or "preSENT" (verb) and "PREsent" (noun).  With no visual cues, the test subjects were good at discerning what was said; unsurprising, as in English (and many other languages) stress is an important clue to meaning.

But when a video was added showing the speaker using a "beat gesture" -- the kind of strong, up-and-down motion of the arms many speakers use for emphasis -- it created something the researchers call a "manual McGurk Effect."  The speaker may have said "obJECT," but if (s)he used a beat gesture on the first syllable, the listener heard "OBject."

"Listeners listen not only with their ears, but also with their eyes," said study co-author Hans Bosker, in an interview in Science Direct. "These findings are the first to show that beat gestures influence which speech sounds you hear...  Our findings also have the potential to enrich human-computer interaction and improve multimodal speech recognition systems.  It seems clear that such systems should take into account more than just speech."

Communication is subtle and complex, which is one of the reasons why it's so hard to learn a second language fluently.  Words with the same literal definition (denotation) can carry an entirely different emotional or subtextual message (connotation), something it takes a long while for a new speaker to pick up.  (To illustrate this point, my Intro to Linguistics professor said, "Consider how your words would be taken if you told someone 'Have a nice day,' as opposed to 'I hope you manage to enjoy your next 24 hours.'")

But as the Bosker and Peeters paper shows, it's more than just the sounds and how they're interpreted.  Facial movements and gestures of the hands and arms all combine to give us information about not only the emotional content of what's said, but its actual denotative meaning.  This might lie behind why it's so common for people to misinterpret written sentences, as in emails and texts.  If the speaker was standing in front of us, we'd see (s)he was joking, or not; cheerful or angry; speaking literally or being sarcastic.  We need more information than the words alone to understand them properly -- adding another layer to the already complicated affair of perceiving and interpreting our world accurately.

************************************

Science fiction enthusiasts will undoubtedly know the classic 1973 novel by Arthur C. Clarke, Rendezvous with Rama.  In this book, Earth astronomers pick up a rapidly approaching object entering the Solar System, and quickly figure out that it's not a natural object but an alien spacecraft.  They put together a team to fly out to meet it as it zooms past -- and it turns out to be like nothing they've ever experienced.

Clarke was a master at creating alien, but completely consistent and believable, worlds, and here he also creates a mystery -- because just as if we really were to find an alien spacecraft, and had only a limited amount of time to study it as it crosses our path, we'd be left with as many questions as answers.  Rendezvous with Rama reads like a documentary -- in the middle of it, you could easily believe that Clarke was recounting a real rendezvous, not telling a story he'd made up.

In an interesting example of life imitating art, in 2017 astronomers at an observatory in Hawaii discovered an object heading our way fast enough that it has to have originated outside of our Solar System.  Called 'Oumuamua -- Hawaiian for "scout" -- it had an uncanny, if probably only superficial, resemblance to Clarke's Rama.  It is long and cylindrical, left no gas or dust plume (as a comet would), and appeared to be solid rather than a collection of rubble.  The weirdest thing to me was that backtracking its trajectory, it seems to have originated near the star Vega in the constellation Lyra -- the home of the superintelligent race that sent us a message in the fantastic movie Contact.

The strangeness of the object led some to speculate that it was the product of an extraterrestrial intelligence -- although in fairness, a team in 2019 gave their considered opinion that it wasn't, mostly because there was no sign of any kind of internal energy source or radio transmission coming from it.  A noted dissenter, though, is Harvard University Avi Loeb, who has laid out his case for 'Oumuamua's alien technological origin in his new book Extraterrestrial: The First Sign of Intelligent Life Beyond Earth.

His credentials are certainly unimpeachable, but his book is sure to create more controversy surrounding this odd visitor to the Solar System.  I won't say he convinced me -- I still tend to side with the 2019 team's conclusions, if for no other reason Carl Sagan's "Extraordinary Claims Require Extraordinary Evidence" rule-of-thumb -- but he makes a fascinating case for the defense.  If you are interested in astronomy, and especially in the question of whether we're alone in the universe, check out Loeb's book -- and let me know what you think.

[Note: if you purchase this book using the image/link below, part of the proceeds go to support Skeptophilia!]