I've been interested for a long while in the interplay between our senses -- how all of our sensory organs work together to create our perceptual world. A simple example -- at least to describe, if not to explain completely -- is proprioception, which is our sense of the position of our bodies. If you ask your typical biology-teacher-type to explain how we sense what direction is up and what direction is down, (s)he will probably tell you that it's the semicircular canals of the inner ear, that act a bit like a carpenter's level to sense the pull of gravity. This is the organ that gets fouled up when you spin around and become dizzy, and it's skew signals from the semicircular canals that seem to be at fault in people who get motion sickness.
The truth, of course, is more complicated. You achieve proprioception not just using your inner ears but at least two other ways -- the tactile sense (i.e. the pressure under the soles of your feet if you're standing, and against your butt if you're sitting), and your visual sense (you can see your surroundings and the position of your body relative to them). Honestly, you need all three, or your sense of balance gets pretty precarious. If you doubt this, try a simple experiment -- stand on one leg, close your eyes, and then tip your head backward, thus confounding all of your proprioceptive senses except the feeling of pressure under your foot. If you can keep your balance more than a few seconds, you're doing well. (It is strongly suggested that if you try this, you either have a spotter or else surround yourself with pillows.)
This isn't just true of proprioception. Even more surprising is the McGurk Effect, a bizarre result of the interaction between sight and hearing in determining what someone is saying. If a person with decent vision and hearing watches a friend speaking and is trying to determine what they're saying, you would think that if the eyes and ears disagree, the ears would win.
But no. In the McGurk Effect, if you are watching someone say a syllable like "va" but you see them at the same time moving their mouth as if they were saying "ba," you hear "ba." The eyes overrule the ears. It's absolutely convincing even if you know what's going on. (In the video I linked in the previous paragraph, I tried listening to the guy saying "va" and mouthing "ba" over and over, but closed my eyes on every other syllable, I heard "ba va ba va ba va." Try it.)
A paper last week in the Proceedings of the Royal Society B gives us another curious example of this. It has to do with "talking with your hands" -- something I do incessantly. In fact, one of my classes once challenged me to deliver my lecture while sitting on my hands. I lasted about two minutes before saying, in some dismay, "I can't do this."
In "Beat Gestures Influence Which Speech Sounds You Hear," by Hans Rutger Bosker and David Peeters of the Max Planck Institute for Biolinguistics, we read about how you perceive the stress difference in words that have the same sounds but a different stress pattern -- such as "disCOUNT" (the verb) and "DIScount" (the noun), or "preSENT" (verb) and "PREsent" (noun). With no visual cues, the test subjects were good at discerning what was said; unsurprising, as in English (and many other languages) stress is an important clue to meaning.
But when a video was added showing the speaker using a "beat gesture" -- the kind of strong, up-and-down motion of the arms many speakers use for emphasis -- it created something the researchers call a "manual McGurk Effect." The speaker may have said "obJECT," but if (s)he used a beat gesture on the first syllable, the listener heard "OBject.""Listeners listen not only with their ears, but also with their eyes," said study co-author Hans Bosker, in an interview in Science Direct. "These findings are the first to show that beat gestures influence which speech sounds you hear... Our findings also have the potential to enrich human-computer interaction and improve multimodal speech recognition systems. It seems clear that such systems should take into account more than just speech."