Impact gestures and percussive sounds: Causality and cross-modal integration - Rehabilitation Institute of Chicago

Skip to Content

Fri, Oct 8

Speaker: Michael Schultz (

Title: Impact gestures and percussive sounds: Causality and cross-modal integration

Abstract:  The “ventriloquist illusion” in which puppets appear to speak is a classic example of audio-visual integration, demonstrating vision’s power to shift the auditory perception of event location. We generally take this process for granted, however it poses serious problems for those suffering from sensory integration dysfunction – such as individuals with autism.  Rather than seamless integration, their everyday experience is more akin to watching a poorly dubbed voice-over in which the lips and voices appear out-of-sync. This talk will summarize some of my recent work on audio-visual integration, discussing new collaborations exploring sensory integration disorders in children with autism and developmentally delayed populations.

These collaborations will use my previous work on a novel audio-visual illusion in which percussionists use impact gestures to control an audience’s perception of note duration.  Curiously, this finding is at odds with previous research and current theoretical models of auditory-visual integration.  Recent work has shown the perception of a causal, cross-modal link plays a key role in the explaining this exception. Impact motions integrate with sounds caused by impact events (such as the sound of a percussion instrument being struck), but not sounds that are produced by different types of events (such as air moving over a reed in the case of a clarinet).  For a demonstration of the illusion, please see a brief YouTube video hosted at

Host: Konrad Kording