The interconnectivity of our auditory and visual senses is an important aspect to consider when composing audiovisual material, as the simultaneous perception of both auditory and visual events can result in one sense changing and manipulating the other. Research undertaken to investigate the affect and influence each sense can exert on the other highlights interesting aspects of the relationship between auditory and visual phenomena.
The human visual system perceives events in three-dimensional space, selecting, identifying and obtaining information regarding an object and its movement. The eye is more spatially aware of events and activity occurring in the surrounding environment (Chion, 1994, p. 11). In contrast, the human hearing system can only provide limited information concerning the spatial aspects of an environment, instead being described by Salter (2011, p. 205) as having a ‘higher temporal resolution than the eye’. This description is supported by Chion (1994, p. 10), who explains that the hearing system processes information faster than the eye. Although these differences exist, Salter (2011, p. 205) argues that perception is seen as co-structuration, with sensorial events being created by the instantaneous connection between bodily, cognitive and environmental stimuli.
Research focused directly on the interconnectivity of auditory and visual senses has proven that the perceived intensity of a visual stimulus is enhanced in the presence of sound, showing that perceiving auditory and visual material simultaneously generates a more effective experience for the viewer.
Once such experiment that emphasizes the interconnectivity of auditory and visual perceptions, and the way one sense can influence and manipulate the other, has been carried out by Shams et al. (2002). This research explored how the perception of visual objects can be influenced by an auditory stimulus. Test subjects were presented with a flashing white circle on a computer screen and a number of audible beeps, and were asked to count the amount of flashes that they saw. When multiple beeps accompanied a single visual flash, subjects reported seeing multiple flashes. The correct number of flashes could be perceived once the time difference between each stimulus exceeded 70ms, thus exposing the temporal limits of this effect.
The results of this experiment prove that auditory stimuli can have a significant influence over the sense of sight, and, through an illusion, actually add to what we visually perceive. This research also demonstrates the interconnectivity of each perception and the ability of sound to influence and enhance vision, and could be manipulated for the creation of complex, perceptually challenging audiovisual compositions.
Link to PDF download of research article:
Download
Third Eye 