Research article written by Scott D. Lipscomb and Eugene M. Kim studying the correlation of audiovisual attributes:
Monthly Archives: July 2013
Project Preview
Current project in-progress. Implying motion to solid shapes. Computer generated rhythms.
Audiovisual piece to be posted soon.
Pierre Schaeffer’s contribution to Audiovisual Theory
Link to an article that documents the work of Pierre Schaeffer, and describes his ideas that informed an unofficial audiovisual theory. He discusses how music and realistic, natural sounds combine differently with visual events, and also the idea that synchronous auditory and visual events can have the same effect as additive synthesis.
Drone 3 (Chinese Dream)
Ambient soundtrack set to Super8 footage shot in China, Berlin and the UK.
These visuals were created in Max/MSP/Jitter and make use of multiple feedback loops to create a blurry, ethereal effect. A small amount of saturation was also added, which is controlled by the amplitude of the audio to add occasional vibrant glimpses of colour.
Projection Mapping Effects
Video demonstrating a number of different effects especially programmed to be mapped onto a specific 3D structure. These effects intend to explore the inside of a solid object and the area surrounding it by using video to invoke movement onto two faces of the object. The accompanying audio consists of sound effects that aim to reinforce the motions of each effect.
Visual effects were mostly programmed in Max/MSP/Jitter, with additional visual content created in After Effects. The projection mapping software used was VPT 6.0. The audio was recorded and produced in Ableton.
Visual Illusion Induced by Sound
The interconnectivity of our auditory and visual senses is an important aspect to consider when composing audiovisual material, as the simultaneous perception of both auditory and visual events can result in one sense changing and manipulating the other. Research undertaken to investigate the affect and influence each sense can exert on the other highlights interesting aspects of the relationship between auditory and visual phenomena.
The human visual system perceives events in three-dimensional space, selecting, identifying and obtaining information regarding an object and its movement. The eye is more spatially aware of events and activity occurring in the surrounding environment (Chion, 1994, p. 11). In contrast, the human hearing system can only provide limited information concerning the spatial aspects of an environment, instead being described by Salter (2011, p. 205) as having a ‘higher temporal resolution than the eye’. This description is supported by Chion (1994, p. 10), who explains that the hearing system processes information faster than the eye. Although these differences exist, Salter (2011, p. 205) argues that perception is seen as co-structuration, with sensorial events being created by the instantaneous connection between bodily, cognitive and environmental stimuli.
Research focused directly on the interconnectivity of auditory and visual senses has proven that the perceived intensity of a visual stimulus is enhanced in the presence of sound, showing that perceiving auditory and visual material simultaneously generates a more effective experience for the viewer.
Once such experiment that emphasizes the interconnectivity of auditory and visual perceptions, and the way one sense can influence and manipulate the other, has been carried out by Shams et al. (2002). This research explored how the perception of visual objects can be influenced by an auditory stimulus. Test subjects were presented with a flashing white circle on a computer screen and a number of audible beeps, and were asked to count the amount of flashes that they saw. When multiple beeps accompanied a single visual flash, subjects reported seeing multiple flashes. The correct number of flashes could be perceived once the time difference between each stimulus exceeded 70ms, thus exposing the temporal limits of this effect.
The results of this experiment prove that auditory stimuli can have a significant influence over the sense of sight, and, through an illusion, actually add to what we visually perceive. This research also demonstrates the interconnectivity of each perception and the ability of sound to influence and enhance vision, and could be manipulated for the creation of complex, perceptually challenging audiovisual compositions.
Link to PDF download of research article:
Corner Projection
Test projections onto a rather elaborate corner of a room:
The Audiovisual Experience
Pierre Schaeffer stated that linking music with an image is more complicated than linking a single sound to an image, as there is no cause and effect relationship between what is seen and what is heard1. If a simple sound is paired with its accompanying visual cause, there is an obvious basis in reality for the two stimuli to become fused together. However, with music, the realistic joining of audio and visual elements does not occur, instead the two are fused together in a wholly different way. The relationship between music and the image has a much stronger basis in subjective emotional and aesthetic responses than in realistic objectivity, causing the simultaneous perception of musical audio and imagery to create a whole new experience entirely; the audiovisual experience. Read More
Representing Music with Visual Space and Colour
Article by Ryan McGee discussing correlation of auditory and visual attributes:
Visual space has been accepted as an analog to musical pitch. The most obvious example of this relationship may be reading a musical score. As the positions of the notes move vertically up or down over time, the musician responds by playing pitches of ascending or descending frequency. Now, consider auditory space. In traditional music scores there may be placement of the performers, but there is no notation or consideration for the spatialization of sound. However, spatialization is a dimension used heavily by composers of computer and electronic music. If visual space relates to musical pitch, then what is a visual representation for auditory space? Do visual spatial movements relate to auditory space as well as these movements relate to pitch? Does visual color relate to pitch as strongly as visual space? Answers to these questions are not only relevant to music but also impact the fields of sonification and visual music. Read More
Early Projection Mapping Tests
Messing around with some projection mapping:
and:
Third Eye 