Color Organ statement

The Color Organ

A collaborative project, David Britton and Jack Ox are the principles with a growing list of generous contributors from both industry and high performance computing universities [1] . Britton is responsible for the graphics programming and the meta-architecture of the programming structure, while Ox contributed the concept, visual images, musical analysis, visualization systems and texture maps.

The 21st Century Virtual Color Organ is a computational system for translating musical compositions into visual performance [2] An instrument, like a musical instrument, it uses supercomputing power to produce 3D visual images and sound from MIDI files, and can play a variety of compositions. Performances take place in interactive, immersive, virtual reality environments such as the CAVE [3] , VisionDome [4] , or Immersadesk.[5] .

Because it is a 3D immersive world the Color Organ is also a place- that is, a performance space. This interactive instrument consists of three basic parts:

The first part of the analysis task is to determine what are the structural parameters of the piece of music to be visualized, or in other words, what are the operating principles and which data sets should be collected? Does the composition exist within a diatonic/chromatic harmonic frame-work, or is it composed from layers of carefully chosen timbres? These two approaches mean very different things and are therefore visually depicted with completely different color systems.
Data sets that are contained in all compositions include patterns of rising and falling melodic lines, changes in dynamics (loudness), and also the rhythmic units and patterns including the initial “attack” of the notes and their articulations. This information is encoded in MIDI files in the Color Organ.

Appropriate visual vocabularies must be found to express attributes of the music. This means that there are metaphorical and/or structural relationships between the music and the images. Images are gathered by making very high resolution photographs on location and then detailed pencil drawings in the studio, often combining three or four photographs into one large view. It is important to make information rich, well rendered drawings so that during the extensive processing to come they will hold their character and be recognizable.
The original two dimensional, hand drawn images are used by the 3D modeler, along with the original photographs, in order to re-create them in three dimensions. The entire first organ stop was modeled by Richard Rodriguez. with Ox drawing all of the texture maps with a real pencil (not a virtual one inside the computer) at a large-scale. These pictures were scanned into the computer in pieces, reassembled electronically, and applied to the surfaces of the models.
Currently in the Color Organ there is one visual environment and vocabulary which corresponds to the idea of a “stop” in a traditional organ. An organ stop is a particular voice in sound. In the future Ox will be making more 3D environments and other artists will also be invited to contribute to a growing collection of visual organ stops. For this first stop there are eight different desert landscapes gathered from real places in California and Arizona. Each one is connected to a particular family of instruments because the structure of the earth and rocks somehow mimics their sound production. Each image is itself a collection of data, containing content in both patterns of lines and colors and also the metaphorical connection to the instrument family. However, further on I will describe the “Gridjam” project. It features Alvin Curran as the first composer of the musical structure ordering the improvised collaboration of sounds and images between performers who are geographically separate although connected through the AccessGrid. In this project the eight different landscapes will be connected to “sets” of sounds instead of instrument families.

The second part of the visual vocabulary developed is based in the application of a transparent layer of color over the landscape image embedded polygons which are created over the virtual desert by the playing of MIDI files. Which information these overlaid color systems expresses is determined early in the analysis process..
The most recent color system created is based on timbre, literally meaning the color of sound. A vital component of music structured on timbre is the combination of specific, differentiated sounds, much like a painter can choose to use color. Whereas the color system Ox made based on the Circle of Fifths maps harmonic movement and quality in music which takes its structure from these concepts , the later system is effective with music whose actual sounds form the structure. She created a list of over 130 mixtures of RGB hues where families of instruments are represented by sets of colors in a graduated series, further modified by different mutes’ and/or playing techniques’ visual equivalences. In order to visually equate timbre changes in the human voice instrument a vowel/color system was created which describes timbre changes for vocal lines through an analysis of how and where vowels are made in the vocal tract. Timbre for a singer is altered by changes in the vocal tract shape which come about through changes in the vowel sound.

WWhen the performance begins the viewer/listeners are in a world of hand drawn landscapes which have been modeled into 3D. All of the landscapes are in black and white with the sky completely black. As the music plays a three dimensional colored and image embedded geometric structure takes shape in the space over the landscape. This is constructed from flat pictures of the landscape images representing the instrument families which produced them. They are colored a specific hue based on a timber analysis of which instrument is being played and what the particular playing technique is at that moment. The saturation of the color reflects changing dynamics (loud and soft). These flat strips of landscape are placed up and down in vertical space by their pitch. A higher pitch will be higher in space and a low pitch will be placed closer to the landscape below. The width of the strips is controlled by the volume (attack) of the signal. After the music has been played there remains a complete sculpture which can be further explored in an interactive way. The viewer can move at will through the space and touch elements of the sculpture and hear the sound which originally produced it.

RRecently completed is the visualization of Clarence Barlow’s “Im Januar am Nil” (1984) for chamber ensemble including soprano saxophone, clarinet/bass clarinet, drums, piano, and strings. Barlow, an internationally known composer, is also professor at the Kunsthochschule in Cologne and director of the department of Sonology at the Conservatory of Music in the Hague. he composed this piece on a computer with a melodic structure coming directly from the mathematics of a spiral. With regularly placed spokes going from the center of the spiral out, the space between the crossing points of the spikes and spiral determine the beginning of each melodic sequence. The lengths of these are extended as the turns around the enlarging spiral grow larger. For performance the strings are microtuned in order to reproduce formants naturally made by vowel sounds.
The Color Organ’s possibilities were expanded when Britton came up with the idea of having the path of time trace a more appropriate pattern than the current left to right on the x axis. He has programed the Organ to have time move in a parabolic spiral taken from the structure from which Barlow’s melody was generated algorithmically by a computer. Of course this happens in a 3D environment instead of the original 2D compositional spiral. Because the violins are actually producing vocalic phonemes, there is a separate polygon hanging perpendicular to the ground, instead of parallel as the actual musical notes are. These “vowel planes” follow along a path on the inside of the notes within the spiral. Each is covered with the vocal metaphor texture map and colored with the appropriate transparent color informed by the vowel’s sound.

Plans are currently being worked on to realize an improvised musical jam by players located at different geographical points on the AccessGrid in the US, with both the MIDI controlled 3D visualizations and synthesized sound files coming together in any connected immersive environments the Grid. This experiment will be called a “GridJam”. Presently, Boston University and the High Performance Computing Center at the University of New Mexico are collaborating with us. Real time video conferences are successfully sent over Internet2 with events, such as the Chautauqua series from the National Computational Science Alliance (NCSA) , becoming normal. . The Color Organ offers a unique application with which to develop further these communicative capabilities for Internet2.

Ox asked Alvin Curran, an internationally known composer who also is a professor at Mills College, to order the structure of the jam to be performed on the AccessGrid. He must musically deal with the one second lag of sound experienced on the Grid, but has experience creating musical events that are produced in several geographical locations simultaneously in his concerts which use the radio airwaves.

Curran has been collecting/creating a vast library of electronic sounds for many years. The sounds are MIDI files which are accessible by MIDI keyboard. In order to realize Curran’s work in the Color Organ a new metaphor based on his vocabulary and structure had to be created. In the first and still only organ stop there are eight categories of landscape 2D texture maps and 3D models. These eight images had been connected to different families of instruments in timbre based music. Now, the library of sounds which Curran sends will be sorted into eight groups, or sets, which are connected to the eight 3D landscape units and sets of texture maps. Ox had a program made in MAX by Holland Hobson to transform sound files into graphic representations and MIDI files. Ox will analyze the visual representations, group them, and create 3D polygonal shapes which reflect the analysis of the sound file. These appropriately texture mapped shapes will be what is generated by the Color Organ when a sound is played by one of the Gridjaming musicians, in a pre-assigned path..

In Ox’s system each one of the eight categories is further divided by colors, which also attached to the image. Each of these combination texture map/color will be associated with a particular sound. The metaphor here is about mathematical sets. One group of objects is being associated to another, with the connection being similar relationships between the objects in one set with another set of objects and their interior relationships. It is even more abstract than the connections to sound production in instrument groups depicted metaphorically by landscape structures..
There will be four musicians stationed geographically apart on the AccessGrid. Each will have a MIDI keyboard with two of the sets of sounds/texture maps/ color. There may be one set of sounds which would be part of a common vocabulary, to be used by all musicians. Different categories of sounds will also be assigned a different path through the virtual space and one will be able to study the virtual musical sculpture left from the real time performance and present in different nodes along the Grid. Curran’s multi pitched sounds will be located on the Z access by their location on the keyboard. The performers will have practiced together at Mills College where they are Curran’s students, and will have worked out strategies for the jam.

Hopefully there will be no final word on the Color Organ. We believe that it will be an ever expanding and progressively improving application. At the moment timbre recognition is not automatic because this is the one element not present in MIDI files. We hope to find a way to have live players performing with some kind of sensing device such as voice recognition for the vowel sounds in vocals. Also, as each new composer works with us we have to add or change the systems operating in order to accommodate the structural needs of other music. As the Color Organ has new visual organ stops added and new capabilities are programmed in, it will also be kept current in its hard and software. There is far more resolution in the texture files than can be seen in present day immersive environments. This adds considerably to the ability to update the Organ for new technology and also makes possible the selling of high quality, one of a kind stillshots.

[1] The Virtual Color Organ has been supported by the National Center for Supercomputing Applications at the U. of Ill. at Urbana-Champaign including time in their CAVE and also funds for software support. Silicon Graphics has given hardware support and EAI has given Sense8’ s World Tool Kit the program being used to develop the Virtual Color Organ. Ars Electronica provided the initial research and development money, Robert Putnam from the Scientific Computing and Visualization Group at Boston University is doing the interactive, kinetic sound placement and 3D localization. Art and Science Collaboration (ASCI). is supporting the project as the umbrella organization to receive non-profit funds. The U. of Loughborough, LUTCHI Research Centre, UK gave hands on support for the project when Jack Ox was a visiting fellow there during the summer of 2000.