David Wessel
Center for New Music and Audio Technology (CNMAT),
Department of Music,
Univ. of California, Berkeley
1750 Arch Street
Berkeley, CA, USA
wessel@cnmat.berkeley.edu
Matthew Wright
Center for New Music and Audio Technology (CNMAT),
Department of Music,
Univ. of California, Berkeley
1750 Arch Street,
Berkeley, CA, USA
matt@cnmat.berkeley.edu
John Schott
Composer/Guitarist
john@johnschott.com
Abstract
In this demonstration we will show a variety of computer-based musical instruments designed for live performance. Our design criteria include initial ease of use coupled with a long term potential for virtuosity, minimal and low variance latency, and clear and simple strategies for programming the relationship between gesture and musical result. We present custom controllers and unique adaptations of standard gestural interfaces, a programmable connectivity processor, a communications protocol called Open Sound Control (OSC), and a variety of metaphors for musical control.
Keywords
Expressive control, mapping gestures to acoustic results, metaphors for musical control, Tactex, Buchla Thunder, digitizing tablets.
INTRODUCTION
We intend to demonstrate some of the computer-based musical instruments that we have been developing and playing in recent years . Our musical tastes run towards improvised interactive live performance, in collaboration with improvising acoustic musicians. We desire instruments that on one hand allow for fine control of subtle nuance and on the other hand can autonomously generate large quantities of musical material.
A computer-based musical instrument is a system consisting of the following components:
We will present novel work in all four of these areas.
GESTURAL INTERFACES
We desire the following features in our gestural interfaces:
Digitizing Tablets
Commercially available stylus/tablet controllers offer high-resolution absolute position sensing, detection of pressure and tilt, and data rates on the order of 100 Hz. Our custom software allows us to define regions of the tablet surface and associate musical behaviors with each region. The relatively large surface area and absolute position sensing allow for large "palettes" of musical material. The high-resolution position, pressure, and tilt detection provide for multidimensional continuous control of musical processes.
Buchla Thunder
Don Buchla’s Thunder (http://www.buchla.com) is a polyphonic continuous multidimensional interface: Thunder is a two-dimensional surface containing more than a dozen continuously pressure- and position-sensitive strips that are all sensed simultaneously.
Tactex MTC Express
Tactex’s MTC Express (http://www.tactex.com) controller senses pressure at multiple points on a surface. The primary challenge of using the device is to reduce the high dimensionality of the raw sensor output (over a hundred pressure values) to a small number of parameters that can be reliably controlled. We interpret the output of the tactile array as an "image" and use computer vision techniques to estimate position and pressure for each finger of the hand. The anatomy of the human hand makes it impossible to control these four variables independently for each of five fingers, so we have developed another level of analysis, based on geometrical analysis of a triangle formed by the positions of the fingers.
Controllers based on Custom MEM Accelerometer and Gyroscopes
We have implemented an arm motion sensing system based on a combination of microelectromechanical (MEM) accelerometers and gyroscopes (http://www.analog.com/technology/mems/index.html). These multidimensional sensors are interfaced to our computer systems with our connectivity processor.
CONNECTIVITY PROCESSOR
We have developed a connectivity processor that handles digital audio formats such as S/PDIF, AES/EBU, AES-3, and ADAT Lightpipe, MIDI, and multichannel analog I/O for both audio and synchronously sampled sub-audio rate continuous gestures. This processor can communicate multichannel audio and gestural data with a computer via the UDP protocol of TCP/IP, allowing it to connect to a portable computer via 100 BaseT Ethernet rather than PCI slots, PCMCIA, etc.
METAPHORS
Super controllers and high-bandwidth data paths do not guarantee compelling musical instruments. The success or failure of a live computer music instrument is determined by the way it maps performers’ control gestures to sound. We believe that the best mappings are informed by metaphors and describe here some of the metaphors we have found most useful.
Scrubbing
Sinusoidal, granular, and other models allow arbitrary time-scale manipulation without changes in pitch or spectral shape. We have built "scrubbing" interfaces for the tablet in which one dimension of the pen’s position on the tablet maps to the time index of a sinusoidal model. Moving the pen gradually from left to right at the appropriate rate results in a resynthesis with the original temporal behavior, but any other gesture results in an alteration of the original. This interface allows a performer to arbitrarily rearrange musical material, while preserving the fine continuous structure of the original input sounds. Other dimensions of the tablet sensing data, e.g., pressure, tilt, and vertical position, can be mapped to synthesis parameters such as loudness and spectral shape.
Drag and Drop
In the drag and drop metaphor, an object is selected, picked up, and dropped upon a process. We have used this metaphor with tablets; it requires a controller with a two-dimensional surface and accurate position sensing. We represent both musical material and musical processes as objects occupying areas of the surface. For example, the musical material might consist of a collection of samples, and the musical processes might be different ways of playing the samples with or without looping, pitch shifting, etc. The performer selects a musical object, drags it to the appropriate part of the surface, and drops it onto the desired musical process. This metaphor can be extended with musical objects representing other parameters of the musical processes, for example, filter shapes.
Catch and Throw
"Catch and throw" is a controller metaphor that affords the capture of live material from a performance, its analysis and transformation, and finally the reinjection of the transformed material back into the performance. In our demonstration "catch and throw" is carried out on a Wacom tablet interface. A unique representation of the past time is used to reach back into the computer’s echoic memory to set the beginning of the capture.
Dipping
In the "dipping" metaphor the computer constantly generates musical material via an algorithmic process, but this material is silent by default. The performer controls the volume and timbral material associated with each process, e.g., using a poly-point touch-sensitive interface with the pressure in each region mapped to the volume of a corresponding process. Other gestural parameters control other parameters of the musical processes.
An advantage of this metaphor is that each musical event can be precisely timed, regardless of the latency or jitter of the gestural interface. Once a given process is made audible, its rhythm is not dependent on the performer in an event-by-event way.
This kind of interface is most satisfying when there are multiple simultaneous musical processes of different timbres, allowing the performer to orchestrate the result in real-time by selecting which processes will be heard.
PRACTICAL CONSIDERATIONS
This will be a hands-on, interactive live demo of the equipment and technology we have described. The gestural interfaces, computers, and connectivity processors are reliable and portable enough to carry on an airplane from California to Ireland.
CONCLUSIONS
We demonstrate a highly portable set of three computer-based instrument systems — one for guitar effects, a second for a tablet controller, and a third for a polyphonic continuous controller and a custom arm motion sensing system based on MEM technology. We also demonstrate a connectivity processor and the use of Open Sound Control. Cognitively meaningful metaphors for control are at the foundation of our design process.
ACKNOWLEDGMENTS
Special thanks to Rimas Avizienis, Adrian Freed, and Takahiko Suzuki for their work on the connectivity processor and to Gibson Guitar, DIMI, and the France Berkeley Fund for their generous support.
REFERENCES