Reactive Computing Initiative




David L. Wessel (PI)
Director, Center for New Music and Audio Technologies (CNMAT)
UC Berkeley Professor of Music, Affiliate Professor of Psychology
1750 Arch StreetBerkeley, California 94709
Telephone: (510) 643-9990 ext 302
FAX: (510) 642-7918


Edward Ashford Lee (Co-PI)
Director, The Ptolemy Project
UC Berkeley Professor of EECS
University of California
518 Cory Hall
Berkeley, California 94720
Telephone: (510) 642-0455
FAX: (510) 642-2739


Keywords: reactive, bounded latency, operating systems, programming tools, multimodal user interfaces, interactive multimedia, concurrency, high performance computing

Problem Area

This multidisciplinary initiative addresses an important, broad class of applications poorly served by existing high performance computer and communications systems: applications that require responsive, predictable-latency computation including:

These applications require reliable, real-time performance compatible with intimate human interaction. A typical performance goal for such systems would be a gesture/response latency of 10+/-1mS.

Central Focus

Many of the above applications were enabled by rapid advances in communications and computer performance. Ironically many techniques used to achieve high throughput, e.g., parallelism and buffering, conflict with the goal of bounded latencies. It is actually harder to build reactive systems today than it was a decade ago. This trend can be reversed by the coordinated, multidisciplinary effort outlined in this proposal. The goal of this project is to develop new tools, algorithms and educational programs that enable reactive systems development on mainstream computing platforms. Achieving this for a diverse range of applications will require coordinated progress on several fronts:



Despite decades of human factors research demonstrating the importance of bounded and predictable latencies, uncontrolled latency is a pervasive feature of many "state of the art" systems with which we interact, e.g.,

In the above examples, poor control of latency results in user irritation and discomfort, and economic loss. More serious, life threatening consequences are associated with telesurgery and "fly by wire" applications.

Project Plan Outline

Year 1: Evaluation

Latency-oriented Benchmark Suite

The aim of this suite is to define a set of measurable performance goals appropriate to common applications of reactive systems. A three-way multidisciplinary effort between experts in each application area (from UC Berkeley EECS dept.), the benchmark suite developers (CNMAT), and human perception and cognition experts (UC Berkeley Psychology dept., Professor Hafter).

Latency Testing, Measurement and Fault Isolation

"In vivo" measurement of latencies in most computing and networking systems is challenging. Even in the rare systems where necessary time stamping is available, most logging techniques interfere with the performance of the system being evaluated. Also, many of the latencies to be characterized are hardware buffers inaccessible to software, e.g. FIR filter delay in audio codec reconstruction filters. A hardware approach is required to address this problem.

Central to all subsequent phases of this initiative is the early development of a flexible hardware-oriented latency measurement and test system. Timing analysis is standard practice for electrical engineers who use specialized test equipment such as logic analyzers and storage oscilloscopes. Configuration of such tools to successfully log the diverse signals found in current computer systems is beyond the reach of most software developers. These developers are the primary target of this initiative. A new specialized tool is required. This testing tool will require three components:

  1. flexible signal conditioning hardware to interface audio, video, Ethernet, fast Ethernet, serial ports and other signals to the data logging system
  2. a data logging system
  3. a computer interface and analysis software

To be the basis of standard practice in reactive systems development, this tool has to be affordable: comparable in price to good profiling tools in CASE systems. CNMAT researchers are exploring the novel strategy of modulating events as digital audio signals that can be cheaply logged on digital audio tape recorders. A three way, multidisciplinary collaboration is therefore planned between CASE experts in the Computer Science department, audio engineers at CNMAT and hardware specialists in the Electrical Engineering department.

Robust, Temporal Scheduling in Mainstream Operating Systems

Existing reactive systems development tools, including the UC Berkeley EE department's Ptolemy project, target embedded systems which run a single, fixed application. Such systems are

well characterized. Hardware-related latencies are known or specified in the hardware/software codesign process. Broad application of reactive systems programming requires progress on a complementary approach where the target applications environments are workstations and personal computers (PC's).

Critical system behavior of such systems is not known a priori as it is for embedded systems. A major contribution of this initiative will be the re-targeting of the considerable corpus of successful research on reactive systems tools to readily available computer platforms.

The first phase of this effort is a thorough characterization of such platforms from the reactive systems standpoint. This will require both the aforementioned hardware measurement system and specialized software profiling tools adapted to a wide variety of operating systems and hardware configurations.

Operating systems to be evaluated include:

Operating Systems Drivers

Plug-in cards and their associated driver software are critical customization elements for many reactive systems applications, e.g., data acquisition boards and sensors for medical and music

synthesis applications. Many of the road-blocks to reactive systems development lie in the driver software for these boards. For example, the primary focus in drivers for data acquisition boards is reliable transfer of data to disk for later analysis. Buffering and process priority strategies directly conflict with the needs of reactive systems applications. A serious evaluation of these boards and drivers in their operating environment is a prerequisite to developing new drivers and new recommendations for future board and driver developers.

Robust, Temporal Scheduling in Programming Languages

In the last decade the computer science research community has identified key flaws in the temporal semantics of the few commercially used programming languages with temporal primitives such as: Ada (compiled), Java (interpreted) and NewtonScript (dynamic). Flaws include insufficient primitives (Java) and indeterminacy (Ada). New languages have been developed with solid temporal semantics including Esterel, an imperative language, and Lustre and Signal that use the dataflow approach. CNMAT has considerable experience with MAX, a component-based, visual, data flow language for reactive multimedia applications, especially computer music. The industrial programming community will not switch overnight from their dominant choices, C++ and Java to these new languages, and for good reason: these experimental languages need considerable further maturity to achieve mainstream status. A particularly important area that needs to be addressed is code generation for mainstream operating systems and hardware. A key question for the evaluation phase of this initiative is: what operating system primitives are required to effectively support these languages? Equally important is a complementary question: what can be done to existing mainstream languages so they can address reactive systems applications?

A recent focus of work both at CNMAT and in the Ptolemy project is how to integrate reactive systems constraints into object-oriented, component-based and dynamic programming paradigms. Progress in this area will require combined efforts from specialists in the areas of programming language, parallel computing, and operating systems.

Robust, Temporal Debugging Tools

The basis of software debugging tools - breakpoints by insertion of software interrupt code - is not viable for reactive systems debugging. Insertion of debugging code changes execution time and therefore the temporal semantics of the program being debugged. Another difficulty is that although execution can be suspended or delayed at a breakpoint, external stimuli from the real world cannot be. A reactive program debugging tool will have to use a different approach, one already developed in the embedded systems and hardware development community based on conditional triggering, state logging, and graphical display of temporal state evolution.

The evaluation stage of this initiative will focus on these areas: operating systems support for temporal constraint triggering, e.g., time tagging, logging strategies to minimize disturbance of reactive systems properties and integration of the hardware-based latency measurement and test system built at the outset of the initiative.

Bounded and Minimum Latency Algorithms

Most software developed today builds on large libraries implementing well-understood and commonly used algorithms, e.g., mathematical functions, sorting, searching, and iteration. Library implementations are not well suited to the needs of reactive systems. For example, the primary goal for mathematical function implementations is the maximization of arithmetic precision over the entire input domain of the function. Many reactive systems applications can trade precision and input domain size for lower latency. In sorting algorithms it is common to trade average case performance to obtain better worst case performance. A multidisciplinary team comprising implementors and algorithm experts such as Professor Kahan, will focus on the particularly important areas of floating point arithmetic, memory allocation and incremental garbage collection.

Year 2: Integration, Implementation

The goal of this stage of the initiative is to integrate what is learnt using the evaluation tools developed in the first year into compelling new reactive computing development tools. An important strategy for the success of the final year and the educational objectives of the project will be to use these tools to build a significant testbed application on a solid computing platform. It is more important to demonstrate a complete, compelling and working development tool than to work through all the implementation issues for the large number of applications and operating systems evaluated in the first phase of the initiative. The reactive systems application area will be music and audio sound synthesis and processing, the major applications focus of CNMAT. This area relies heavily on digital signal processing, a major focus of the Ptolemy group, the other major partner in this multidisciplinary venture. This application area has particularly stringent reactive requirements and failure to meet them is audibly obvious. The public appeal of this area will facilitate the educational work of the last phase of the project.

Year 3: Education


Broad publication, education and distribution of the results of this initiative is planned for the final year.


Educational materials and an associated laboratory will be created for the following courses:

  1. Undergraduate Reactive Systems Programming
  2. Multimedia Systems Programming (collaboration with Professor Rowe)
  3. Music and Audio Systems Programming (CNMAT and Professor Lee)

Additionally, a reactive systems graduate level seminar would be created. Course materials and laboratory designs would be made available to other teaching centers. These classes would be taught in a semester within this last year.


An international conference on reactive systems will be hosted towards the end of the initiative. This would include collaboration with the ACM and IEEE organizations.


An industry/university reactive systems consortium will be announced to promote ongoing work in the area, to accelerate transfer of technology from the academic to the university sector and pursue relevant standards such as the latency benchmarks, and new operating system and programming language primitives.

Estimated First Year Budget

A. Senior personnel

1. Faculty I (PI) $10k/mo. x 3 (summer salary) 30,000
2. Faculty II (Co-PI) $10k/mo. x 3 (summer salary) 30,000

B. Other personnel

2. Other professionals technician, programmer 80,000

3. Grad students $36k/yr x 5 180,000

5. Secretarial 50% of 40k/yr 20,000

D. Equipment 50,000

E. Travel 9,000

G. Other direct costs

1. Materials and supplies 1,000
2. Publication costs 1,000
6. Other Conference 15,000

H. Total direct costs 416,000

Indirect costs (Direct - equipment) x 49.9% 182,634

TOTAL $598,634

Names of individuals with whom PI or CO-PI's have had close relationship in the last 48 months:


David Wessel (PI)




Edward Lee (Co-PI)




Biographical Sketch: David Wessel (PI)



David L. Wessel
Director, Center for New Music and Audio Technologies (CNMAT)
UC Berkeley Professor of Music, Affiliate Professor of Psychology
1750 Arch Street
Berkeley, California 94709
Telephone: 510-643-9990 ext 302



1988 - to present UC Berkeley Professor of Music
1985 - 87 Conservatoire National Superieur de Musique de Paris
1979 - 88 IRCAM, Paris France Director of Pedagogy, Director of

Software Development for personal computers. Research on real-time interactive performance systems.

1974 - 79 Michigan State University Associate Professor of Psychology
1969 - 74 Michigan State University Assistant Professor of Psychology
1972 Stanford University PhD Mathematical Psychology
1964 University of Illinois BS Mathematical Statistics


Research Interests: Interactive high performance real-time computer music systems, music perception and cognition, representation of musical material, gesture sensing and controllers.


B. Publications


Five publications most closely related to the proposed project:


Wessel, D. Improvisation with Highly Interactive

Real-Time Performance Systems, Proceedings of the 17th

International Computer Music Conference, International Computer

Music Association, 1991.

Freed, A., Zicarelli, D. & Wessel, D. MAX Objects for

Media Integration, Proceedings of the 17th International

Computer Music Conference 1991, International Computer Music

Association, 1991.

Wessel, D. Let's Develop a Common Language for Synth

Programming, Electronic Musician, August, 1991.

Wessel, D., Lavoie, P., Boynton, L., & Orlarey, Y.

"MIDI-LISP: A LISP based programming environment for MIDI on

the Macintosh". Music and Digital Technology Strawn, J. (ed)

Audio Engineering Society Press, New York, 1987, 185-197.

Wessel, D. L., "Timbre space as a musical control

structure". Foundations of Computer Music, Roads, C. &

Strawn, J. (eds). M.I.T. Press, 1985, 640-657.


Five other significant publications:


Risset, J. C. & Wessel, D. L., "Exploration of timbre

by analysis and synthesis (2nd edition)". The Psychology of

Music 2nd edition, Deutsch, D. (ed). Academic Press, 1997 (in


McMillen, K., Wessel, D., & Wright, M. ZIPI's Music

Parameter Description Language, Computer Music Journal , 18(4),

52-73, 1994.

Lee, M., Freed, A. & Wessel, D. Neural Networks for

Simultaneous Classification and Parameter Estimation in Musical

Instrument Control, Proceedings SPIE Conf. on Robotics and

Intelligent Systems, May 1992.

Lee, M. & Wessel, D. Connectionist Models for Real-Time

Control of Synthesis and Compositional Algorithms, Proceedings

of the 18th

International Computer Music Conference 1992, International

Computer Music Association, 1992.

Wessel, D. Instruments That Learn, Refined Controllers,

and Source Model Loudspeakers, Computer Music Journal, Vol 15,

No. 4, Winter 1991, 82-86 MIT Press, 1991.


U.S. Patent Application Serial No. 08/551,890 Filed: October 23, 1995

Control Structure for Sound Synthesis


Musical works involving reactive computer systems:



Go Where? (1983-85) Japan Victor VDR-1026.

Antony (1977), Wergo (WER 2030-2) 1992.

Situations I (1993) , Lovely Music 1997


Recent Works/Performances:

Ngai Pha Yul, IRCAM, 1996 and New York, 1997.

Dangu Duo, IRCAM and CNMAT, 1996.

Situations II, Guanajuato, Mexico, 1995 .

Situations I, Korea, 1993

Interplay, San Francisco, 1993


C. Other collaborators

Miller Puckette, UC San Diego

Pierre Boulez

John Wawrzynek, UC Berkeley

Roger Reynolds, UC San Diego

Edward Lee, UC Berkeley


D. Graduate Students (past five years)


Pre-doctoral students supervised:

Joseph Rovan, expected 1998

Vijay Iyer, expected 1998

Rafael A. Irizarry, expected 1998

Paul Debevec, 1997

Tod Hodes, 1997

Krste Asanovic, 1996

Tom Parks, 1995

George Hajdu, 1995

Lynn Plemondon, 1995

Thorsten Helmut von Eicken, 1993

Joanna Weber, 1993

Kourosh Saberi, 1993

David Zicarelli, 1993

Ramesh Govindan, 1992


Post-doctoral scholars supervised:

Sophie Donnadieu, 1997-98

Cyril Drame , 1996-97

Pierre Korzilius, 1995-97

Givoanni Muller, 1993-94

Greg Sandell, 1991-93

Xavier Rodet, 1991- 93



Graduate students advised: 20

Postdoctoral students sponsored: 6


E. Prof. Wessel's advisors:

William K. Estes

Richard C. Atkinson

Patrick Suppes





Edward Ashford Lee

EECS Department

University of California, 518 Cory Hall, Berkeley, California 94720

phone: (510) 642-0455, fax: (510) 642-2739,


RESEARCH INTERESTS: Signal processing, parallel computation,

real-time software, digital communication



UC Berkeley, Ph.D. in Electrical Engineering, December, 1986

Major field of study: Signal Processing, Minor fields of study:

Computer Science, Applied Mathematics

Dissertation title: A Coupled Hardware and Software Architecture

for Programmable Digital Signal Processors


MIT, SM in Electrical Engineering, December 1981


Yale University, BS, May 1979

Double Major: Computer Science and Engineering and Applied

Science, Magna cum Laude



UC Berkeley: Associate Professor, July 1990-present; Professor,

June 1986-July 1990; Graduate Student Researcher, September

1982-June 1986


Co-director: Ptolemy Project. January 1990- present.

Ptolemy is a system-level design framework with ambitious

objectives to include practically all aspects of designing

signal processing and communications systems.


Director:Gabriel Project. September 1986- January 1990.

Gabriel is a software environment for rapid prototyping of

real-time implementations of signal processing applications on

parallel programmable processors.


B. Publications


Closely related to the proposed project:


W.-T. Chang, S.-H. Ha, and E. A. Lee, "Heterogeneous

Simulation - Mixing Discrete-Event Models with Dataflow,"

invited paper, to appear, RASSP special issue of the Journal on

VLSI Signal Processing, January, 1997.

E. A. Lee, "A Denotational Semantics for Dataflow with

Firing," Memorandum UCB/ERL M97/3, Electronics Research

Laboratory, U. C. Berkeley, January 1997.

S. Bhattacharyya, S. Sriram, and E. A. Lee,

"Latency-Constrained Resynchronization for Multiprocessor DSP

Implementation," Proc. ASAP Conference, Chicago, August 19-21,


A. Lee, "Computing and Signal Processing: An

Experimental Multidisciplinary Course", Proc. of IEEE Int. Conf.

on Acoustics, Speech, and Signal Processing, vol. VI, pp. 45-48,

Adelaide, Australia, April, 1994.

M. Parks and E. A. Lee, "Non Preemptive Real-Time

Scheduling of Dataflow Systems," in Proc. of IEEE Int. Conf. on

Acoustics, Speech, and Signal Processing, Detroit, MI, May 8-12,

1995, pp. 3235-3238.

J. T. Buck, S. Ha, E. A. Lee, and D. G. Messerschmitt,

"Ptolemy: A Framework for Simulating and Prototyping

Heterogeneous Systems," Int. Journal of Computer Simulation,

special issue on "Simulation Software Development," vol. 4, pp.

155-182, April, 1994. .


Other significant publications:


Lapsley, J. Bier, A. Shoham, and E. A. Lee, DSP

Processor Fundamentals - Architectures and Features, published

by Berkeley Design Technology, Inc., 39355, California St.,

Suite 206, Fremont, CA, 1996.

R. Barry, E. A. Lee, and D. G. Messerschmitt, "Capacity

Penalty Due to Ideal Zero-Forcing Decision-Feedback

Equalization," IEEE Trans. on Information Theory. Vol. 42, No.

4, pp. 1062-1071, July 1996.

Chiang, B. L. Evans, W. T. Huang, F. Covet, E. A. Lee,

H. J. Reeky, D. G. Messerschmitt, and S. Sister, "Real-Time DSP

for Sophomores," in Proc. IEEE Int. Conf. on Acoustics, Speech,

and Signal Processing, Atlanta, GA, May 7-10, 1996, vol. 2, pp.


W.-T. Chang, A. Kalavade, and E. A. Lee, "Effective

Heterogeneous Design and Cosimulation,", chapter in

Hardware/Software Co-design, G. DeMicheli and M. Sami, eds.,

NATO ASI Series Vol. 310, Kluwer Academic Publishers, 1996. Also

presented at NATO Advanced Study Institute Workshop on

Hardware/Software Codesign, Lake Como, Italy, June 18 - 30,


E. A. Lee and D. G. Messerschmitt, Digital

Communication, Second Edition, Kluwer Academic Press, Norwood,

Mass, 1994.


C. Other Collaborators

David Wessel

B.L. Evans

J.H. McLellan

M.A. Yoder

R.H. Bamberger

A. Sangiovanni-Vincentelli


D. GRADUATE STUDENTS (past 5 years)


John Barry,1992.

Gilbert Sih, 1991.

Ho-Ping Tseng, 1990.


Brian Link, December, 1992.

Gregory Walter, 1992.

Phil Lapsley, 1991.

Asawaree Kalavade, 1991.

Shuvra Bhattacharyya, 1991.


Total number of graduate students advised: 8

Postdoctoral students sponsored: 3



E. Advisor: Prof. David Messerschmitt