http://cnmat.CNMAT.Berkeley.EDU/RSVP/
David L. Wessel (PI)
Director, Center for New Music and Audio Technologies (CNMAT)
UC Berkeley Professor of Music, Affiliate Professor of Psychology
1750 Arch StreetBerkeley, California 94709
Telephone: (510) 643-9990 ext 302
FAX: (510) 642-7918
wessel@cnmat.berkeley.edu
http://www.cnmat.berkeley.edu/
Edward Ashford Lee (Co-PI)
Director, The Ptolemy Project
UC Berkeley Professor of EECS
University of California
518 Cory Hall
Berkeley, California 94720
Telephone: (510) 642-0455
FAX: (510) 642-2739
eal@eecs.berkeley.edu
http://ptolemy.eecs.berkeley.edu/
Keywords: reactive, bounded latency, operating systems, programming tools, multimodal user interfaces, interactive multimedia, concurrency, high performance computing
This multidisciplinary initiative addresses an important, broad class of applications poorly served by existing high performance computer and communications systems: applications that require responsive, predictable-latency computation including:
These applications require reliable, real-time performance compatible with intimate human interaction. A typical performance goal for such systems would be a gesture/response latency of 10+/-1mS.
Many of the above applications were enabled by rapid advances in communications and computer performance. Ironically many techniques used to achieve high throughput, e.g., parallelism and buffering, conflict with the goal of bounded latencies. It is actually harder to build reactive systems today than it was a decade ago. This trend can be reversed by the coordinated, multidisciplinary effort outlined in this proposal. The goal of this project is to develop new tools, algorithms and educational programs that enable reactive systems development on mainstream computing platforms. Achieving this for a diverse range of applications will require coordinated progress on several fronts:
Despite decades of human factors research demonstrating the importance of bounded and predictable latencies, uncontrolled latency is a pervasive feature of many "state of the art" systems with which we interact, e.g.,
In the above examples, poor control of latency results in user irritation and discomfort, and economic loss. More serious, life threatening consequences are associated with telesurgery and "fly by wire" applications.
The aim of this suite is to define a set of measurable performance goals appropriate to common applications of reactive systems. A three-way multidisciplinary effort between experts in each application area (from UC Berkeley EECS dept.), the benchmark suite developers (CNMAT), and human perception and cognition experts (UC Berkeley Psychology dept., Professor Hafter).
"In vivo" measurement of latencies in most computing and networking systems is challenging. Even in the rare systems where necessary time stamping is available, most logging techniques interfere with the performance of the system being evaluated. Also, many of the latencies to be characterized are hardware buffers inaccessible to software, e.g. FIR filter delay in audio codec reconstruction filters. A hardware approach is required to address this problem.
Central to all subsequent phases of this initiative is the early development of a flexible hardware-oriented latency measurement and test system. Timing analysis is standard practice for electrical engineers who use specialized test equipment such as logic analyzers and storage oscilloscopes. Configuration of such tools to successfully log the diverse signals found in current computer systems is beyond the reach of most software developers. These developers are the primary target of this initiative. A new specialized tool is required. This testing tool will require three components:
To be the basis of standard practice in reactive systems development, this tool has to be affordable: comparable in price to good profiling tools in CASE systems. CNMAT researchers are exploring the novel strategy of modulating events as digital audio signals that can be cheaply logged on digital audio tape recorders. A three way, multidisciplinary collaboration is therefore planned between CASE experts in the Computer Science department, audio engineers at CNMAT and hardware specialists in the Electrical Engineering department.
Existing reactive systems development tools, including the UC Berkeley EE department's Ptolemy project, target embedded systems which run a single, fixed application. Such systems are
well characterized. Hardware-related latencies are known or specified in the hardware/software codesign process. Broad application of reactive systems programming requires progress on a complementary approach where the target applications environments are workstations and personal computers (PC's).
Critical system behavior of such systems is not known a priori as it is for embedded systems. A major contribution of this initiative will be the re-targeting of the considerable corpus of successful research on reactive systems tools to readily available computer platforms.
The first phase of this effort is a thorough characterization of such platforms from the reactive systems standpoint. This will require both the aforementioned hardware measurement system and specialized software profiling tools adapted to a wide variety of operating systems and hardware configurations.
Operating systems to be evaluated include:
Plug-in cards and their associated driver software are critical customization elements for many reactive systems applications, e.g., data acquisition boards and sensors for medical and music
synthesis applications. Many of the road-blocks to reactive systems development lie in the driver software for these boards. For example, the primary focus in drivers for data acquisition boards is reliable transfer of data to disk for later analysis. Buffering and process priority strategies directly conflict with the needs of reactive systems applications. A serious evaluation of these boards and drivers in their operating environment is a prerequisite to developing new drivers and new recommendations for future board and driver developers.
In the last decade the computer science research community has identified key flaws in the temporal semantics of the few commercially used programming languages with temporal primitives such as: Ada (compiled), Java (interpreted) and NewtonScript (dynamic). Flaws include insufficient primitives (Java) and indeterminacy (Ada). New languages have been developed with solid temporal semantics including Esterel, an imperative language, and Lustre and Signal that use the dataflow approach. CNMAT has considerable experience with MAX, a component-based, visual, data flow language for reactive multimedia applications, especially computer music. The industrial programming community will not switch overnight from their dominant choices, C++ and Java to these new languages, and for good reason: these experimental languages need considerable further maturity to achieve mainstream status. A particularly important area that needs to be addressed is code generation for mainstream operating systems and hardware. A key question for the evaluation phase of this initiative is: what operating system primitives are required to effectively support these languages? Equally important is a complementary question: what can be done to existing mainstream languages so they can address reactive systems applications?
A recent focus of work both at CNMAT and in the Ptolemy project is how to integrate reactive systems constraints into object-oriented, component-based and dynamic programming paradigms. Progress in this area will require combined efforts from specialists in the areas of programming language, parallel computing, and operating systems.
The basis of software debugging tools - breakpoints by insertion of software interrupt code - is not viable for reactive systems debugging. Insertion of debugging code changes execution time and therefore the temporal semantics of the program being debugged. Another difficulty is that although execution can be suspended or delayed at a breakpoint, external stimuli from the real world cannot be. A reactive program debugging tool will have to use a different approach, one already developed in the embedded systems and hardware development community based on conditional triggering, state logging, and graphical display of temporal state evolution.
The evaluation stage of this initiative will focus on these areas: operating systems support for temporal constraint triggering, e.g., time tagging, logging strategies to minimize disturbance of reactive systems properties and integration of the hardware-based latency measurement and test system built at the outset of the initiative.
Most software developed today builds on large libraries implementing well-understood and commonly used algorithms, e.g., mathematical functions, sorting, searching, and iteration. Library implementations are not well suited to the needs of reactive systems. For example, the primary goal for mathematical function implementations is the maximization of arithmetic precision over the entire input domain of the function. Many reactive systems applications can trade precision and input domain size for lower latency. In sorting algorithms it is common to trade average case performance to obtain better worst case performance. A multidisciplinary team comprising implementors and algorithm experts such as Professor Kahan, will focus on the particularly important areas of floating point arithmetic, memory allocation and incremental garbage collection.
The goal of this stage of the initiative is to integrate what is learnt using the evaluation tools developed in the first year into compelling new reactive computing development tools. An important strategy for the success of the final year and the educational objectives of the project will be to use these tools to build a significant testbed application on a solid computing platform. It is more important to demonstrate a complete, compelling and working development tool than to work through all the implementation issues for the large number of applications and operating systems evaluated in the first phase of the initiative. The reactive systems application area will be music and audio sound synthesis and processing, the major applications focus of CNMAT. This area relies heavily on digital signal processing, a major focus of the Ptolemy group, the other major partner in this multidisciplinary venture. This application area has particularly stringent reactive requirements and failure to meet them is audibly obvious. The public appeal of this area will facilitate the educational work of the last phase of the project.
Broad publication, education and distribution of the results of this initiative is planned for the final year.
Educational materials and an associated laboratory will be created for the following courses:
Additionally, a reactive systems graduate level seminar would be created. Course materials and laboratory designs would be made available to other teaching centers. These classes would be taught in a semester within this last year.
An international conference on reactive systems will be hosted towards the end of the initiative. This would include collaboration with the ACM and IEEE organizations.
An industry/university reactive systems consortium will be announced to promote ongoing work in the area, to accelerate transfer of technology from the academic to the university sector and pursue relevant standards such as the latency benchmarks, and new operating system and programming language primitives.
A. Senior personnel
1. Faculty I (PI) $10k/mo. x 3 (summer salary) 30,000
2. Faculty II (Co-PI) $10k/mo. x 3 (summer salary) 30,000
B. Other personnel
2. Other professionals technician, programmer 80,000
3. Grad students $36k/yr x 5 180,000
5. Secretarial 50% of 40k/yr 20,000
D. Equipment 50,000
E. Travel 9,000
G. Other direct costs
1. Materials and supplies 1,000
2. Publication costs 1,000
6. Other Conference 15,000
H. Total direct costs 416,000
Indirect costs (Direct - equipment) x 49.9% 182,634
TOTAL $598,634
A. CURRICULUM VITAE
David L. Wessel
Director, Center for New Music and Audio Technologies (CNMAT)
UC Berkeley Professor of Music, Affiliate Professor of Psychology
1750 Arch Street
Berkeley, California 94709
Telephone: 510-643-9990 ext 302
wessel@cnmat.berkeley.edu
EDUCATION AND EXPERIENCE
1988 - to present UC Berkeley Professor of Music
1985 - 87 Conservatoire National Superieur de Musique de Paris
1979 - 88 IRCAM, Paris France Director of Pedagogy, Director of
Software Development for personal computers. Research on real-time interactive performance systems.
1974 - 79 Michigan State University Associate Professor of Psychology
1969 - 74 Michigan State University Assistant Professor of Psychology
1972 Stanford University PhD Mathematical Psychology
1964 University of Illinois BS Mathematical Statistics
Research Interests: Interactive high performance real-time computer music systems, music perception and cognition, representation of musical material, gesture sensing and controllers.
Wessel, D. Improvisation with Highly Interactive
Real-Time Performance Systems, Proceedings of the 17th
International Computer Music Conference, International Computer
Music Association, 1991.
Freed, A., Zicarelli, D. & Wessel, D. MAX Objects for
Media Integration, Proceedings of the 17th International
Computer Music Conference 1991, International Computer Music
Association, 1991.
Wessel, D. Let's Develop a Common Language for Synth
Programming, Electronic Musician, August, 1991.
Wessel, D., Lavoie, P., Boynton, L., & Orlarey, Y.
"MIDI-LISP: A LISP based programming environment for MIDI on
the Macintosh". Music and Digital Technology Strawn, J. (ed)
Audio Engineering Society Press, New York, 1987, 185-197.
Wessel, D. L., "Timbre space as a musical control
structure". Foundations of Computer Music, Roads, C. &
Strawn, J. (eds). M.I.T. Press, 1985, 640-657.
Risset, J. C. & Wessel, D. L., "Exploration of timbre
by analysis and synthesis (2nd edition)". The Psychology of
Music 2nd edition, Deutsch, D. (ed). Academic Press, 1997 (in
press).
McMillen, K., Wessel, D., & Wright, M. ZIPI's Music
Parameter Description Language, Computer Music Journal , 18(4),
52-73, 1994.
Lee, M., Freed, A. & Wessel, D. Neural Networks for
Simultaneous Classification and Parameter Estimation in Musical
Instrument Control, Proceedings SPIE Conf. on Robotics and
Intelligent Systems, May 1992.
Lee, M. & Wessel, D. Connectionist Models for Real-Time
Control of Synthesis and Compositional Algorithms, Proceedings
of the 18th
International Computer Music Conference 1992, International
Computer Music Association, 1992.
Wessel, D. Instruments That Learn, Refined Controllers,
and Source Model Loudspeakers, Computer Music Journal, Vol 15,
No. 4, Winter 1991, 82-86 MIT Press, 1991.
U.S. Patent Application Serial No. 08/551,890 Filed: October 23, 1995
Control Structure for Sound Synthesis
CD's:
Go Where? (1983-85) Japan Victor VDR-1026.
Antony (1977), Wergo (WER 2030-2) 1992.
Situations I (1993) , Lovely Music 1997
Recent Works/Performances:
Ngai Pha Yul, IRCAM, 1996 and New York, 1997.
Dangu Duo, IRCAM and CNMAT, 1996.
Situations II, Guanajuato, Mexico, 1995 .
Situations I, Korea, 1993
Interplay, San Francisco, 1993
C. Other collaborators
Miller Puckette, UC San Diego
Pierre Boulez
John Wawrzynek, UC Berkeley
Roger Reynolds, UC San Diego
Edward Lee, UC Berkeley
D. Graduate Students (past five years)
Pre-doctoral students supervised:
Joseph Rovan, expected 1998
Vijay Iyer, expected 1998
Rafael A. Irizarry, expected 1998
Paul Debevec, 1997
Tod Hodes, 1997
Krste Asanovic, 1996
Tom Parks, 1995
George Hajdu, 1995
Lynn Plemondon, 1995
Thorsten Helmut von Eicken, 1993
Joanna Weber, 1993
Kourosh Saberi, 1993
David Zicarelli, 1993
Ramesh Govindan, 1992
Post-doctoral scholars supervised:
Sophie Donnadieu, 1997-98
Cyril Drame , 1996-97
Pierre Korzilius, 1995-97
Givoanni Muller, 1993-94
Greg Sandell, 1991-93
Xavier Rodet, 1991- 93
Summary:
Graduate students advised: 20
Postdoctoral students sponsored: 6
E. Prof. Wessel's advisors:
William K. Estes
Richard C. Atkinson
Patrick Suppes
A. CURRICULUM VITAE
Edward Ashford Lee
EECS Department
University of California, 518 Cory Hall, Berkeley, California 94720
phone: (510) 642-0455, fax: (510) 642-2739, email:eal@eecs.berkeley.edu
RESEARCH INTERESTS: Signal processing, parallel computation,
real-time software, digital communication
EDUCATION:
UC Berkeley, Ph.D. in Electrical Engineering, December, 1986
Major field of study: Signal Processing, Minor fields of study:
Computer Science, Applied Mathematics
Dissertation title: A Coupled Hardware and Software Architecture
for Programmable Digital Signal Processors
MIT, SM in Electrical Engineering, December 1981
Yale University, BS, May 1979
Double Major: Computer Science and Engineering and Applied
Science, Magna cum Laude
ACADEMIC POSITIONS:
UC Berkeley: Associate Professor, July 1990-present; Professor,
June 1986-July 1990; Graduate Student Researcher, September
1982-June 1986
MAJOR RESEARCH PROJECTS:
Co-director: Ptolemy Project. January 1990- present.
Ptolemy is a system-level design framework with ambitious
objectives to include practically all aspects of designing
signal processing and communications systems.
Director:Gabriel Project. September 1986- January 1990.
Gabriel is a software environment for rapid prototyping of
real-time implementations of signal processing applications on
parallel programmable processors.
B. Publications
Closely related to the proposed project:
W.-T. Chang, S.-H. Ha, and E. A. Lee, "Heterogeneous
Simulation - Mixing Discrete-Event Models with Dataflow,"
invited paper, to appear, RASSP special issue of the Journal on
VLSI Signal Processing, January, 1997.
E. A. Lee, "A Denotational Semantics for Dataflow with
Firing," Memorandum UCB/ERL M97/3, Electronics Research
Laboratory, U. C. Berkeley, January 1997.
S. Bhattacharyya, S. Sriram, and E. A. Lee,
"Latency-Constrained Resynchronization for Multiprocessor DSP
Implementation," Proc. ASAP Conference, Chicago, August 19-21,
1996.
A. Lee, "Computing and Signal Processing: An
Experimental Multidisciplinary Course", Proc. of IEEE Int. Conf.
on Acoustics, Speech, and Signal Processing, vol. VI, pp. 45-48,
Adelaide, Australia, April, 1994.
M. Parks and E. A. Lee, "Non Preemptive Real-Time
Scheduling of Dataflow Systems," in Proc. of IEEE Int. Conf. on
Acoustics, Speech, and Signal Processing, Detroit, MI, May 8-12,
1995, pp. 3235-3238.
J. T. Buck, S. Ha, E. A. Lee, and D. G. Messerschmitt,
"Ptolemy: A Framework for Simulating and Prototyping
Heterogeneous Systems," Int. Journal of Computer Simulation,
special issue on "Simulation Software Development," vol. 4, pp.
155-182, April, 1994. .
Other significant publications:
Lapsley, J. Bier, A. Shoham, and E. A. Lee, DSP
Processor Fundamentals - Architectures and Features, published
by Berkeley Design Technology, Inc., 39355, California St.,
Suite 206, Fremont, CA, 1996.
R. Barry, E. A. Lee, and D. G. Messerschmitt, "Capacity
Penalty Due to Ideal Zero-Forcing Decision-Feedback
Equalization," IEEE Trans. on Information Theory. Vol. 42, No.
4, pp. 1062-1071, July 1996.
Chiang, B. L. Evans, W. T. Huang, F. Covet, E. A. Lee,
H. J. Reeky, D. G. Messerschmitt, and S. Sister, "Real-Time DSP
for Sophomores," in Proc. IEEE Int. Conf. on Acoustics, Speech,
and Signal Processing, Atlanta, GA, May 7-10, 1996, vol. 2, pp.
1097-1100.
W.-T. Chang, A. Kalavade, and E. A. Lee, "Effective
Heterogeneous Design and Cosimulation,", chapter in
Hardware/Software Co-design, G. DeMicheli and M. Sami, eds.,
NATO ASI Series Vol. 310, Kluwer Academic Publishers, 1996. Also
presented at NATO Advanced Study Institute Workshop on
Hardware/Software Codesign, Lake Como, Italy, June 18 - 30,
1995.
E. A. Lee and D. G. Messerschmitt, Digital
Communication, Second Edition, Kluwer Academic Press, Norwood,
Mass, 1994.
C. Other Collaborators
David Wessel
B.L. Evans
J.H. McLellan
M.A. Yoder
R.H. Bamberger
A. Sangiovanni-Vincentelli
D. GRADUATE STUDENTS (past 5 years)
Ph.D. STUDENTS GRADUATED:
John Barry,1992.
Gilbert Sih, 1991.
Ho-Ping Tseng, 1990.
MASTERS STUDENTS GRADUATED:
Brian Link, December, 1992.
Gregory Walter, 1992.
Phil Lapsley, 1991.
Asawaree Kalavade, 1991.
Shuvra Bhattacharyya, 1991.
Total number of graduate students advised: 8
Postdoctoral students sponsored: 3
E. Advisor: Prof. David Messerschmitt