tsKit - A Real-Time 3D Visualization Toolkit
for Sound and other Signals

1997 - Author: Andreas Lücke

Abstract:

The subject of this thesis is a software toolkit to visualize and transform audio and other signals. The presented solution for this interesting intramedia problem makes use of visual computing, to enhance the understanding of auditive perception. I have developed a reusable software system for rapid prototyping called tsKit using high-end software technology for recent graphic workstations (In particular from Silicon Graphics Inc.).

The main software engineering task was to generalize a mono-media framework architecture (3D graphics library Open Inventor (Inventor-Home, Inventor-Summary) into a multimedia library capable of supporting a synchronous real-time rendering of signals as auditory and visual output. Time is established as an integrated dimension for rendering in Open Inventor.

The result of this work is a collection of objects, that are integrated in the Open Inventor architecture and can used like built-in objects by scripting scene descriptions or C++ programming. The extension offers specific features for visualizing, computing and the playback of sound signals. The tools produced in this work have many potential applications in the fields of audio signal analysis and processing, speech recognition, computer music and other fields of research.

Features

• tsKit is an extension of the 3D graphics library Open Inventor and comes as a single dynamic shared object (DSO) for SGI IRIX 6.3. The new building blocks can be used in the same way as standard building blocks.
  
• tsKit offers a collection of new builing blocks (Open Inventor term: nodes) that are arranged in a scene description (scene graph).
  The most importand nodes are (click for manual page):
  
  • Master Clock: tsTime, tsTimeCtrlUI for synchronized feedback.
  • Data nodes: Nodes for storing audio data as a regular (tsData1RFloat[File]) or irregular sampled (tsData1NFloat) signal, or as a time-frequency signal (tsData2RNFloat[File]. The file, to read the signal data from, is simply noted in the scene description file.
  • Feedback Nodes:
    • tsSurface: Visualization as a surface.
    • tsLine: Visualize as a line strip.
    • tsAudio: Audio feedback.
      
• The scene graph can be created by simply scripting (trivial xample, realistic example) a scene graph file and loading it into a standard viewer or by programming in C++ (example).
  
• tsKit is open for new ways of visualizing audio data. New feedback nodes (beneath as a surface or line) or data formats can be derived as new C++-classes.

Documentation

• Thesis
  (Postscript version .ps.gz, (~1.7MB), Latex DVI Version(~MB))
• The tsKit Reference Manual Dokumentation of C++ classes and applicatioons. Enable Java to see class inheritance graph!
  (Postscript Version .ps.gz (~126KB), Latex DVI Version (~200 MB))

Demonstration Data

• Screenshot: (click thumbnail to enlarge)
• Some Open Inventor Files (.iv)
  Load these files in a viewer (e.g., ivview) to view and hear a sound. Download and install the tsKit Execute Only Environment (tsKit_eoe.tar) before.
• Audio Data Sets (.grid=wavelet analyze data; .mat=combined FFTs; aiff/.aifc=audio files)
  This data files are read by the Open Inventor files from above.
• Movies: (.mv=Fullsize MJPEG, big!; .mps=MPEG, 320x240; .qt=Quicktime, 240x180)
  Quicktime-Movies, rendered with tsKit's tsRenderer:
  
  30sec,~1MB5sec,~2MB
  12sec, ~4MB
  
  

Download

• tsKit Execute Only Environment for SGI/IRIX 6.3, including the library and all applications:
  tsKit_eoe.tar.gz (~ 600 KB)
• tsKit Developement Environment for SGI/IRIX 6.3 including source code:
  tsKit_dev.tar.gz (~ 250 KB)