This chapter deals with the software engineering and software technological questions of this project. After an introduction of the architecture of the reused software Open Inventor, it introduces the concepts for storing, real-time rendering, transforming and controlling different time media.
All class names and code examples are set in typewriter style, e.g., SoCone.
To keep track with rapid hardware evolution in the sphere of application software, higher software productivity and quality is tried to be reached by effective software reuse. Object-orientation is seen as an enabling technique to create fine-grained software components which are reusable, extensible and flexible enough to (re)design software architectures of complex systems without tremendous expenditure. On the other side, object-oriented software reuse often could not keep the promises this technique has made. Objects often cannot be reused fine-grained, because single objects or mechanisms cannot be used separately, without importing the complete class-tree of objects and those, on which it relies on.
Another weakness lies in the concrete language, object-orientation is mostly realized in: C++. It is not possible, or at least not easy to build mechanisms in C++, that are generic enough to deal with customers needs and extensions. E.g. until the last revision of the ANSI C++, there was no possibility for a dynamic type checking.
Libraries are collections of function or classes. The protocol for
using a library's function is kept simple and often covered by the
documentation of input and output parameters. Therefore the functionality
provided by libraries can be integrated into many application domains. The
user of a library still has the duty
and opportunity to build his own specific 'frame' from where he calls the
library.
When the functionality is encapsulated in classes of basic data
types, this class library is often called toolkit.
Frameworks contain solutions for classes of problems and building
blocks of higher levels by providing object classes and a general
architecture of how objects cooperate. Frameworks separate the generic and
the specific parts of a solution and structure the generic parts as
collaborating objects. It often comes with a set of ready implemented
objects as a starting point, that build a running system. To implement a
specific part of a solution the user derives new classes from existing
classes, be they abstract or not. To build this work a framework for
3D graphics called Open Inventor from SGI was chosen to shorten the period
of development. Without using a framework this work could hardly be done
in the period of a thesis.
One task of this works is to verify the statement, if Open Inventor's
concepts and mechanisms are kept generic enough and not limited
only to render geometric data to OpenGL calls, but to render any media
data to any
domain.
The next section gives a introduction into Open Inventor.
This section gives an introduction to Open Inventor (recent Version 2.1.3) on SGI workstations. This introduction elaborates the abstractions in the framework's architecture in respect to use of them for other media data, especially time derived media like audio (and video in future works).
Open Inventor is a powerful toolkit for programming 3D graphics providing
building blocks for 3D graphics applications of different size in a fully
extensible C++ library. Open Inventor is one of the main libraries
available from SGI, developed over several years. To support 3D graphics
programming on every level it offers black-box reuse as a file
format
or a class library as well as
white-box reuse by providing its infrastructure as a framework for custom
extensions. Black-box reuse designates a reuse of components that does not
require knowledge of its internal implementation but only of their
interface. In Inventor this is the file format syntax and the programming interface,
i.e. class structure and their public methods and members. A white-box
reuse, i.e. integrate own classes into the system requires the knowledge
at least of a part of the internal implementation of the mother classes.
In this work Open Inventor is used as a framework because its generic parts are employed for an extended problem domain: rendering audio specific data to 3D shapes and audio-feedback in real-time and off real-time (movie file). Once Open Inventor is extended by new C++ classes, it offers these new functionality for black-box reuse on all levels.
In general Open Inventor is an interactive renderer independent in the domain it renders to. It supports hierarchical structuring of data to be rendered in an acyclic directed graph (see paragraph Scene Graph), interaction with this data in the render domain (see paragraph Manipulators), caching mechanisms, reading from and writing to a file representation (see paragraph File Format), generic event scheduling (see paragraph Callback Nodes and Sensors). Additionally to the data flow allong the graph-tree it allows data to flow from one object to another object including a processing in the flow (see paragraph Field Connections and Engines).
The core functionality of Open Inventor is independent from
the operating system, window system, devices and render engines. The
native purpose is to render 3D graphics using the low level graphics
library OpenGL, which is a industry standard. OpenGL is hardware accelerated on many SGI workstations and
integrated in the X-window system. Open Inventor is shipped with a
supplement with X-Windows components, like several viewers and editors.
Open Inventor is ported to many platforms.
The main task of Open Inventor is to interface between a high level usage of graphical objects in terms of description, rendering and interaction to the demands of a low level render engine, here OpenGL.
It is necessary for a real understanding of Open Inventor to have access at least to the reference manual of the libraries classes. There are several ways to get access to the Open Inventor's manual pages:
To understand the following introduction it is necessary to know the way classes are named in Inventor. See the following sections for further descriptions of these classes.
There are basic classes, for basic data structures, having the prefix 'Sb' for 'Scene Base', e.g. SbString for a string, SbVec2f for a vector composed out of two floats. All classes of objects that can be placed in a scene graph have the prefix 'So' for 'Scene Object'. Most of them are nodes. Nodes contain typed data as member variables, called fields. Fields provide a generic access protocol. All field classes have the prefix 'SoSF' for fields containing single values ('Single Field') and 'SoMF' for fields containing multiple values ('Multiple Field').
Classes that are specific to OpenGL, have the extended prefix 'SoGL', e.g., SoGLRenderAction is an action that traverses the scene graph to render it to OpenGL calls.
The naming convention in this work tsKit, for 'Time Signal Toolkit', is that all classes have the prefix 'ts'.
Like many other C++ libraries, Open Inventor has its own generic type
system based on the class SoType. This is necessary, because C++ has
no possibility for requesting the type of an object, referenced by a
pointer. Therefore every class derived
from the very base class SoBase has a method for requesting the
object's and class' type:
SoType getTypeId()
and the static
version
static SoType getClassTypeId(). Other important services
of the type system is to query the inheritance relations with
SbBool SoNode::isDerivedFrom(SoType parent)
and type generic object instanciation:
void* SoType::createInstance()
The design pattern, to create objects generic by choosing the type at runtime, is called abstract factory (see [9]). This mechanism allows to create objects of classes, that are designed after compile time of the program creating them! This is an essential design pattern to keep libraries and frameworks open.
Open Inventor's concept for representing all information affecting the scene (i.e. what is rendered to an image) in an acyclic directed graph is called scene graph. For a better understanding of this model, the simpler model of the state machine is introduced first. Open Inventor can be seen as an interface to such a state machine.
The simplest state machine is nothing more than a set of functions that affect a domain. Beginning from a defined start situation, these functions are called with specific arguments and change the domain's state. The set of functions might be drawing commands of a graphic library, and the domain that is affected is a bitmap on the screen that is blank at the beginning. There are commands, that set drawing parameters, like color and affect commands executed in the future by changing the librarie's internal state. Other commands only affect the bitmap, by, e.g., drawing a line. The state, i.e., the picture that is drawn is determined by the sequence of function calls the program produces. It is Important that there is no other notation of this sequence of calls manipulating the domain and the systems state, than the program itself.
This simple way of dealing with manipulations of a domain can be implemented very efficiently in terms of performance and storage. This basic interface gives the interfacing system the biggest freedom to its internal organization. This becomes important, when dealing with tasks that are accelerated by special hardware. In general, low-level APIs are often realized as state machines.
An example is OpenGL. This is an industry standard for an platform independent API for 3D, 2D graphics and image processing. The former version IRIS GL was invented by Silicon Graphics Inc. for making usage of special graphics hardware, if this is available on a concrete workstation. The frame buffer and the set of library internal variables are the state of OpenGL. There is no particular concept for time in OpenGL.
Other examples are low level graphics libraries, e.g., Microsoft's Direct3D.
The disadvantage from the view of an application programmer is the low abstraction level that state machines offer. Especially for vision, we are used to deal with objects and their properties, e.g., we think of wheels of a car, instead of command sequences drawing a wheel. The application programmer has to make this mapping from an object idea to a sequence of drawing commands. A state machine gives neither the opportunity to summarize commands to an object, nor to deal with relations these objects have with each other! The mapping from a object description to a sequence of drawing commands is common to many applications and can be standardized, because it is redundant in many aspects. This gives the motivation to introduce the graph/traversal model described in the next subsection.
There is another concept to representing data (3-D graphics, images, audio, GUI-elements, windows ....) in conjunction with an rendering process on this data: graph/traversal model.
A very widely used concept for the given problem is to represent all data concerning the domain in a ordered directed acyclic graph (DAG) and to traverse the graph for a special purpose. A graph consists of connected nodes. Starting from a unique root-node, there are connections to its child nodes. For a child node (short: child) there is only one node, it is connected from, called its parent node. Nodes that have no children are also called leaf nodes (short: leaf). The connection of two nodes is used to model a relation between these two nodes, which is specific for the interpretation of the graph. A traversal is the software process of interpreting the graph, starting from a node and following the node's connection in a ordered way. See fig. 4.1 for a simple acyclic graph.
Acyclic means, that there is a node A connected to a child node B, but no connection that makes A a direct or indirect child of B.
Figure 4.1: A simple acyclic directed graph (DAG) with 7 nodes,
symbolized as circles. The number in the circle shows the order of
traversal (from left to right) used in Open Inventor.
In Inventor the scene graph is built out of different C++ objects, all are
derived directly or indirectly from the abstract base class SoNode. This
class contains among others the methods and member variables to be
connected in a scene graph. Most nodes in Inventor are leaf nodes. All
information that makes up a scene is stored in nodes, either in a shape
node, that represent an actual shape or in a property node, that represent
a specific property for a shape, e.g.
its color or size. Examples for nodes are SoCube to represent a
cube, SoMaterial to represent color and transparency information,
SoLight and SoCamera to represent lights and cameras of
various types. These nodes contain relevant information to the scene,
but they cannot have children. Only classes derived from class
SoGroupNode can have children (methods among others:
addChild(SoNode* nodeToAdd)). These group nodes do not contain relevant
data, but have important influence on the interpretation of their
children on traversal.
Nodes store scene graph data in objects called fields, derived from the abstract base class SoField. There are fields for one or multiple values of different types, e.g., SoSFFloat is a field class for one float value, SoMFColor is a class for multiple colors (of class SbColor).
The traversal is done by objects, derived from the abstract base class
SoAction. Each action traverses the scene graph for a different
purpose and with a different interpretation of each node type. The most
important action is the SoGLRenderAction. The SoGLRenderAction
does the core functionality of Open Inventor by traversing the scene graph
and producing OpenGL calls, so that the scene is rendered into a 2
dimensional frame buffer possibly mapped to the screen. There are other actions e.g.,
for writing the scene to a file (SoWriteAction), searching the scene
graph for a special node (SoSearchAction) or computing the bounding
box for parts of the scene graph (SoBoundingBoxAction). Finally, on
traversal, the actions call node methods to perform their action specific
behavior, e.g., the SoGLRenderAction calls GLRender() for
every node.
There is one more important mechanism that is essential, before a simple
scene graph scenario can be fully explained. Where does Inventor take all
the properties from that can affect, e.g., a cube shape? Where is the
cube positioned, which color is used, and how is it sized? It would be
very unconvenient and inefficient to attach all possible properties to
every shape object explicitly. It is the position in the scene graph that
determines the scope of a node's influence. In general, a node that
contains information, that should be shared by other nodes and so affects
these, stores this information in a special object when traversed, that
brings it to other nodes along the scene graph. Other nodes that follow in
the scene graph read this information and use it for rendering
themselves. These objects are called elements (derived from a
abstract base class SoElement) and store information for the period
of a traversal. Elements function as a
vehicle for the propagation of scene graph information from a property
node to a shape node. There are elements for information describing, e.g.
material (SoMaterialElement), geometry (SoCoordinateElement )
and drawing style (SoDrawStyleElement).
The collection of all elements that are relevant for an action is called a state (class SoState). A state stores elements in a separate stack for each element. The top element on the stack is the valid one, that following nodes read from.
It is often neccessary to limit the effect of nodes to only a subtree of the scene graph. An important group-node class SoSeparator makes its child nodes having only a local scope by storing the complete state before its children are traversed and restores it, when this traversal is done. Fig. 4.2 shows a simple scene graph and how it is traversed by an SoGLRenderAction. Separators are used to group nodes to a bigger entity and to protect the rest of the scene graph from settings that are specific to this entity. At the beginning of a traversal elements with default values are created.
Elements affect the state in different ways. The simplest and
most used policy is to replace the values in the former element. These
elements are derived from the class SoReplacedElement and model an
absolute change of properties. Translation, rotation and
scaling are the most important example of object properties that
should be changed relatively to the former setting. This policy is
practiced by all elements derived from SoAccumulatedElement and
models a relative change of property values. The corresponding element
(SoModelMatrixElement) does it by accumulating (i.e. adding) its
translation, rotation and scale vector to the one valid in the
state.
Figure 4.2: A simple scene graph and how its nodes affect the state
when traversed by an SoGLRenderAction using a SoGroup node
(left) and a SoSeparator node (right).
A well known example for the graph/traversal model is the X-Windows system: The X-Windows system models the GUI of an application in the computer system. It organizes GUI atoms (widgets) as nodes in a widget-tree. The relation that the graph represent, is a 'part of' relation. The tree is traversed by actions for drawing the widgets and handling events. Both action behaviors are coded into the widgets. Widgets draw and handle events themselves.
SoXtComponents are collections of X-Motif widgets wrapped in C++ objects for convenience (abstract base class: SoXtComponent). They are attached to a single node or the whole scene graph. Their purpose is to provide a GUI for those. Examples for these classes are viewers for scene graphs (SoXtExaminerViewer) and editors for single nodes ( SoXtMaterialEditor and SoXtDirectionalLightEditor).
The scene is normally viewed in a window on the screen by a program or widget-component called viewer (various classes derived from SoXtViewer). A viewer employs among others a render action to render a image of the scene, whenever the scene has changed, e.g., due of change of perspective. The navigation and change of perspective is realized with a manipulator that the user cannot see, that is attached to the first SoTranslation. See the following subsection Manipulators for further information about this direct control mechanism for nodes.
A way to group nodes that have a common context and represent a more complex shape is a nodekit. A group of nodes and its scene graph structure are encapsulated within a nodekit. Its internal scene graph structure is partly hidden. A nodeKit can be equipped with new fields.
In a previous versions of the tsKit, nodekits were used to capsule the representation, e.g., of spectral audio signal with standard nodes (SoCoordinate3/4 and SoQuadMesh) and its special behavior (animation, cursor) with new fields using a nodeKit derived from class SoShapeKit.
There are several disadvantages to represent media data in a hierarchical scene graph structure. They are listed in [21], in the chapter Optimizing the Data Organization:
It is common for scenes to have hierarchical definitions. Scene management techniques may rely on specific hierarchical information. However, a hierarchical organization of the data raises several performance concerns:
- The time spent traversing pointers to different sections of a hierarchy can create a CPU bottleneck.
This is partly because of the number of extra instructions executed, but it is also a result of the inefficient use of cache and memory. Overhead data not needed for rendering is brought through the cache and can push out needed data, causing subsequent cache misses.
- Traversing hierarchical structures can cause excessive memory paging.
Hierarchical structures can be distributed throughout memory. It is difficult to be sure of the exact amount of data you are accessing and where it is located; traversing hierarchical structures can therefore access a costly number of pages.
- Complex operations may need access to both the geometric data and other scene information, complicating the data structure.
- Caching behavior is often difficult to predict for dynamic hierarchical data structures
Open Inventor offers several features that are specific for the runtime behavior after a scene graph is created. Runtime behavior is everything that happens dynamically and is event driven, not program driven. As it becomes more possible to express runtime functionality with built-in scene graph constructions, it becomes less necessary to write an application for a special problem domain. The generic architecture for creation and communication of objects and a system to arrange this architecture at runtime lets users express behaviour with data instead of code. In contrast to the signal processing patchwork editor MAX (see. 3.3.3), this runtime editing of object arrangement is not an native purpose of the Open Inventor system, and is supported only with programs code like gview, even though it is supported by the system's internal architecture (generic object creation and querying).
Here is an overview of the runtime behavior for the scene graph and how it can be used.
Often it is necessary to let one field having the same value than another field.
Field connections are a basic way to express this dependencies in the scene graph. Connecting fields establishs a second data flow additionally to the data flow along the scene graph. A field B that is connected from another field A is constantly updated with the value of field A. A Field connection between fields of different types is automatically routed through a converter, if one exists for the two types. Field connections are a way to construct a distribution of data from one field to one or more without a processing of the data. The mechanism of engines (see next paragraph) makes extensive use of field connections.
Engines (all derived from the base class SoEngine) construct a dependency of field data that include a processing of these data by an engine's type specific algorithm. Engines have input and output fields of fixed types that are connected to other fields of the scene graph. When executed, the input fields are read, transformed and written to the output fields. Engines are not nodes, but part of the scene graph by their field connections to nodes (or other engines or global fields).
An examples of a general engine is the class SoCalculator. It has input and output fields for single and multiple fields of floats (class SoSFFloat) and an input field of type SbString containing a formula for mapping the input values to the output values. This formula string contains the field names as variables and is evaluated when the engine is executed.
Animation of the scene graph is done by taking the global field
'realTime' as an input field. Engines are evaluated only when necessary,
i.e., when a) the input fields' values change, and b) the output fields'
values (or any fields connected to these) are requested. That way engines
are evaluated only on demand and their execution is only done when
necessary. That way engines are never evaluated without their output
needed.
Manipulators are property nodes, with a graphical representation of
their field values as extra shapes. See fig.
4.3 for an example. When the user
drags such a graphical shape, the manipulator changes its field values
accordingly. Because a class of a manipulator node must be derived from
the node
class it manipulates, it has exactly the same functionality. For editing,
nodes are replaced with their manipulator versions. The author of a
manipulator class only has to represent the nodes data as a extra shape
the user can manipulate with the mouse and change the fields content accordingly. The shape and its
ability to be dragged by the user is offered by objects called
draggers (derived from the class SoDragger). Draggers allow
standard or user-defined geometry to be dragged with a specific policy
and a specific degree of freedom.
There are some manipulators in Open Inventor for the SoTransformation node. They offer various ways to change the translation, rotation and scale fields of this node, by creating boxes or circles around the objects they affect(that can be dragged by the user).
Figure 4.3: Two examples of manipulators replacing a node of class
SoTranformation: SoTabBoxManip, SoTransformerManip. Here
the node in the scene graph under the SoTranformation is of
class tsSurface.
Sensors construct reactions to events and are derived from the base class SoSensor. Sensors invoke a callback, whenever a sensor-specific event happens, such as a changes of the scene graph (field, node, group of nodes), the expiration of a timer, or the processor status being idle. Other dependencies can be introduced by deriving new sensors classes.
Viewers make use of sensors, by guarding the entire scene graph with a sensor of class SoNodeSensor. Whenever the scene graph changes its content, the viewer starts a new render action to update the scene to the render area. The scene graph might be changed by user interaction or any other reason of field changes.
Open Inventor provides automatic sensor scheduling according to a priority level.
Another way to introduce event dependencies are callback nodes SoCallbackNode. Open Inventor has its own event scheduling system including the window system events (keyboard, mouse). On UNIX machines these are X-Windows events. User can install callbacks to handle specific event types. If there is no callback to handle a particular event, it is handled by Open Inventor.
Inventor has a file format for serializing scene graphs, to write and read them into/from files. This mechanism also covers all new classes. There is an ASCII and an binary version of the file format with a default extension '.iv'. The ASCII format is well structured and can be used to design scene graphs with only a simple text-editor.
Figure 4.4: This firgure shows a simple scene graph that contains some
tsKit nodes as a ASCII file (left), rendered in the standard viewer
ivview (right, top) and a schematic view (right, bottom).
The file format is the simplest interface to Open Inventor.Here are the objects of a scene graph that are written to file:
Open Inventor is an object-oriented, C++ based, value-adding interface for OpenGL. It offers a hierarchical object structure for describing a scene that is translated into OpenGL calls, allowing the user to deal with abstract graphical objects instead with graphics commands of the low level state machine OpenGL.
Here is an abstract view of the scene graph concept: In the 'real world' there are independent objects occupying space with different properties. The implementation to render a computer model of this with OpenGL needs a mapping to a list of drawing commands. The scene graph model of Open Inventor organizes objects and their properties hierarchicaly to generate a serial description on traversal. Hierarchy is used for an efficient representation (data of one property node can be used by multiple feedback nodes) and control (one property can affect all following feedback nodes) of objects. You can see Inventor's graph/traversal model as an inheritance of data and of behaviour. A node inherits data from nodes due to its position in the scene hierarchy and it inherits behaviour (methods and member variables) due to its position in the C++ class hierarchy.
Time is handled in Open Inventor only as data type for fields ( SoSFTime). Time can only influence the scene graph by field connections and engines .
The basic idea of my work is to fuse Open Inventor's 3D graphics and other its general capabilities with digital media libraries to create a universal multimedia and intramedia rendering tool. The term 'intramedia' reflects a motivation to support the inter-medial transcription from audio to video.
The digital media library from SGI is in fact a collection of libraries to support media programming on SGI workstations. Until now the audio library is only used for reading and playing audio files (tsData1RFloat and tsAudio node). The image conversion library and the movie file library is used for hardware accelerated motion JPEG compression of rendered frames (tsRendererViewer). In future work the tsKit should be extended to be an easy interface to all digital media facilities including video and image processing.
The GUI builder RapidApp from SGI is used to simplify the process of building X-Motif based user interfaces. RapidApp generates C++ code for the user to insert his custom specific code. Although RapidApp makes use of the application programming framework ViewKit, here only the VkComponent class is used isolated, because this makes tsKit's reuse more easier. A VkComponent in RapidApp is very similar to a SoXtComponent in Open Inventor. See [17] and [20] for further documentation of these tools.
Scrub is a collection of classes written by Tobias
Kunze to play audio
files with various speed in both directions. To prevent the playback of
artifacts due to aliasing, an expensive real-time resampling algorithm is
applied to do antialiasing for achieving high audio quality. It implements
a signal processing algorithm from Julius O. Smith III, Bandlimited
Interpolation - Introduction and Algorithm, 1994
I added the following features into scrub to integrate it into my work:
class Scrubber {
....
public:
/// Sets the read position in audio file.
int SetFilePos(long newpos);
// Sets the fiel's current read/play position in seconds.
void SetFileTime(double newTime);
// Returns the file's current read/play position in frames and seconds.
long GetFilePos() ;
double GetFileTime() ;
....
}sample rate.
class Scrubber {
....
public:
// changes the output sample rate.
int SetOutputRate(double newOutRate);
// changes the output sample rate relatively to the source rate (in file).
int SetIORateRatio(double newIORateRatio);
double GetIORateRatio();
...
}
This section describes the design decisions made to reach the project's goal. See The tsKit Reference Manual for a detailed look at a specific classes.
All classes that are introduced by an extension of Open Inventor have to be initialized as new types of Inventor objects. Being registered by the Inventor's own type system, object types can be queried and objects can be instantciated from a type (of class SoType). This mechanism is needed when unknown Inventor objects (nodes) are detected in a Inventor file. (See section 4.2.1 for an introduction to Open Inventor's type system.)
The type initialization of a class is done in its static method
initClass(). For convenience and security there is a macro doing this. It
takes the name of the new class and the name of the parent
class. Its definition for nodes is:
SO_NODE_INIT_CLASS(<newClassName>, <parentClassName>,
"parentClassFileName");
Normally non standard Inventor classes are type initialized after their first occurrence (while reading them from a file or by explicit instantiation). A dynamic shared object (DSO) of its name is loaded and its object code is dynamically linked. During this project the class type initialization policy became a problem, because
A robust solution to the above problems is to initialize all new classes
in an ordered way in one method. Therefore I created a new node
tsKit whose only purpose is to summarize the class initialization. The
global initialization is done by calling the static method
tsKit::initClasses() or by putting tsKit{ } in the top of
a file.
In future work this procedure should be done automatically and so stay hidden to the user.
This subsection is about how tsKit represents its most interesting data
structure, the time signals. The representation should cover all kinds of time derived
media signals. Audio and its related signals are subject of this first
realization and therefore shaping the design.
In Inventor there are already data structures to represent arrays of basic data types like floats, integers and vectors. Scene graph nodes like SoCoordinate3 and SoCoordinate4 use these to represent geometric information of shapes
Reasons for introducing a data representation for signals as a new class family are:
All nodes storing data of a signal are derived from the abstract base
class tsDataBase and are called tsData nodes. It is assumed
that classes derived from this class represent a signal that has a
dimension 'time' at least.
The access protocol includes methods for querying the minimum and maximum values, because these operations are fastest when directly done on the underlying data representations. If a client object knows the type of a tsData node, it can circumvent the unified access and use its type-knowledge to practice a node-type specific access. Figure 4.5 shows the methods defined in this class, realizing a unified access to the signal data.
Figure 4.5: Methods building a unified access to signal data, as defined in
tsDataBase.h.
The next subsection introduce the nodes for storing signal data that are implemented so far. See The tsKit Reference Manual for a detailed description of all tsKit classes.
The term 'one-dimensional signal data' means here that the signal data is sampled in one-dimension, i.e. 'time', whatever the type of the signal data is. To provide unified access for all one-dimensional data, there is the abstract base class tsData1Base. The numeral '1' in the class name denotes the dimensionally, for this class and its derived classes.
There are node classes to represent float signals, sampled regularly and
irregularly. Their classes are tsData1RFloat and
tsData1NFloat. The class name contains a 'R' for regularly sampled
or a 'N' for
irregularly sampled and 'Float' to denote the signal's type of values.
The node class tsData1RFloat contains the following fields:
The node class tsData1NFloat contains the following fields:
A regular float signal sampled in n points needs only n floats for the signal values plus two floats for the time axis: 'startTime' and 'interval'. The native irregular representation needs n floats for the signal values and n floats for the points in time when the data is sampled.
Comparing the use of memory for a signal of n values:
Nodes of class tsData1RFloat can represent audio as a time signal. See 4.4.4 for a class derived from this class, to make the signal data being read from a file.
Nodes of class tsData1NFloat are often used to represent signals that are derived from other signals, like the pitch contour. Fig. 4.6 shows how these nodes represent the signal.
Figure 4.6: This figure
shows how regular and non-regular sampled float signals are stored.
The signal access interface is extended for methods that are specific for one-dimensional signals. The actual getValue() method takes a time axis index of the data and returns the value as a void* Pointer. Figure 4.7 shows the additional methods.
Figure 4.7: Methods completing the unified interface to one-dimensional
signal data, as defined in http://www.cevis.uni-bremen.de/ilab.h.
The term 'two-dimensional signal data' means here that the signal data is sampled in two-dimensions, 'time' and one additional one. The signal data itself can be any data type. To provide a unified access for all two dimensional data, there is the abstract base class tsData2Base. The numeral '2' in the class name indicates the dimensionality for this class and its derived classes.
There is a new node class tsData2RNFloat to store a two-dimensional
float signal that is sampled regularly in time but irregularly in the
second dimenion. The second dimension is irregularly sampled (called dim2
for short) for all the signal. Fig 4.8 makes this
more clear. The class name contains a 'R' for the first and regular
sampled dimension 'time' and a 'N' for the second and irregular sampled dimension. '
Float' indicates the signal's type of values.
The node class tsData2RNFloat contains the following fields:
A float signal regularly sampled in it n points along the time axis and it m points along the irregularly sampled second axis need only n*m+m+2 floats, n*m floats for the signal values, plus two floats for the time axis, and m floats for the second axis. The two floats are used for it stroring the start time and it interval. The native irregular representation needs n*m*3 floats. That is n*m value, each containing the actual signal value and its position at the two axes.
Comparing the use of memory for a signal of n values and m values in the 2nd dimension:
Nodes of class tsData2RNFloat are used to represent audio signal as
a sequence of frequency spectra. The field 'data' stores the amplitude
values sampled in frequency (values in 'dim2') and time ( fields
'startTime' and 'interval').
Fig. 4.8 shows how the tsData2RNFloat node represent the signal.
Figure 4.8: This figure shows how the two-dimensional sampled float
signals of class tsData2RNFloat is stored.
The signal access interface is extended for methods that are specific for two-dimensional signals. The actual getValue() method takes a time axis index and a dim2 index and returns the value as a void* Pointer. Figure 4.9 shows the additional methods.
Figure 4.9: Methods completing the unified interface for two-dimensional
signal data as defined in tsData2Base.h.
Elements function as vehicles for information between nodes. They are employed by an action that traverses the scene graph for a special purpose. Each type of elements transports a special type of information. Property nodes store information, and set data in an element, providing it for nodes lower in the scene graph. Other nodes, e.g., shape-nodes use this information to draw a shape. This way geometry data stored in a SoCoordinate3 or tsData1RFloat node, can be used by more than one feedback node. Usually a feedback node uses the information from more than one kind of element.
A property node models the static representation of a property and is created and arranged by the user. An element models the spreading of a information from the 'producer', a property node, to the 'consumer', a feedback node and stays invisible to the user. The granulation of property nodes and elements is not the same! The granularity of elements is driven by the needs of the destination nodes consuming them and is kept fine to making caching effective. Nodes can employ several elements to propagate their data.
Because tsKit introduces new propertys nodes (property nodes and feedback
nodes for signal data) there are corresponding elements introduced as
well. In this project there are several elements needed to propagate the
content of one node with multiple fields (fields that store multiple
values) to a corresponding element. Therefore in tsKit there is an
abstract element base class tsNodeElement for propagating a
reference of a node and that contains all neccessary methods and
variables. This is done by simply storing a pointer refering this node
and a type (SoType) storing the type of the node being referenced.
This policy differs from how Open Inventor propagates multiple field
data and is chosen for simplicity.
Here is an overview of the new elements and which classes set and get information to/from them:
Practice will show if this model must be extended in future work for feedback-nodes that need information from more than one signal of the same type, e.g., one two-dimensional and multiple one-dimensional signals.
An important requirement for this toolkit is to access signal data in file
formats different from the Inventor file format. Why not let the tsData
nodes read in signal data from custom file on creation time ? How generic must this
mechanism be? Here are some possible mechanisms:
.
In tsKit the second mechanism is implemented. Classes that are derived from a tsData node to implement file reading are called tsDataFile nodes and contain an additional field 'filename' of type SoSFString.
The signal data of such a node is being assumed as the default (
isDefault() method returns true) value of the corresponding field(s) and
therefore is not written when the scene is written to a file to keep the
file small and simple to edit. This data is prepared for written down by
changing on of its field content or simply by touching
one field.
Here is a description of which file format is read by which tsData class:
. If the
field 'interval' is not empty, the data is resampled accordingly using
the audio file library's mechanism of virtual rate.
The node maps the amplitude and frequency values to a logarithmic scale.
A more generic handling of file formats is the subject of future work.
Another subject of future work is to let the user specify a special part
of the file content or a stage of preprocessing. This gets important when
the tsKit will be extended to read the CNMAT's sound description file
format . This
file format can contain different signals of several types.
In this subsection time as an important property for the rendering of media signals is introduced. Time and space are the most important dimensions in which we perceive the world around us. Hearing and seeing use these dimensions differently. In [10] Graver summarizes: ''Images exist in space, but over time. Sounds exist in time, but over space.''.
Visualizing audio requires rendering a time signal as a signal of time
and space (animated 3D graphics). An essential challenge of this
work is to establish time as a integrated dimension for rendering in Open
Inventor. Time is handled in Open Inventor only as a basic data type
SbTime for fields (SoSFTime). Time can influence the scene graph
only by field connections and engines and is provided as a global field
called 'realTime'.
The first version of tsKit used field connections for distributing the parameter time. Feedback nodes were performed according to a SoSFTime field they contained. This approach could not use the hierarchical structuring of the scene graph for propagating this property.
Time can be seen as just another property for rendering. The tsKit introduces feedback nodes that uses this new property as a built-in character.
Time itself has another special characteristic: for an ingenious realization, time must be changing continuously during the playback. Media like audio and video are very sensitive to their continuous performance in time. They are stored and performed in a constant density of frames (audio: samples, video: frames and fields) per time unit. This is called the frame rate of an audio or video stream. Uncontrolled changes in the frame rate, e.g., because of bad timing or insufficient performance causes a decrease in perceived quality. Audio is even more sensitive to changes in frame rate than video. In contrast animated graphics are least sensitive to the continious performance in time because they are not bound to frames but can be performed at any point in time.
Another view says that audio and video are best controlled by the parameter speed (or rate), expressing the continious change of time. Mathematically speaking, speed is the first derivative of time. Animated computer graphics are best controlled by absolute time. User interfaces of time media uses absolute time and speed (or rate) as a control parameter. Time is controlled absolutely by controls like 'go to begin' and 'go to end' and speed with 'play forward', 'play backward' and 'stop'.
One application for the control of time is the non-real-time playback. This is used for slow motion playback or to render graphics and audio to a movie file, frame by frame. Later this movie can be played back in real-time. That way visualizations can be viewed in real-time that cannot be rendered in real-time: they are too intensive in computation to be done in real-time.
As a solution for the time/speed dualism I created a property node
containing both parameters 'time' and 'speed' and that keeps these values
consistent. The node class is called tsTime and consists of the
fields 'time' of class SoSFTime and 'speed' of class
SoSFFloat. The field 'time' is constantly updated according to the value in field 'speed'(, as
long as speed is not 0.0). This is done in a constant frequency of 30 Hz.
Here is how the tsTime node works as a master clock: When a tsTime node is part of a scene that is viewed in a viewer, the changes of the field 'time' causes a regular redraw of the scene. This way a tsTime node becomes a master clock for all feedback nodes lower in the scene graph (and for all nodes or engines, that are connected to the fields 'time' or 'speed').
To propagate these new properties 'time' and 'speed', a corresponding element of class tsTimeElement is created.
There are three kinds of latencies between a change of the clock in a tsTime node (the master) and the reaction of the feedback nodes (the slaves). The reason for a change of the tsTime node can be a user command, e.g., 'play' that sets the 'speed' field from 0.0 to 1.0 or the automatic updating of the field 'time'. Figure 4.10 shows a schema of these latencies. It shows a scene graph with a tsTime node that is controled by a tsTimeCtrlUI node and two feedback nodes (tsSurface, tsAudio).
Here is how the feedback node in tsKit are synchronized and how they
handle the diffrent latencies of their reaction. The first latency is the
same for all nodes and is not handled further. To let feedback nodes
handle the second and third latency, the tsTimeElement contains an
additional high resolution UST time stamp of the time when the element's
data was set. This time stamp can be used by feedback nodes to calculate
the latency between the time when the tsTime node was traversed and
when they were able to react. The UST time stamp is provided additionally to the
time in the fields because it has a much higher resolution and is fast in
processing.
This synchronization mechanism cannot prevent latency in the reaction of feedback nodes caused by limited processing power, but it lets the nodes handle this latency explicitly. Once nodes have started their playback, they are in time. See fig. 4.10 for a schema of this mechanism.
Figure 4.10: This schema shows how a feedback node (here tsAudio)
can make use of the time stamp included in the tsTimeElement.
Media playback is hard, because the media data is often played back from
different media buffers by different threads at different rates and because of additional hardware
constraints. This synchronization mechanism is chosen with respect to audio
(and video) being played in separate threads.
Here is how the feedback node tsAudio for audio playback is implemented: the node creates a thread that keeps a frame buffer filled to a minimum level and sends the samples to an audio device. It communicates changes in speed and absolute time to that thread only when necessary. Therefore the node reads the tsTimeElement when traversed and checks if its thread is in time, using the element's time, speed and time stamp information. If necessary the node tells the thread to use new parameters (absolute time position and/or speed) in order to be in time again.
In Inventor and so in tsKit the scene graph is the central structure for data and its behavior. The scene graph contains objects and their relations. There are at least two aspects the user can control: The topology of the scene graph (including field and engine connections) and the values in the fields of nodes.
Fields (see section 4.2.3) are typed containers of data and are part of nodes and engines or exist as global fields.
Besides the trivial way to edit an Inventor ASCII file, there are several interactive ways to edit field values. The fields of a node can be edited in dialogs, which are placed in extra windows or integrated in the scene. Here are the dialogs, either general or specificly to a node type:
There is a set of standard classes in Inventor derived from SoXtComponent that build a X-Motif dialog for editing only nodes of a specific class. These dialogs are not nodes and therefore cannot be part of the scene graph. They only can be created by programming. Examples are SoXtMaterialEditor and SoXtDirectionalLightEditor.
There is a class FieldEditor (derived from SoXtComponent and is part of the demo source code from SGI) that builds a X-Motif dialog for editing a field of any class type using the common field data access methods from the abstract base class SoField.
In tsKit there is one GUI node: tsTimeCtrlUI. Its purpose is to create a X-Motif dialog for editing the fields of a tsTime node. That way the editor can be created as part of the scene graph. These node(s) are called tsGUI nodes because they build a user interface for tsKit nodes. The tsTimeCtrlUI node must be placed in the scene graph after a tsTime node.
An interactive way to control the structure and content of a scene graph is offered by the viewer gview. Gview shows both, a rendered scene graph and a visualization of the scene graph as a graph of icons, each representing a node. A double click on a node icon opens a dialog for editing the field's content. See Fig. 4.11 for a screenshot of gview. Gview also allows to create new nodes and delete existing ones. The list of nodes that can to be created covers all known nodes, including the classes that are added by tsKit or the user.
Until now there is no way to create engines at all. This is subject of future work.
The C++ source code of the program gview is part of the demonstration
source code for Open Inventor and is public domain. For the program
tsgview I only added some minor functionality:
Figure 4.11: This figure shows the viewer gview with a scene graph
loaded. The right window shows the structure of the scene graph
itself, the left window its rendered version. The selected node is
rendered with a bounding box. The node field's values are edited in
the generic dialog window 'tsTime' (right, bottom). Additionally the
scene graph contains a tsTimeCtrlUI node that generated the
dialog window 'tsTimeCtrlUI' (left, bottom).
It is subject of future work to improve the component SoXtFieldEditor used by gview/tsgview for editing fields more adaptive to their type. This includes e.g. editing enumeration fields (SoSFEnum of class type SbEnum) as menu entries instead of typing its values.
The control concepts described above, all put the place of control away from the model it controls: the rendered scene. A much more ergonomic way is to integrate the control in the scene itself. When the control is part of the scene graph, control and model are not only put in the same place on the computer screen, but the semantic of the manipulation is much more realistic. Only this mechanism simulates the real world experience of manipulating objects, where the manipulation and the feedback of the manipulation is done in the same place: the object itself.
It is clear that external control is much more simple to create than control that is integrated in the scene. Therefore Open Inventor offers mechanism for direct control of nodes, called manipulators. See section 4.2.3 for an introduction. This concept is not yet used by tsKit. It is planned to build manipulators for various property nodes of the tsKit, especially for the tsTime node. Because time and speed cannot be manipulated in their native dimensions, they are mapped to a geometry in space. Time can be a frame that is dragged along the time axis of a tsLine/tsSurface node.
The final feedback, whether it is visual or auditive, is rendered in new nodes of the tsKit, called feedback nodes. Their common feature is to provide feedback for time signals. All feedback nodes use data from tsData nodes, time/speed from the tsTime node and several other properties from other nodes to render them in their specific way. These feedback nodes contain their specific way to produce a feedback of given signals.
Until now there are nodes to render one-dimensional signal data to graphics and audio and two-dimensional signal data to graphics only.
This node uses the current tsTime and tsData1 node ( tsData1RFloat or tsData1NFloat) to render the signal as a line. The signal data is directly mapped to the height of the line strip over time.
See figure 4.12 for an example.
The node class tsLine contains the following fields:
Figure: A tsLine node drawing data from a file. The signal data
comes from a node of class tsData1RFloatFile and was created
reading data from an AIFF file. See 4.9 for the
file describing this scene.
Figure 4.13: An Inventor file drawing audio data with a tsLine node
as a line strip.
This node uses the current tsTime to render an audio stream at the current time and speed of the tsTime node.
The node class tsAudio contains the following field:
To make the implementation easier the software scrub (see destion 4.3.3) is reused, it can only can read the signal data from a file. The field 'filename' from a tsData1RFloatFile node is used for accessing an audio file.
Figure 4.14: Four tsSurface nodes drawing the same
tsData2RNFloat data, with different draw styles and light Models(
SoDrawStyle::POINTS, SoDrawStyle::LINES, SoLightModel::PHONG and
SoLightModel::BASE_COLOR).
This node uses the current tsTime and tsData2 ( tsData2RNFloat[File]) node to render a surface representing the signal. The signal values are directly mapped to the height of the surface.
This node computes normals if necessary and uses the current materials in SoMaterial or SoPackedColors. Figure 4.15 shows how the normals are computed. The normals of the surface are computed before the first rendering. The computation is not done by employing an existing helper object SoNormalGenerator providing this service, because the normals are computed as vectors of floats. See section 4.4.8 for how tsKit deals with normals.
The node class tsSurface contains the following fields:
Figure 4.15: This figure shows which vectors (arrows) are used to compute
the normals for the vertices (small circles) of a 3x3 tsSurface.
The dottet lines shows the border lines of the triangle polygons that
build the surface.
As one way to render shapes more realistic, their surfaces are shaded.
Shading is coloring a shape's surface depending on the intensity and angle
of incidence it is lightened. Open Inventor offers the
phong lighying algorithms. Like all other realistic lighting
algorithm, phong shading needs to know how the surface and its polygons
are oriented. This information is expressed as normal vectors (short:
normals). These normals can be computed automatically from the geometry of
a shape, given as polygons.
Before it is described how tsKit deals with normals, the native policy of Open Inventor is given.
The drawing of shapes is very different, whether the shape is a vertex shape (also called complex shape) or a parametric shape (also called simple shape) like a cone (node class SoCone). The geometry of vertex shapes is completely described by coordinates of vertices. These vertices are stored in coordinate nodes (SoCoordinate3, SoCoordinate4). Vertex-shapes interprets these data in different ways to draw their geometry. Parametric shapes are specified by parameters. A parametric shape interprets these parameters, e.g., the radius for a cube (SoShere), and generates its geometry by internal calculations.
All shapes have to use normals for drawing with phong shading. When they have not fixed normals (e.g., cube), they have to be computed, before drawing. There are objects (SoNormalBundle, SoNormalGenerator) that do the computation of the normals for a vertex-shape, according to a crease angle (SoCreaseAngle, SoCreaseAngleElement). Normal vectors are stored in a SoMFVec3f field of a SoNormal orSoVertexProperty node. These normals are stored as 3 single precision floats using 3 * 4 Bytes = 12 Bytes.
To keep the signal representation and its rendering memory efficient, tsKit does not use the native concept for calculating and storaging normals described above. Because the normal information is less sensitive to inaccuracy than geometry information, it can be stored (and send to GL for rendering) only as 3 short integers instead of 3 floats.
The alternative to compute the normals while the shape is rendered is assumed to be much too expensive in computation. The computation would be very redundant, because for two different render traversals most normals to compute are the same!
A new set of a node and an element is created, that store 3 byte normals in a SoMFInt32 field, until a field of class SoMFVec3b is created in future work. The class of the node storing the data is called tsNormal and is derived directly from SoNode. Because the class tsNormal does not contain a time signal it is not derived from tsDataBase.
To store the normal data of this node in the state, there is a new element called tsNormalElement. This class stores a reference to the tsNormal node.
It should be mentioned, that the memory efficient storage and rendering of normals as vectors of bytes involves cost in rendering performance, as documented in chapter 14 of [21]:
Using Geometry Operations Effectively
If your application seems transform limited, you can improve it by considering the tips in this section. The section starts with some general points, then discusses optimizing line drawing and using triangles and polygons effectively.
To improve performance in the geometry subsystem, follow these guidelines:
- Use single-precision floating point parameters for vertices, normals, and colors.
- Transform paths use single-precision floats is fastest to use glVertex3fv() and glVertex2fv().
- [....]
- Perspective transforms that require multiplication by 1/W or division by W are much slower.
- To minimize the time used for floating point conversion, use floating point coordinates where possible, except where memory size is critical.
- Don't enable normalizing of normals if the modelview matrix doesn't include scaling and if you have unit-length normals
In spite of this, there was no measurable loss in performance.
In Open Inventor normals are cached for rendering, so that normals are only computed once and shared between shapes, that use the same geometry data. Because this mechanism is not fully documented and there is no source code of nodes using this mechanism, tsKit implements a separate and simpler way to cache normals.
Until now only the feedback node tsSurface needs normals to be phong shaded. On the traversal of the tsSurface node it is tested if there are normals stored in the tsNormalElement that can be used for rendering. Otherwise the normals are computed, stored in a new object of class tsNormals and set in the tsNormalElement in the state. Normals are reused for a tsSurface node, if the number of normals fits its demand ( The number normals in the tsNormalElement object is equal number data values in the field 'data') and the element's type information is equal or derived from tsSurface! Every surface node also saves a pointer to the tsNormals node, storing its normals.
Using color and transparency as a visual dimension is extensively used in all fields of visualization. It should be well supported in the tsKit to use color and transparency as a visual representation of signal features in terms of efficiency and easiness of use.
tsKit uses the Inventor standard nodes/elements (SoPackedColors
and SoMaterialBinding) to store and map colors and transparency
values to a shape. A memory efficient way to store the diffuse
colors and the transparency value, is to pack them into on 32 bit
integer (4 bytes) in the hexadecimal form: 0xrrggbbtt, where the rrggbb
part stands for a RGB color, the tt part stands for
a 8 bit transparency value. This is called a packed color in Inventor.
The more memory-expensive way to store colors and transparency values for every facet, vertex or part, is to use the SoMfVec3f fields of the SoMaterial node for diffuse color (diffuseColor) and transparency (transparency). This way needs 4 floats = 4 * 4 bytes = 16 bytes for each color/transparency pair!
The actual computation of color/transarency values is ment to be done with a engine of class tsData2ToPackedColorEngine (See section 4.4.10).
Transforming one or more time signals into another time signal is an important need and concept of this work. Signal transformations are used to realize the central function of this system: a transformation from an audio signal into a visual signal!
Transformation of time signals is realized using the Inventor's concept of engines (see section 4.2.3 for an introduction.). The idea of an Engine is to transform one more input fields to one or more output fields, whenever the input fields have changed their values. When notified they read the input field(s), do something with this field-data and write something to their output fields. Before they can access a field (whether read from an input or write to an output field), this field must be connected to a field of another node, engine or global fields. All information an engine needs, must be accessible by the fields, the engine is connected to.
Inventor engines work on fields. In the tsKit all signal representations are stored in nodes derived from tsDataBase. For this reason it is be more convenient to transform tsData nodes instead of fields as entities for the operation. When the user wants to transform different types of signals, he does not need to know the representation in memory or what fields the nodes contain! Therefore the engine concept of Inventor is abstracted to deal with signals, rather then with fields. All classes derived from tsDataBase contain a field 'node' of type SoSFNode containing a reference to the node itself. This field functions as a handle for the complete node. That way tsData nodes can be connected using this field and engines can access the entire node, including all node specific methods!
The engine tsData2ToPackedColorEngine is only partly implemented until now. This engine computes the colors and transparency values for a tsData2 node according to an internal algorithm. It takes a tsData2 node as input and computes entries in the connected output field of class SoMFInt32. It is assumed that this output is connected to the field 'orderedRGBA' of a node of class SoPackedColor!
Because of unsolved programming problems the engine is not running yet!
A topic not yet covered by the tsKit is the selection of the data to be visualized. There are several reasons for selecting a subset of data for being rendered:
Another mechanism to scale rendering demands is to control the property node SoComplexity. This node contains a float value, that lets feedback nodes adapt their complexity absolute or relative to the size of screen they occupy.
The property complexity is not yet evaluated by nodes of the tsKit!
Until now this selection is done with the fields 'past' and 'future' of the feedback nodes tsLine and tsSurface, but will be put in a separate node and element.