Next: STSDAS Tables Package Uses FITSIO
Previous: MATADOR: Software for the Manipulation of 3D Data
Table of Contents --- Search ---
PS reprint
Richard Gooch
Australia Telescope National Facility, CSIRO and Macquarie University
The references cited at the end of this paper describe recent work at the CSIRO Australia Telescope National Facility which has produced Sutra, a suite of powerful new visualization tools designed for astronomical data. Other work at the CSIRO Division of Radiophysics has produced integrated computational and visualization tools, developed using the Karma library, in a diverse area of research ranging from cellular automata to interactive hypothesis testing for geophysical data. The Karma library has been successful in such a diverse range of applications because it provides an integrated toolkit which can act as a complete programming environment while not forcing the programmer to do everything the Karma way.
One of the key design goals of Karma is to ensure the library is simple to navigate. It should be clear to a programmer seeking a certain functionality from the library where to look. This is coupled with the realization that functional concepts may be repeated in the library when complex functional systems are implemented by using more primitive systems. When the programmer is faced with several functional systems which appear somewhat similar, it should be immediately obvious which system provides the greatest flexibility, power and ease of use.
To implement this design goal, the structure as set out below was developed.
An example of how this works is the core communications infrastructure, which spans four levels. These four levels are:
A programmer using the communications facilities of the Karma library would naturally choose to use the ``conn'' package, since it is the highest level package providing generic communications support.
Basic data types currently supported are ``atomic elements'' (numeric types such as float, integer, complex and so on) and pointers to multi-dimensional arrays and doubly linked lists. Because of the recursive, hierarchical nature of the data structure, arbitrarily complex structures may be represented. In practice, most applications have need for simple multi-dimensional arrays only or linked lists of atomic data values. The benefit of this scheme is that a single interface supports both simple and complex data.
The data are stored and transferred in binary format. The library routines transparently handle conversion between machines of differing binary data formats. The native data format is IEEE floating-point format, most significant bit first (big endian).
As powerful as the Karma data structure may be, real data come in (too many) different formats. The library provides a package for reading foreign data formats such as FITS and Miriad n-dimensional arrays. Common image formats such as PPM and Sun Rasterfile are also supported.
The basic concept of ``Intelligent Arrays'' was developed by Patrick Jordan of the CSIRO Division of Radiophysics. The following requirements were defined:
Jordan's first implementation of Intelligent Arrays used the conventional indexing scheme used with multi-dimensional dynamically allocated arrays in C:
where for a two-dimensional array m is the number of columns in the array, j is the row (vertical) position and i is the column (horizontal) position. This has the disadvantage of requiring a multiplication for every array access, which incurs a considerable performance penalty on some systems.
The implementation of Intelligent Arrays I developed for Karma uses address offset arrays, one for each dimension of the array. Thus, for a two-dimensional array the following C code is used for accessing data elements:
where x is an array of offsets for the horizontal dimension and y is an array of offsets for the vertical dimension. The values in these arrays may be calculated thus:
This scheme is much faster than the conventional indexing scheme and has the remarkable feature that non-contiguous arrays may be implemented by simply changing the index arrays. This is possible when the array mapping to memory is separable for each dimension. Examples of non-contiguous arrays where this is true are tiled or toroidal arrays. In both these cases, conventional techniques used to access array elements can slow down access several times. Using address offset arrays there is no performance penalty.
Critical to the success of Karma in distributed processing are the integrated communications facilities. As discussed above, the communications infrastructure is spread across four levels of the library. Packages which provide high-level functionality (such as Intelligent Arrays and various sections of the graphics library) are able to call upon the communications facilities. Because a universal interface is provided, packages need only register their specific protocols, leaving the initiation of connections to another agent (such as a connection management tool). Packages are notified when connections are attempted, succeed, have incoming data and when they close.
To support visualization applications, an integrated, portable graphics library is provided. The graphics library makes full use of other major packages in the library (such as the data format and communications infrastructure), allowing seamless integration between computational functions, data distribution and the display system. The graphics library is modelled around an interactive pixel canvas such as provided by the X Window system, but is not limited to X (it also supports the XGL library). Built on top of pixel canvasses are world canvasses (which can deal with curvilinear co-ordinate systems), viewable images (mappings of array data to displayed images) and overlay lists (for drawing annotations). Under development is a three-dimensional canvas which will deal with all the mechanics of projecting a data volume onto a two-dimensional (possibly stereoscopic) screen, leaving the applications programmer free to specify viewer orientation and shading algorithms.
I thank Patrick Jordan for his support and lively discussions on the design of Karma.
Norris, R. P. 1994, in Astronomical Data Analysis Software and Systems III, ASP Conf. Ser., Vol. 61, eds. D. R. Crabtree, R. J. Hanisch, & J. Barnes (San Francisco, ASP), p. 51
Gooch, R. E. 1995, in Astronomical Data Analysis Software and Systems IV, ASP Conf. Ser., Vol. 77, eds. R. A. Shaw, H. E. Payne, & J. J. E. Hayes (San Francisco, ASP), p. 144
Gooch, R. E. 1995, ``Visualisation: from Data to Understanding'', WARS 95, Australian Academy of Science National Committee for Radio Science
Gooch, R. E. 1995, ``Astronomers and their Shady Algorithms'', IEEE Visualisation '95