Datastream Compression for IRAF Image Display

Michael Fitzpatrick, Doug Tody

IRAF Group, NOAO,¹ PO Box 26732, Tucson, AZ 85726

Abstract:

¹National Optical Astronomy Observatories, operated by the Association of Universities for Research in Astronomy, Inc. (AURA) under cooperative agreement with the National Science Foundation

1. Introduction

CCD technology has progressed to the point where 2K square CCDs are common at most large observatories, and larger mosaics will be coming on-line in the near future. This presents not only a problem in data storage and reduction (where each frame represents 8 MB or more of data), but also in the display and analysis of the data. The situation only worsens in the case of remote observing where the astronomer is physically separated from the detector by perhaps thousands of miles and slow networks inhibit the ability to view the data as it's collected. With the release of PC-IRAF (Tody & Fitzpatrick 1996) increasing numbers of users will be able to work from home over a modem or other relatively low speed network connection. Some sort of data compression is necessary to adequately support image display over slow network connections in such applications.

The current IRAF display server, Ximtool, allows display from an Internet host via sockets, but even for relatively small frames slow connections mean the time to display an image can take minutes. For the remote observer or eavesdropper on such a session what is needed is a way to speed the display of images from a remote client. This can be accomplished most practically by compressing the displayed data at the source so that transmission times are reduced, since for a slow network connection the time spent uncompressing the data is typically much less than the time spent transmitting it. Since the compression schemes being considered can also be used for the disk storage of data we get the side benefit of being able to access compressed images on disk with little extra effort.

Planned Ximtool enhancements include not only the ability to read and write images in a variety of formats (compressed or otherwise), but also a new generalized communications protocol to permit progressive display of compressed images allowing the user to view the image as it's being received. This interface will allow any client application, including non-IRAF clients, to use Ximtool as a display server. Derivatives of Ximtool such as SAOtng should also inherit these capabilities.

2. Compression Algorithms

There are a variety of image compression algorithms in use today. Aside from the lossiness tradeoffs usually made for higher compression rates, an algorithm's practical utility depends on how it is being applied: the type of data being displayed and the relative speeds of the server, client, and network connection. Preliminary tests were conducted on four images: a starfield, a galaxy image, an Echelle spectrum, and a bias frame. Results of these tests are summarized in Table 1. Several distinct (dis)advantages emerged for each algorithm as it was used on each of the test images:

• Bitplane Slicing---The main advantages here are that the compressed images can be quickly calculated, each bitplane slice can be compressed using a simple algorithm, and the image can be displayed as it's being transmitted. The image usually looks complete after the high order bits have been transmitted making this a good method for progressive image transmission. Unfortunately, the compression rates for each bitplane were not as high as expected meaning in some cases the total number of bytes transmitted was higher than sending the uncompressed image. The use of bi-level compression schemes such as JBIG or CCITT Group 4 fax may improve results here however they are also more sensitive to image noise.

• Wavelet Transforms---There are several wavelet transforms available, and much ongoing research in this area, but initially only the 2-D H-transform implemented in the HCOMPRESS package from STScI was tested (STScI is working on a newer version tailored for image display which we will test as soon as it becomes available). This algorithm yields good compression however setting an optimal scale requires some knowledge of the data characteristics. Progressive transmission is possible by decompressing the image after only a small percentage of the wavelet coefficients have been received.

• JPEG---This scheme is primarily used to compress images so that they have the same appearance as the original. For data requiring analysis this can affect quantitative analysis but for image display it can be quite effective. Even at near lossless quality the test images compressed well. The easiest part about implementing JPEG is that the code needed to do the compression is already available in a freely distributed library. New display clients won't need to implement the compression code in order to use Ximtool as a server.

• Fractal Compression---Fractal compression does produce very good compression rates for certain types of images, however the time needed to compute can be excessive making this an impractical approach unless a fast client host is available.

Obviously this is not an exhaustive survey of all techniques available or their relative strengths and weaknesses. What is clear though is that no one algorithm stands out as being the best for all possible uses, some combination of algorithms will need to be implemented.

3. Preliminary Work

As an experiment a prototype display client was written using the HCOMPRESS algorithm (White 1992) and an extension to the IRAF IIS/imtool remote display protocol. Initially the entire image was compressed before transmission but this caused a noticeable delay in display even though the total time required was still shorter than displaying the uncompressed image. A second attempt broke the image into smaller sections for display. Since the Ximtool server could be decompressing one section while the client is compressing the next display times fell by more than half, however the compression rates for each section were not as good as when compressing the entire image.

As a test, the display client was run on a machine in Canada to an Ximtool running in Tucson. The average display time for dev$pix (the standard 512x512 IRAF test image) fell from 55 seconds to less than 8, similar results were found when connecting to a Linux PC over a 14K modem. It is expected that better results can be obtained by using an approach that compresses the entire image but the server incrementally reconstructs the images as it's being transmitted rather than waiting for the entire file. As found by Percival & White (1993) a useful image can be seen after only a small percentage of the data has been received. Alternatively some sort of interlacing can be done as with GIF files. We expect that similar prototypes using other compression algorithms will need to be implemented to effectively judge which combination of algorithms and transmission schemes will be required in the final version.

4. Future Work

Our plan is to first complete the survey of compression algorithms available and decide which are to be implemented. By implementing a variety of compression options, in such a way that they can be used for datastream or disk file compression, we can not only handle a larger number of disk formats but also allow a method to be used that is optimal for a given data frame.

The current IIS/imtool protocol has been in use for client-server image display for years and will continue to be supported for compatibility with older clients. This is being extended to handle the requirements of a compressed datastream. The simplest approach is one in which a ``magic number'' is passed in the header packet indicating that compressed data is to follow. This is the approach taken in the prototype task but is clumsy to use and not very flexible given the requirement for backwards compatibility. A generalized client-server display protocol based on messaging is being devised in which the compression type is just one attribute of the image block being sent, along with WCS information, LUTs, and other display parameters. This will permit not only compressed image transmission but also general interaction of the display server with one or more remote clients.

References:

Pennebaker, et al., 1988, IBM Journal of Research and Development, 32, 721

Percival, J. W., & White, R. L. 1993, in Astronomical Data Analysis Software and Systems II, ASP Conf. Ser., Vol. 52, eds. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes (San Francisco, ASP), p. 321

Starck, J.-L., Murtagh, F., & Louys, M. 1995, in Astronomical Data Analysis Software and Systems IV, ASP Conf. Ser., Vol. 77, eds. R. A. Shaw, H. E. Payne, & J. J. E. Hayes (San Francisco, ASP), p. 268

Tody, D., & Fitzpatrick, M. 1996, this volume

White, R. L. 1992, High Performance Compression of Astronomical Images, available via FTP from stsci.edu