Next: AIPS++ and the GBT
Previous: Lessons Learned: The Object-Oriented Design of StarView
Table of Contents --- Search ---
PS reprint
Nelson Zarate, Perry Greenfield
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD
21218
The FITS standard (see NASA 1995) allows the appending of any number of optional Standard Extensions to a simple FITS file which consists of the usual Header and Data Unit (HDU). These Standard Extensions also consist of Header and Data Units. These Standard Extensions may be of different types; the type is specified by the value of the XTENSION keyword in the extension header (e.g., IMAGE or TABLE). When a FITS file contains such extensions, the first HDU is referred to as the Primary HDU and the following HDUs as Extension HDUs.
By using extensions of type IMAGE, it is possible to combine multiple images in one file (along with table extensions for that matter). Different extension headers can have different keyword sets and the different extension data arrays may be of different dimensionalities, sizes, and data types. In this respect it is much more general than the ``group'' type format employed by current HST data products. These GEIS (Generic Edited Information Set) files allow multiple images, but these images share the same keyword values found in the header and must all be of the same dimensionality, size and data type. Group Parameters are used to specify keywords whose value change between images; however, these Group Parameters must exist for all the images and cannot be dynamically added to a given image without changing the structure of the whole file.
The currently used GEIS files have embedded machine dependencies which complicate the use of such files on different machines since conversion programs are needed to move the files from one type of computer to another (e.g., VAX and Sun). FITS files, on the other hand, have machine independent data formats. Because the FITS format is widely used, its use results in data analysis system interoperability also.
The acceptance of FITS as an archival format means that no data format conversion is necessary between HST archives and HST data processing systems.
The ground-based software being developed for the two new instruments to be installed in HST during Servicing Mission in 1997, STIS and NICMOS, will generate images and spectra with much associated data. Rather than deluge the observer with a huge number of files, it was agreed that packaging related data products together in one file would simplify the management of the data products for the observer.
The FITS file structure is inherently sequential in design. Successive accesses of a FITS Image Extensions file that contains large numbers of extensions will result in a significant performance penalty because of the repeated scans through all the headers prior to the referenced extension. To avoid this the kernel constructs an internal directory of the extensions with pointers to the offsets for the headers and data units. The current version of the kernel fills the index entries for all headers examined so that subsequent accesses to any of those headers can determine their offset immediately, and any access of a following header not yet indexed can begin at the last header indexed. The only performance penalty incurred is when files are accessed out of order, and in the worst case the total penalty is the time it takes to scan all the headers once, regardless of how many extensions are accessed.
The current implementation caches the previous FITS files index tables so that different IRAF image opens (even from different tasks in the same executable) can access the index table created by previous image open calls to different extensions (logically treated by IRAF as a separate file). The index tables are identified by the full IRAF filename. It is possible that a previously cached index table may be rendered invalid by a user replacing a file with a different one outside of the IRAF environment (or otherwise modifying the file). For that reason, the kernel always verifies the modification time for the current file to be consistent with the corresponding cache entry.
Asking a user to access extensions in a FITS file solely by extension number is asking for frustration and trouble, particularly in the case where there are repeated sets of extensions. A mechanism for identifying extensions by name is crucial. Two standard FITS keywords for extensions serve this purpose nicely. EXTNAME and EXTVER are used in combination to identify extensions. EXTNAME takes a string value whereas EXTVER takes an integer value. One can give all the image extensions different values of EXTNAME and thus not rely on the value of EXTVER, or give some image extensions a common name to represent an image type (e.g., SCI) and use EXTVER as an effective array index (e.g., SCI,3 for third SCI image). Even though EXTVER can be used as an array index, the values of EXTVER need not reflect the order in the file or form a complete sequence of integers. Use of EXTNAME and EXTVER is constrained to the extensions only and cannot be used to identify the Primary HDU. The current FITS kernel can use either extension names or numbers to access extensions.
The kernel will by default try to prevent the user from creating duplicate extension names in the same FITS file to prevent confusion. But it will not assume that all extension names are necessarily different.
Always requiring users to specify extensions as part of the filename specification, especially when there is an array or image of primary interest in the file, is bound to make the data format an unpopular one. For this reason, when the user specifies a filename only, without an extension specification, the kernel will presume the first non-null image in the file is being referred to (the Primary Data Unit if it exists, otherwise, the first non-null image extension).
The relationship of the keywords in the Primary Header Unit (PHU) to the Extension Header Units (EHU) is nowhere defined in the FITS standard. Many treat the headers as independent (and most FITS readers adopt this interpretation), whereas some have presumed that the PHU keywords apply to all the extensions. It is natural to assume that the extensions grouped together in the same FITS file should have a ``global'' set of keywords that apply to all, and hence that the PHU keywords could serve that role. But allowing such a mechanism introduces many potential complexities. This single aspect of the kernel produced by far the most discussion and controversy of any aspect of the kernel design.
After much discussion (within STScI, with NOAO, and with the FITS community in general via sci.astro.fits) it was decided to adopt an ``inheritance'' mechanism for keywords using the following rules:
Rules 1, 2, and 7 can be overridden by use of an option in the image name specification.
Further limitations of the inheritance feature should be noted. If the purpose of using inheritance is to prevent needless repetition of keywords in EHUs it will effectively be negated by the propagation of the EHDU through a single use of standard IRAF program or utility that generates a new image file from the input image file. The ``global'' header and extension header distinction can only be preserved by software that takes special care to disable the automatic inheritance and propagate the headers separately.
Most of the IRAF applications that handle images through the IMIO interface can access and create FITS files and FITS extension with the important point that each IRAF task treats an input file specification as a connection to only one image. It is not possible for an existing general image application to manipulate all of the IMAGE extensions with one file specification; there needs to be one input file specification per image extension. All FITS image data types can be read; all but byte format can be written.
The IRAF image model is basically a single image model. There will be some aspects of dealing with FITS Image Extensions that will either be quite tedious because of the repeated operations necessary to apply some operations to each extension of a file when in principle it could be done all at once (e.g., imcopy), or will not behave as expected (e.g., imdelete). It certainly will be necessary to develop utilities or modify existing utilities to make using FITS Image Extensions more convenient.