NICMOS Calibration Pipeline---A Collaborative Project Between IDT and STScI

Next: ASC Data Analysis Tool Architecture
Previous: Creating an Object-Oriented Software System---The AIPS++ Experience
Table of Contents --- Search --- PS reprint

Astronomical Data Analysis Software and Systems V
ASP Conference Series, Vol. 101, 1996
George H. Jacoby and Jeannette Barnes, eds.

NICMOS Calibration Pipeline---A Collaborative Project Between IDT and STScI

H. Bushouse, J. MacKenty, C. Skinner, D. Axon

Space Telescope Science Institute, Baltimore, MD 21218

E. Stobie, G. Schneider

University of Arizona, Steward Observatory, Tucson, AZ 85721

Abstract:

The Near Infrared Camera and Multi-Object Spectrometer (NICMOS) is a second-generation instrument to be installed on the Hubble Space Telescope (HST) during the second servicing mission in early 1997. The Space Telescope Science Institute (STScI) and the NICMOS Investigation Definition Team (IDT) are collaborating on the development of a data calibration pipeline where not only algorithms, but also code will be shared. STScI is developing its pipeline in the IRAF environment, while the IDT pipeline is being developed in an IDL environment. Code common to both environments is written in ANSI C. We describe the methodology used for this project, as well as hurdles overcome in making it work.

1. Instrument Overview

NICMOS will offer direct, coronagraphic, and polarimetric imaging, as well as GRISM spectroscopy in the 0.8--2.5m wavelength range. Three cameras can operate simultaneously, yet independently, on adjacent non-contiguous sky fields at different magnifications. Cameras 1 and 2 provide diffraction-limited imaging to 1.0 and 1.75m, respectively, while camera 3 provides wide field imaging. Three short-wavelength and three long-wavelength polarizing filters are available in cameras 1 and 2, respectively. Camera 3 contains three GRISMs for slitless multi-object spectroscopy.

Thermal background signals will be significant at wavelengths greater than 1.8m. Measurement of the background will be accomplished by taking patterns of offset images located around the target of interest. Field displacement for performing these patterns will be done by either repointing the telescope or moving an internal field offset mirror.

The NICMOS detectors can be read out in four different modes:

ACCUM: A single image is returned from a single integration period. Multiple non-destructive reads at the beginning and end of the exposure can be averaged in order to reduce read noise by . This mode is also sometimes referred to as ``MIF'' (Multiple Initial Final reads).
MULTI-ACCUM: Multiple images are returned from intermediate non-destructive reads as the integration progresses. Subsequent ground processing of the images can provide cosmic ray and saturation detection and rejection. MULTI-ACCUM readout mode provides a large effective dynamic range.
RAMP: Similar to MULTI-ACCUM, but a single image is returned from on-board processing of multiple intermediate readouts. The maximum number of readouts is 254, and are equally spaced throughout the exposure. The on-board processing allows for cosmic ray and saturation detection and rejection. The variance and the number of samples used to compute the final pixel values are also returned as images.
BRIGHT OBJECT: A single image is produced by sequentially exposing and reading individual pixels in each detector quadrant (the four quadrants are operated simultaneously in parallel), with integration times per pixel less than 0.262 seconds.

2. Data Processing Methodology

2.1. Software

The NICMOS data calibration pipeline supported by STScI will be used in the IRAF environment, while the IDT will support a version to be used in IDL. In order to allow for a collaborative software development effort it is necessary to adopt a common, high-level language, at least for the algorithmic portions of the calibration software. We have chosen to use ANSI C, along with C language bindings to both the IRAF/STSDAS and IDL environments.

The use of IRAF and IDL-specific routines will be confined to basic data I/O operations and will be isolated from the calibration algorithms that work on the data. The IRAF and IDL I/O routines form a mapping between external storage media and C data structures that are used by the calibration algorithms (see ``Data Structures in STIS and NICMOS'' by A. Farris elsewhere in this volume). The calibration algorithms will operate only on the C data structures.

Isolation of the data I/O routines from the algorithms requires all necessary data to be in memory for processing. For NICMOS observations, this typically amounts to 5 or 6 images containing calibration information (e.g., flat fields, dark current, etc.), plus the science image itself, for a total of 15 Mbytes of data in memory.

2.2. Data File Format and Contents

The run-time and archive data file formats for NICMOS will be FITS with Image and binary Table extensions (see ``A FITS Image Extension Image Kernel for IRAF'' by N. Zarate and P. Greenfield elsewhere in this volume). The files for image-mode data will consist of 5 FITS image extensions for each exposure and will contain:

the ``science'' image returned from the instrument,
an ``error'' image giving estimated uncertainties in the science image,
a ``data quality'' image containing bit-encoded flags to indicate known problem conditions for each pixel in the science image,
a ``number of samples'' image indicating the number of samples used to compute each science image pixel value, and
an ``integration time'' image with the effective exposure time per pixel.

This image quintuplet will be treated as a unit throughout all processing steps. The data quality and error values will be propagated and updated at each processing step, and the data quality flags will be used to exclude bad pixels from processing. The same data file format and organization will be used at all stages of the pipeline, which will allow the processing to be re-entrant. Processing may be stopped and restarted at any point for purposes of examining intermediate results, and the results can be reinserted into the processing mid-stream.

Multiple exposures that are part of a pattern of background images will be logically grouped into an ``associated set'' and treated as a unit starting at the proposal stage, and all the way through the scheduling, observing, processing, and archiving stages.

3. Calibration Processing Steps

3.1. Pipeline Phase 1

The following steps will be performed to remove the instrumental signature from each individual exposure (each readout for MULTI-ACCUM):

Subtract initial readout for MULTI-ACCUM images. This is performed on-board for other readout modes.
Flag known hot/cold pixels.
Estimate noise in the science image.
Correct for non-linear detector response. The response will be characterized for each pixel using polynomials of 2nd or 3rd order, where one polynomial is used for each one of the low, middle, and high-end response regions.
Subtract detector dark current. If the exposure time does not match an existing image in the library of darks, a suitable dark image will be constructed in real-time by interpolating within the library.
Flat field correction.
Flux calibration, using STSDAS Synphot-generated sensitivity factors.
Cosmic ray and saturation identification or rejection.

3.2. Pipeline Phase 2

These steps will be performed on an entire associated set of images in order to estimate and remove the background signal and combine the images:

Compute the sky background by averaging all images in the pattern.
Subtract the sky background from the individual images.
Create a composite target image by registering and mosaicing the background-subtracted images.

3.3. Post-pipeline Processing

Some necessary steps are not easily implemented in a non-interactive pipeline environment and therefore we expect to have the following standalone tasks:

Geometric and astrometric corrections, when necessary for the science. This will include removal of optical distortions, inter-camera registration, and reduction of coordinates to the Guide Star frame of reference.
GRISM spectral extraction, including flat field corrections, as well as wavelength and flux calibration.
Polarization processing, including computing Stokes parameter images.

4. Data Products

All NICMOS data will be stored in FITS files using image and binary table extensions. Raw data sets will be composed of the science data file, which will contain the science, error, data quality, samples, and time image quintuplet for each exposure, and a support data file which will contain schedule information and instrument engineering data. The pipeline output products will include a calibrated science data file for each exposure (produced by the phase 1 pipeline), in the same format as the raw science data file, and a single science data file containing the background-subtracted, composite target image from an associated set (produced by the phase 2 pipeline). Post-pipeline products should include background-subtracted images from non-standard pattern sequences, extracted spectra from GRISM observations (stored in FITS binary tables), and Stokes parameter images from polarimetry observations.

5. Calibration Files

Calibration reference files needed by the pipeline include a bad pixel mask image, a detector read noise image, dark current images covering a broad range of exposure times, flat field images, non-linearity coefficient arrays, and a sensitivity factor table for flux calibration. Dark current images will be obtained on-orbit by observing an opaque, low-emissivity blank in each filter wheel (maintained at 155 K). Flats will be obtained on-orbit by combining observations using a given detector/filter combination and combining them into a ``super-flat''. Sky and/or earth (``streak'') flats may be used as appropriate. Internal calibration lamps will allow monitoring of the stability of the flats.

Next: ASC Data Analysis Tool Architecture
Previous: Creating an Object-Oriented Software System---The AIPS++ Experience
Table of Contents --- Search --- PS reprint

Wed Jul 3 07:30:55 MST 1996