1. The primary objective is to produce and distribute, both immediately to observers and via an NOAO archive, high quality data products from raw observational data in a automated manner; i.e. to create data products pipelines. The types of data products include:
The data products are produced:
The difference between data products produced immediately and those produced the following day is whether the data is calibrated using the best calibration available at the beginning of the night or with updated calibrations derived from data taken during the night. The differences in the calibrations will be minor if recent calibrations, such as from the previous night, are used for the real-time calibrations. The real-time data products are in demand for providing immediate data quality accessment and for programs requiring further real-time analysis, such as detection of transient and solar system sources.
2. It has long been an objective of NOAO to export useful software developed at the Observatory to the community, particularly those obtaining data with NOAO facilities. Therefore another objective of this project is to export the pipelines to interested users. This means that portability and ease of installation of the software must be considered.
3. The data products will be used to populate NOAO archives and an NVO node with data from NOAO telescopes.
4. The automatic production of data products, particularly the
derivative ones such as catalogs and unusual object alerts, will be
required for the large volume of data to be produced by an LSST. The
pipelines developed in this project are potential prototypes for
the LSST pipelines.
Description
The project will develop pipelines to take raw data from NOAO supported telescopes and automatically produced uniform data products with data quality information useful to observers and archive researchers. The pipelines will run at the telescope for immediate delivery to observers and an NOAO archive. The same pipelines will also run on archived raw data such as from Save-the-Bits or user's tapes.
The types of data products will include basic calibrations and derived information such as catalogs and extracted spectra. Data products involving multiple exposures will also be produced. These include stacked exposures, alerts for variable sources, and alerts for unusual objects.
The pipelines running at the telescope during the observations provide several new capabilities. For the observer the data quality monitoring will be important along with the basic calibrated data. For the observer and the community the variable and unusual object alerts will be an important new data product from NOAO supported telescopes.
The data products pipelines will be developed in stages. The stages
differ by instrument, types of data products, and level of completeness.
For instance the first stage objective is a prototype data products
pipeline for basic calibration of individual exposures from the NOAO Mosaic
Imagers.
Development Plan
To achieve all the objectives requires a long term effort proceeding in stages. Each stage builds on the results of the previous stage, though some aspects of later stages may begin before the earlier stage is completed. The stages and approximate timeframe for completing them are:
The Prototype Mosaic Calibration Pipeline will produce basic calibrated individual exposures from the NOAO Mosaic Imagers. The pipeline will operate at the telescope to produce real-time (within a few minutes) and fully calibrated (during the following day) exposures. It will also be used away from the telescope to process taped raw data. It will be able to automatically feed the calibrated exposures into an NOAO image server archive.
The basic calibrations include removal of instrumental effects -- CCD electronic signatures, response, internal reflections (pupil), and fringing -- and astrometric calibration. Data quality information, such as seeing and transparency, will also be measured and included with the data products, in the pipeline monitoring, and to the observer.
The purpose of the prototype is to explore pipeline architecture issues, real-time issues, user interface issues, the generality of the processing capabilities for a variety of programs, and documentation of the data quality. It also provides pipeline capabilities at the telescope as soon as possible, with a target of the 2003A semester.
Though it is a prototype the software will be of sufficient utility and portability that observers at the telescope may choose to use the pipeline and data products for part of their data reductions and as input to their own pipelines. This may be both at the telescope or at their institutions. Since the primary objective is a telescope pipeline within a short time frame the prototype pipeline may not be easy to install or customize. This aspect will be addressed in the next stage of the project.
The connection of this pipeline to an archive is a lesser goal in this
stage, though importing calibrated mosaic exposures into a basic image
server archive is straightforward and will be included. This will be of
most value for data from the NOAO survey programs producing large numbers
of consistently obtained mosaic exposures.
2. Mosaic and Transient Event Data Products Pipeline
This stage extends the prototype processing pipeline capabilities to produce data products from multiple exposures and comparison with previous imaging that overlap the exposures on the sky. The data products will include dither stacks, detection of transient objects, and catalogs.
In addition to providing transient event data products to the observer and quickly to an archive, aspects of the archive for providing new types of data products beyond single mosaic exposures will be emphasized. One aspect of this is development of advanced query methods to mine time domain datasets.
Pipeline architecture issues, such as ease of configuration,
control, and installation, will be refined.
3. Generic Imaging Pipeline
This stage extends the mosaic data products pipeline(s) to other
imaging instruments such as individual CCDs and single and mosaic IR
imagers.
4. Spectroscopic Pipeline
The basic pipeline technology, including common instrumental calibrations which are the same or similar to imaging data, will be combined with existing and new spectral reduction tools to automate spectroscopic processing and produce spectroscopic data products. This will be developed first for the most heavily used NOAO supported spectroscopic instruments (Cryocam/MARS, Flamingos, etc). Since there are a variety of forms of spectroscopic data -- slit masks, fibers, IFUs -- this stage will likely be a set of related pipelines for the different types of raw spectral data.