NOAO Data Products Pipelines

Frank Valdes and Chris Smith

The Vision

We envision all NOAO operated instruments having pipelines to automatically produce and archive scientifically useful data products from the raw observations. The pipelines run at the telescope in near real-time, process taped raw data, perform on-the-fly recalibration in archives, and are exported to the astronomical community. At the telescope the data products are produced for the observer and automatically entered into an NOAO Data Products Archive with suitable proprietary protections. The data products include data quality information. At the telescope this is displayed in real-time for instrument and exposure evaluation. In an archive it allows evaluation of the data. by potential users.

The Mosaic Data Products Pipeline

The Mosaic Data Products Pipeline is the first step towards achieving this vision. It targets a single major NOAO instrument, the Mosaic Imagers. The objectives of this pipeline are to:

The data products to be produced and archived include:

The moving, variable, and unusual color products depend on observing programs that take multiple exposures of the same part of the sky or having an archive of reference images.

The Development Plan

The major elements that must be developed for a Mosaic Data Products Pipeline are:

This list is long and each element is fairly complex. Assuming a modest manpower ( ~2 FTE divided over 3-4 people ) it is not possible to attempt to attack all the elements simultaneously and achieve measurable progress. Instead, we propose to divide the list in two as described below. Timescales are conservative ballpark guesses. Note that some initial work on elements the second stage may overlap completion of the first stage. Also each stage can be broken down into smaller steps with specific measurable objectives.

Stage 1: Mosaic Calibration Pipeline

The first stage targets two primary data products, calibrated mosaic images and data quality information. The methods needed to produce these data products require some additional development but most of the basic concepts are well understood and implementations in IRAF are available. Limiting the data products allows concentration on the elements of:

While there is considerable effort on infrastructure in this stage, a mosaic calibration pipeline producing the two basic data products is a valuable result with significant impact.

The target for a useful first stage near real-time telescope pipeline, including testing, is August 2003 . Components of the pipeline, especially the data quality measurements, can be made available earlier as tasks in the MSCRED package. A prototype pipeline, one which is restricted to specific programs such as the SuperMacho/SuperNovae (SMSN) surveys, may be developed sooner.

Stage 2: Mosaic Catalog and Transient Event Pipeline

The second stage builds on the pipeline and archive infrastructure and calibrated Mosaic data products from the first stage. In this stage we can concentrate on the methods for producing the less well understood data products and algorithms. These include:

The target for a general catalog and transient event pipeline is June 2004 . Standalone modules, such as for difference imaging, and a limited set of data products may appear sooner.

The Impact

WHO AND HOW DO WE IMPACT OUTSIDE SDS?

HOW DO WE IMPACT SDS AND WHAT DO WE LEARN?