PROJECT: NOAO Pipeline and Time-Domain Science Support --------- OBJECTIVE: Develop the capability to reduce, archive, and distribute high quality data products in near real time, including not only the images, but also the associated catalogs of both stable and transient objects, to provide support for time-domain science on present and future NOAO facilities. --------- RESOURCES: o Data stream from the Mosaic survey programs, including Deep Lense Survey (Tyson et al.) SuperMACHO Survey (Stubbs et al.) Deep Ecliptic Survey (Millis et al.) proposed SN search(es) o OPUS pipeline infrastructure o mscred, DoPhot, SExtractor, ACE, and other extant software o DLS, SuperMACHO, SN time domain pipelines o Interim NOAO Surveys Archive o Project Scientist (Chris Smith) o Software engineering staff (Effort TBD, lead by F. Valdes, and including Rafael Hiriart) to develop additional tools as needed --------- EXPECTED PRODUCTS: o Capability to pipeline process NOAO Mosaic data, at least for the most common modes. This will be of great value to ALL survey teams, in that they will benefit from the capability of reducing their data much more quickly and efficiently. It will benefit ALL of the user community by providing uniformly reduced and documented data products for the NOAO archives. o Development of automated data quality assessment utilities standard DQ flags, and ancillary products (seeing, sky brightness, focus, etc.) o Capability to ingest, archive, and distribute complex data products in real time. The extant time-domain survey science programs will add products of great value to the archive holdings, both in terms of the basic images and their time domain products. o Experience in processing time-domain data "near" the telescope, which is relevant for the LSST system. o Identification and generalization of time-domain products which are scientifically most interesting/useful, and of ways to represent these complex data products in the archive. Examples could include: - object light curves - catalogs/time sequences of asteroids and other moving objects - means to derive deep images from multi-epoch exposures - automated object & transient classifications o Development of advanced query methods to mine time domain datasets, and the associated data structures necessary to support such queries. --------- DESCRIPTION: ... still to be edited... The Pipeline and Time-Domain Science Support (PTDSS) team will develop the necessary knowledge, techniques, algorithms, processes, and software to produce calibrated data products from time-domain survey projects, and the means for the community to discover and preview the more advanced products from the NOAO Surveys Archive. At present we have the hardware and software necessary to produce certain data products from the Stubbs et al. survey, and the general capability to calibrate mosaic data (including the removal of instrument artifacts) with the latest version of mscred. These are enormous assets. What is needed is to invest some creative thought into the kinds of data products that would be of great scientific interest, and of ways of presenting or visualizing them from an archive that are unique (or at least, unusal) and compelling. Having done that, it will be necessary to define methods and procedures to produce such products (and data quality assessments) in an automated way. Various existing tools should be explored to support this data stream, including OPUS; certain new tools and observing procedures may be necessary as well. The latter phase of the project will be to translate the ideas, procedures, prototype systems, etc. into a robust system capable of operating at either NOAO-N or NOAO-S, and of producing products that are ingested into and distributed from the NOAO Surveys Archive. --------- BREAKDOWN OF PROJECT: The project can be broken down into three parts: (1) Real-time Mosaic pipeline with data quality assessment (2) Basic data products and real-time archiving (3) Time-Domain Science Support, including (3.1) Algorithms and tools for detecting sources in real-time (3.2) Data products and archives for time-domain science support (1) We have the knowledge and basic capabilities in hand to pipeline process Mosaic data. There are certainly details which will need to be addressed, but this requires little R&D. (2) We will have to coordinate the process of archiving Mosaic data in real-time with phase 2 of the NOAO Archives effort, including desired data products and QA flags, etc., as well as an automated ingestion process. (3) The Time-Domain science support requires the most R&D. It should be started early, but implemented late in the project, allowing time for evaluation of current TDSS methods and --------- SPECIFICATIONS: (1) Real-time Mosaic pipeline - Mosaic data reduction pipeline, through flat fielding, WCS updating, and fringe correction, in <= the readout time of Mosaic using 16 amplifiers (~100s) - Pipeline management system which requires little or no human intervention during the night - Completely automatic generation of basic data quality flags and inclusion into the data headers (2) Data Products + Archive - Completely automatic ingestion of images into NOAO Archives, with no human intervention except for review of problematic data - Ability to access images and associated data products within, at worst one day, at best one hour, of data acquisition. (3) TDSS - Study and implentation of algorithms and tools for detecting AND classifying sources (static and variable) - Addition of time domain analysis to real time reduction pipeline, producing automated alerts for transients within one hour of data acquisition - Addition of time domain data products into archive, and associated tools for mining --------- PTDSS TEAM: 10-15% Chris Smith (project scientist, mgmt) 20-30% Frank Valdes (Mosaic reduction, pipeline, oversight) 100% Rafael Hiriart (pipeline, archives, inter-process communication) 20-30% Mike Fitzpatrick (GUIs, infrastructure, tools) 20-30% NSA team member (archives) --------- TIMELINE: ... still to be edited ... Within 6 months from the start of the project, it should be possible to define the major kinds of data products that would be of interest to offer in the archive, and the techniques necessary to generate them. It should also be possible to design the pipeline and archive ingest software and procedures that will be necessary, and to implement prototypes. Finally, it should be possible to develop a viable way for the community to discover these data products in the archive, and effective ways of representing these data products through the archive interface. Within a year it should be possible to implement all necessary software to process time-domain data streams, ingest them in the archive, to (pre-)compute and/or catalog all derived data products from such surveys, and to provide tools for visualizing and retrieving the data products from the arc