Figure 1: The relationships among the various on-line archives and data manipulation tasks.
Figure 1: PS 7 Kb
Next: The Aladin Project: Status Report
Previous: Archives of Data from Ground-Based Observatories
Table of Contents --- Search ---
PS reprint
Fabio Pasian
Osservatorio Astronomico di Trieste,
I-34131 Trieste, Italy
In this paper, the archive system at the TNG is described in terms of archiving facilities, production of hard media and exportable database tables, on-line technical, calibration and transit archives, interaction with the quick-look utilities for the different instruments, and data access and retrieval mechanisms. The interfaces of the system with other TNG subsystems are discussed, and first results obtained testing a prototype implementation with a simulated data flow are shown.
The TNG (Telescopio Nazionale Galileo) is a 3.5 meter telescope which will be operating at La Palma at the end of 1996. The project, headed by Prof. C. Barbieri, involves the whole Italian astronomical community, under the coordination of the Observatory of Padova.
Optically and mechanically, the TNG can be basically considered as derived from ESO's NTT, while it is original from the instruments and control points of view. The instrumentation plan includes an optical imager (OIG), an infrared imager and spectrograph (GIRIS), a low-resolution spectrograph (LDS), and a high dispersion spectrograph (SARG). Adaptive optics are planned to be installed quite at an early stage. A seeing monitor (DIMM) and a meteo station will provide their data to complement at all times the ambient data, hence giving complete information on the observing conditions.
One of TNG's most advanced features is the telescope and instruments control environment, tightly connected to data handling and archiving. Data are planned to follow a predefined path, where the information is handled and stored homogeneously, and the integrity and completeness of data are preserved.
The mode of operation for TNG will be based on an end-to-end data flow model, and on assisted observing, with the possibility of flexible scheduling. These up-to-date characteristics will allow TNG to become one of the first telescopes of ``The New Generation'', thus yielding another meaning for its acronym.
Refer to Barbieri et al. (1994), Fusi Pecci et al. (1994), Pucillo (1994), di Serego Alighieri et al. (1994) for additional information on TNG, its instrumentation, its control software, and its end-to-end model of operations. A TNG WWW Home page is also available.
This paper will concentrate on the design and the implementation of the data handling and archiving system for the TNG, and on the on-line archives which will be available at the TNG for the observers, the instrument teams and the observatory staff.
The requirements for the TNG archives have been identified by two different working groups set up by the Italian Council for Research in Astronomy (CRA). The goal is to implement an archiving system giving astronomers not only an efficient data storage system, but also a tool for astronomical research, and a tool to monitor the performance of the telescope and instruments. The documents produced by the working groups are available under a TNG archives WWW Home page.
The requirements on the TNG archives are listed in Benacchio et al. (1993) and in Benacchio et al. (1994). The basic ones are:
The need to build a complex archive system has been identified, composed of: a General Archive, accessible by the scientific community at large, and where all data are stored; a Transit Archive, where data are temporarily stored before shipment to the General Archive; a Technical Archive, holding the data needed for maintenance and monitoring of telescope and instruments; a Calibration Archive, holding a set of calibration data useful for the on-line reduction of scientific frames.
As discussed in Pasian & Smareglia (1995b), the critical issues related to the implementation of an archive for a ground-based observatory, and specifically for the TNG, are:
As a result of the requirements described in the previous section, a number of implementation principles are being followed, and are described in this section.
The archiving activity is tightly connected to, or better, is completely integrated with the TNG control software (and specifically with the WSS, the higher-level software running on the control workstations). As mentioned in Pucillo (1994), all control workstations run the same code, but behave differently according to their startup configuration and/or to the messages they receive during operations. Data are distributed using a unique replicated database structure divided into a number of subsets, each maintained by a different workstation: the access to data not pertaining to the local system is guaranteed by a message exchange mechanism. In the design of the TNG data archiving system (Pasian 1995), the archive computers are linked to the telescope LAN and, at boot time, they run the WSS processes, thus sharing the code and the real-time data base structure. As a consequence, archiving is integrated with data handling and instrument control, thus ensuring data integrity.
Figure 1: The relationships among the various on-line archives and data
manipulation tasks.
Figure 1: PS 7 Kb
Of the system of archives identified by the TNG archives working groups, the following are available at the telescope:
Figure 2: The logical flow of scientific data, from the Instrument Workstation
(IWS) to the Archive Server (AS), where they are temporarily stored on the
Transit Archive and, after association with the proper calibration files,
saved for shipment to the General Archive. Telemetry (technical) data follow
the same logical path, although handled by different tasks.
Figure 2: PS 9 Kb
Due to its integration with the TNG control software, the archiving activity is to be considered as a section of telescope operations. The hardware and software system supporting archiving and the on-line archives has been therefore designed as capable to cope with system failures, and able to automatically reconfigure to continue supporting operations. Two computers, the Archive Server (AS) and the Archive Server Backup (ASB), are available; they are interchangeable, since each is equipped with redundant peripherals, and the database is mirrored between the two. This design allows reconfiguring the system in the case of failures without stopping the data acquisition and telescope control mechanisms.
The software managing the archives at the TNG is divided in two logical sections: a Handling and Archiving Tool (HAT), and a DAta Retrieval Tool (DART). The HAT is responsible for handling the data flowing from the instruments and ingesting them in the Transit Archive. Both technical and scientific data follow the same logical path from the telescope to the archive, although using different software processes. The HAT also associates in the database the files corresponding to scientific observations with the related calibration files by means of a Data Organizer (see, e.g., Péron et al. 1994), and produces physical media and database tables to be shipped to the General Archive. The data flow for scientific data is shown in Figure 2. Appropriate tools to update the contents of the Technical and Calibration Archives, and to ingest auxiliary information originated by the observatory, are also included in the HAT.
The data retrievable from the archives at the telescope are information on instrument response, trends, calibration procedures, and ``standard'' calibration files. Access to these facilities will be restricted to observers and observatory staff, although there is no technical obstacle in allowing a wider access when no sensitive operations (e.g., observing) are being performed. The tools to retrieve data are grouped in the DART software. They have been designed to be used at the TNG, but in principle they can be exportable to the site selected to host the General Archive. The scheme for data retrieval from the archives at the TNG is shown in Figure 3. Through a suitable interface (a GUI, a command interface, or an API) the user can formulate queries to access the database, and subsequently gather information on the files to be retrieved; extracted files may undergo an unscrambling phase if they are telemetry data, so that results are presented to the user in an understandable way.
Figure 3: Scheme for data retrieval from the archives at the TNG.
Figure 3: PS 6 Kb
The possibility of performing ``quick-reduction'' on data while observing at the telescope in order to evaluate the quality of the scientific data being acquired has been taken into account for the TNG (Pasian & Pucillo 1995). In this case, DART processes combine with the Data Organizer to allow the proper calibration files (either just acquired, or the ``best possible'' available in the Calibration Archive) to be selected and loaded on the Instrument Workstation, where a batch-like procedure (``pipeline'') will process the data. The possibility of implementing quick-reduction is left to the consortia developing the individual instruments. Should the relevant consortium decide to provide quick-reduction of its data, the system and data interfaces between the DART process providing the data and the quick-reduction software need to be defined for every specific instrument.
The GUI currently developed is based on HTML (see Pasian & Smareglia 1994, 1995a), and is able to transform the contents of the fields filled on the HTML page (see Figure 4) into constraints to be submitted to the database as SQL queries. Additional capabilities are:
Figure 4: Querying the database by specifying constraints in fields available
on a HTML interface.
Figure 4: PS 142 Kb
Figure 5: Building a view on the database via a HTML interface. The available
tables are displayed, and the desired fields can be selected by clicking the
proper buttons. The view shown in Figure 4 has been built with
this HTML page.
Figure 5: PS 184 Kb
To ensure that the General Archive of TNG data will be properly populated, a clear interface between the archives available at the TNG and the General Archive has been identified. The following steps are envisaged:
Figure 6: Output from a query. On-line data can be retrieved by clicking
the retrieve button on the HTML interface.
Figure 6: PS 160 Kb
In December 1994, the working group nominated by CRA officially issued the documents on TNG Archives; since, the system is being implemented. Since August 1995, a prototype is running on the OAT infrastructure and simulates downlinking of files from the IWS, the storage of data on magneto-optical disks, database updating, database access, and creation of FITS files. At the time of the conference (October 1995), the archive hardware was only partially delivered, and completion of the hardware system is expected in November.
In December 1995, the prototype will be running on the archive hardware within WSS, and will also include database maintenance procedures, and the creation of an observation summary table. In March 1996, it is envisaged that DIMM data (installation is foreseen in November) will be available in the form of a browsable archive. In June 1996, a tested complete prototype will be available, and will be shipped with the rest of the control hardware on the mountain. The final version of the software is planned for December 1996.
As for the General Archive, a decision on its location is expected in early 1996; the availability of initial General Archive services can be expected by December 1997.
The author is indebted to R. Smareglia for the skills and competence shown in the implementation of the HAT and DART systems, and to M. Pucillo for many invaluable discussions. Exchanges of ideas with M. Albrecht, A. Balestra, A. Baruffolo, L. Benacchio, P. Benvenuti, C. Bonoli, D. Crabtree, S. di Serego Alighieri, D. Durand, S. Gaudet, M. Péron, B. Pirenne, D. Ponz, G. Russo, P. Santin, C. Vuerli, and W. Wamsteker have influenced this work, and are gratefully acknowledged. The TNG data flow is being simulated with data from the EMMI instrument of the ESO NTT telescope.
Benacchio, L., Bonoli, C., Brocato, E., Cutispoto, G., Ferraro, I., Pasian, F., Pucillo, M., di Serego Alighieri, S., & Stirpe, G. M. 1993, Osservatorio Astronomico Padova-Asiago, (in Italian)
Benacchio, L., Bonoli, C., di Serego Alighieri, S., & Pasian, F. 1994, Osservatorio Astronomico Padova-Asiago
Fusi Pecci, F., Stirpe, G. M., & Zitelli, V., eds. 1994, Osservatorio Astronomico Bologna
Pasian, F. 1995, Galileo Project Technical Report, 51
Pasian, F., Pirenne, B., Albrecht, R., & Russo, G. 1993, Exp. Astronomy, 2, 6, 377
Pasian, F., & Pucillo, M. 1995, in Calibrating and Understanding HST and ESO Instruments, in press
Pasian, F., & Smareglia, R. 1994, Int. Jour. of Mod. Phys. C - Physics and Computers, 5, 817
Pasian, F., & Smareglia, R. 1995a, in Astronomical Data Analysis Software and Systems IV, ASP Conf. Ser., Vol. 77, eds. R. A. Shaw, H. E. Payne, & J. J. E. Hayes (San Francisco, ASP), p. 68
Pasian, F., & Smareglia, R. 1995b, Exp. Astronomy, in press
Péron, M., Albrecht, M., & Grosbø l, P. 1994, in ESO Conferences and Workshop Proceedings, Vol. 50, Handling and Archiving Data from Ground-Based Telescopes, eds. M. Albrecht & F. Pasian, 57
Pucillo, M. 1994, in ESO Conferences and Workshop Proceedings, Vol. 50, Handling and Archiving Data from Ground-Based Telescopes, eds. M. Albrecht & F. Pasian, 77
di Serego Alighieri, S., Benacchio, L., Bonoli, C., & Pasian, F. 1994, Osservatorio Astronomico Padova-Asiago