Figure 1: Part of the retrieval form for the HST archive.
Figure 1: PS 200 Kb
Next: Archiving TNG Data
Previous: Portable Astronomical Scheduling Tools
Table of Contents --- Search ---
PS reprint
Dennis R. Crabtree, Daniel Durand, Séverin Gaudet, Norman Hill
Canadian Astronomy Data Centre, Dominion Astrophysical Observatory,
Herzberg Institute of Astrophysics, National Research Council of Canada
This talk will start by briefly describing the current state of `ground-based archives' (a term which I will use instead of the more cumbersome `archives of data from ground-based observatories'). The archives of data archives of the CADC will be discussed with the goal of presenting an architecture which could be expanded to include other ground-based archives. This will be followed by a list of requirements for the ideal ground-based archive which the new large telescopes such as Gemini and the VLT are hoping to build. Specific attention will be given to the role of automatic data reduction and including calibrated data in ground-based archives.
Finally, a short discussion on the mechanisms by which ground and space-based archives, data centers such as NED and the CDS and electronic publication and bibliographic services could communicate to provide a seamless transition from one service to another will be described.
When ground-based observatories were using photographic plates they were in a sense archiving their data automatically. The plates were a permanent storage media and with a simple catalog one could easily find data for a particular object or section of the sky. The advent of electronic detectors changed the situation dramatically and ground-based observatories stopped archiving their data. At about the same time, space missions started and the policy introduced, mainly by NASA, was to save this unique and valuable data so that the maximum scientific benefit could be extracted from the data.
These archives of data from space missions, while not inexpensive to build and maintain, have in general demonstrated that archives are scientifically useful. The situation with regard to the technology and approaches to archiving have changed significantly so that many, if not most, of the larger observatories have either started archiving data or have expressed interest in doing so.
It is worthwhile to examine the differences between ground- and space-based observatories and how this affects the development of archives. It is also interesting to examine the operating modes of traditional ground-based observatories, contrast this with the observing modes suggested for some of the new 8--10 meter class telescopes, and study the impact that observing modes have on astronomical archives.
Another interesting question is that of whether it is feasible and worthwhile to store calibrated data in ground-based archives. This is the current practice but recent work on the Hubble Space Telescope archive (Crabtree et al., this volume) suggests that it may not be necessary to store calibrated data.
All of these issues will be examined in this paper. As well, a general framework, based upon existing tools and approaches, for creating a new archive for a ground-based observatory will be put forth.
There are many differences, both in the environment and in the way telescopes are operated, between ground- and space-based telescopes. These not only affect the type of data acquired but also have interesting implications for the archiving of data. In general, space-based telescopes operate in a much more stable environment than ground-based telescopes. They have much more stable instrumentation, in the sense that instruments are not being added and removed from the telescope, and the processes for operating the telescope are standardized and not under the control of the astronomer.
The stability of the instrumentation for space-based telescopes implies a much more stable calibration environment than for ground-based telescopes. This means that the telescope and instrumentation are characterized for a much longer period and that calibrations are shared between programs. This is to be compared with ground-based telescope where each observer/project does all of the calibration they need for their data. This allows space-based observatories to establish procedures for taking the data and obtaining standard calibration procedures.
These differences make the problem of archiving data from a ground-based telescope different, and I would say somewhat more difficult, that from a space-based telescope.
However, the new observatories such as the VLT and Gemini will be operated in different modes, very similar to space-based telescope, which will actually make the archiving of this data easier than for traditional telescopes and traditional observing modes.
The requirements for Gemini or the VLT imposed by new observing modes such as queue scheduling are actually the same requirements needed to produce a scientifically useful archive. Compare an archive user to an observer who has been queue-scheduled at one of these new telescopes.
It is worthwhile to examine the Hubble Space Telescope Archive and compare it to the requirements of an ideal ground-based archive. The HST archive is probably the most capable and most useful archive yet developed and serves as a good goal for ground-based archives.
The HST archive is a complete record of the HST mission. Engineering, raw and calibrated data are all saved on optical disk. Not only is every bit of information saved, but it is saved on a media which is stable and has a long lifetime. Also, the data is fully described and this information which includes the telescope, instruments, spacecraft, proposal, etc., is stored in a relational database system.
While the HST archive contains calibrated data one must examine this issue more closely. Is this the calibrated data you want from the archive? While current archives of ground-based data do not contain any calibrated data, the calibrated data which is available from the HST archive was calibrated at the time of observation using the current software and calibration files. Is this the best available calibrated data?
One of the strong points of the HST archive is the fact that information on which calibration data is best to use is kept in the database. This information is updated as better calibration files are produced. Thus, it is very straightforward to query the database and find out which are the current best reference files to use for a particular observation. This, and the fact that the calibration S/W is freely available, make it feasible to recalibrate data from the archive using the best calibration files and the best S/W at the time of the request rather than the time of the observation. The CADC provides this service for users of the Hubble Archive (http://cadcwww.dao.nrc.ca/).
Rather than focus on calibrated data in an archive, I would suggest that archives should be able to produce recalibrated data on demand. This requires a reliable calibration pipeline and a database containing pointers to calibration files. If one can produce a reliable calibration pipeline like the CALXXX routines within STSDAS, then this software could be used at the telescope by astronomers reducing their data and for recalibration of data from an archive.
In this section we will examine the steps needed to implement an archive for a ground-based telescope, look at what some of the current archive sites are doing and suggest a set of building blocks which observatories could use to simply and efficiently construct an archive. Rather than having observatories develop their own solutions and software, it makes more sense to use standard pieces of software which already exist.
The first requirement in any archival system is to reliably save the data coming from the telescope. At CFHT we have a script which manages data flowing from the telescope and writes it to optical disk. La Palma has a similar approach to saving the data except that they write data to Exabyte tape. NOAO/KPNO has developed a system called Save the Bits which uses the lpd daemon to write data to Exabyte tape. This mechanism is very robust and fits very easily into any data acquisition system. Save the Bits is now used at both Keck and Lick and will be used at the VATT. In fact, CFHT is likely to change over to Save the Bits in 1996. We would recommend that an observatory implementing an archive take a serious look at Save the Bits for saving data from the telescope. Once the data is on Exabyte it can be transferred to some other media such as CD-ROM.
Figure 1: Part of the retrieval form for the HST archive.
Figure 1: PS 200 Kb
An archive also needs a catalog system which allows for efficient storage of information describing the observations, telescope, instrument, etc. This includes the information stored in the header (FITS) of the data plus other information such as weather, observer notes, etc. which may or may not be included in the header. The content, completeness and accuracy of this information is critical as it describes the archive and is used to search for particular observations. A relational database system fits these needs very well. The CADC, STScI, UKIRT and others use Sybase, a commercial system, but essentially any SQL-based relational database system will work. Options include Sybase, Oracle, Postgres'95, Msql, etc. This layer, which contains information describing the data and where to find it, and which is used by users to search for data is crucial to the operation of an archive.
Finally, there needs to be some sort of system by which users can search for data of interest, request this data from the archive and retrieve it. Thus far proprietary systems have been favored for accessing the various archives. However, the rapid development of the World Wide Web has changed the approach and we predict that most, if not all, custom interfaces to network resources will be replaced by Web interfaces within two years! Archive systems are no exception and in fact the CADC has already moved access to the Hubble and CFHT archives to the Web (see http://cadcwww.dao.nrc.ca/). Samples of this interface can be seen in Figures 1 and 2. In addition, the UKIRT archive and the JCMT archives are also available.
Figure 2: A sample results page for the HST archive.
Figure 2: PS 189 Kb
This Web access to the archives is mostly Perl-based and uses public-domain software such as Sybperl to access the database. Again, we have used available public domain software as much as possible to accomplish our goals.
Archives for ground-based observatories present different challenges from archives for space-based ones. However, those of us building such archives can learn from the experiences of both space-based and previous ground-based archives. Some key points to consider:
We believe that a global system of archives and other network services built upon common building blocks, reuse of code and approaches and linked via the Web is possible. This would provide a very user friendly and powerful interface to such resources for astronomers world-wide.
We are grateful to the Canadian Space Agency for the continued support of the Hubble archive at the CADC. This support allowed us to broaden our horizons and bring archiving to ground-based telescopes.