Next: Associating Science Exposures with Calibration Exposures in the CFHT Data Archive
Previous: Transforming Images into Icons to Remotely Retrieve Information from Astronomical Archives
Table of Contents --- Search ---
PS reprint
Wendy Erdwurm, James A. Pintar
National Solar Observatory, National Optical Astronomy Observatories,1
P. O. Box 26732, Tucson, Arizona 85726-6732, USA
1The Global Oscillation Network Group (GONG) is an international community-based project funded principally by the National Science Foundation and administered by the National Solar Observatory. NSO is a division of the National Optical Astronomy Observatories, operated by the Association of Universities for Research in Astronomy, Inc. under a cooperative agreement with the National Science Foundation.
The Global Oscillation Network Group (GONG) project is conducting a detailed study of solar internal structure and dynamics using helioseismology. GONG has deployed a six-station network of sensitive and stable solar velocity imagers located around the Earth to obtain nearly continuous observations of the velocity fields on the surface of the Sun.
Each of the six GONG observing stations produces three, 16-bit, 256 
242 pixel images of the Sun every 60 seconds of sunlight. Each station in the
network produces more than 200 megabytes of data every day. The various
processed data sets will result in an archive of over one terabyte per year to
be stored on approximately 10,000 8-mm Exabyte cartridges and will be managed
and distributed by the Data Storage and Distribution System (DSDS).
Raw data arrives at the GONG Data Management and Analysis Center (DMAC) at the rate of one gigabyte per day. Upon its arrival at the DMAC, the field tape cartridge containing the raw data is copied to another 8-mm Exabyte cartridge and sent to off-site storage. The field tape copy is checked into the DSDS, which also generates a label identifying the cartridge contents, and stores the cartridge in a vault (the library).
A GONG DMAC reduction pipeline stage is defined as that collection of machines and operators that uses data stored in the library as input, and that produces scientific data products to be stored in the library. The successive pipeline stages are coupled through Exabyte cartridges. Cartridges move between only a single pipeline stage and the DSDS library, never directly between pipeline stages.
The DSDS is comprised of two operations machines and a user machine. A Relational Database Management System (RDBMS), ORACLE, resides on both of the operations machines. Each Exabyte cartridge in the DSDS library, wears on its spine a six digit bar code label (volume serial number). This volume serial number is a unique key into a database table, XBYTCTLG, which tracks the physical location and general description of each library cartridge. Whenever a cartridge is removed from the library, as an input to a pipeline stage or returned to the library from a pipeline stage, this label is scanned. A DSDS operator inputs whether the transaction is to or from the library. Database tables containing choices of location and authorized DMAC pipe operators impose constraints on XBYTCTLG. Given the highly repetitive nature of checking cartridges into and out from the DSDS library, the use of scanning devices and database integrity constraints minimize the opportunities for operator error.
The vault containing the cartridges is controlled for temperature and humidity. Database reports are run weekly to compare the physical library inventory with the contents of XBYTCTLG. Borrowers of cartridges outstanding for more than twenty days are contacted. The DSDS has developed a strict policy of limiting cartridge checkout to the DMAC building only. Red dots are placed on the cartridge labels as a reminder. GONG project scientists who are not located at the DMAC have their data staged to a DSDS operations machine by a DSDS operator. The data is then copied across the network to their local computer.
Each data volume is copied. The copy is sent to an offsite storage facility for disaster recovery. Every cartridge returned to the library is assumed to have been read. This constitutes a quality check. Once data has been moved through all pipe stages and archived, a cartridge is read only when requested. To insure the viability of both the archive and offsite cartridge libraries, the DSDS plans to read those tapes whose last quality check has been greater than 365 days.
Archive data cartridges are written using the UNIX utility tar. The first tar file on the cartridge contains identifying label and table of contents files. The succeeding tar files consist of archive data products. Each tar entity can contain only one type of product for a 24-hour period. Data products are named by using a five character data product code and a unique time stamp. The DSDS takes advantage of this data organization on the tape to create a file catalog database table, FILE_CTLG, to track each library cartridge's contents.
For each physical tar file on the cartridge, a row is inserted in FILE_CTLG. The row data consists of the volume serial number of the cartridge, its volume file number (position on the tape), the data product name, the time stamp of the earliest (start) and latest (stop) data products in the tar file group and the pipeline processing version of the data product. We can identify on which tape and tar file a given data product resides by comparing its data product name and time stamp against the data product, start and stop code entries in FILE_CTLG. To further compress the storage requirements of FILE_CTLG, the start and stop time fields have been converted to integer values representing the number of seconds since January 1, 1970.
To track which individual data products are stored in the DSDS, a series of files called bitmaps have been defined for each of the data products. Each bit in the bitmap file represents a time slot for each possible instance of data product. When a data product is added to the library, its corresponding bit in its bitmap is turned on. With 393 data product types defined to date, the entire GONG data archive currently requires only 69 MB of storage. This figure includes prototype data. The archive from the GONG network sites is anticipated to number approximately 250 data products and require 25 MB of storage.
The bitmaps are copied to the User's Machine once per day. The User's Machine is a SOLARIS workstation on which GONG members are given accounts. The DSDS has provided several programs, both command line and menu driven, which enable users to query the bitmaps for product availability. A HyperText Markup Language interface to the bitmaps via the World Wide Web (WWW) has just been released. These query programs create editable files of available data products. These files serve as inputs to a program which submit s a request for the products listed in the files. The query programs and bitmap files can be ported to the GONG member's home institution computer. Only the program to order the data, must be run on the GONG User's machine.
When a data request is submitted, the DSDS Operations Machines are notified via sockets that a data request is available for processing. Daemon programs running continuously on the DSDS Operations Machines, remote copy the file containing the data request and automatically create all of the UNIX shell scripts necessary to remove the data from the archive media to an intermediate staging disk and to create the distribution cartridge(s). Network data requests are staged directly to the requestor's User Machine account. These UNIX scripts are run by DSDS operators from an ORACLE FORMS application. The operator interaction required to fill a data request is minimal, requiring only that the archive cartridge prompted for be placed into the specified tape drive.
In order to track any anomalies in archived data products due to the quality of the data collection process or errors in the pipeline data processing, the DMAC manager has available a data events tracking application. When an archived data problem is detected, usually by its input into a latter pipeline processing stage, the DMAC manager must decide how seriously the problem affects the quality of the archived data product. A decision must be made as to whether the data product must be regenerated.
The data event is logged into the database using the events application software. Events are tracked by an assigned event id and the time span of the data affected by the event. The database is searched and all products whose date matches the data event date are displayed. A subset is selected by the DMAC manager as those actually affected by the event. Products needing reprocessing, if any, are marked. A report containing a detailed description of the data event is placed on-line on both the DSDS Operations Machines and the User's Machine. The report is e-mailed to all of the DSDS pipeline operators. An option to e-mail the data events report to all of the GONG Users who received the affected data products as part of a data distribution is available.
When data is reprocessed, the location of the newest version is added to the database. Previous versions of data products are kept for historical purposes. Only the latest version is distributed. When requesting data for a distribution, users can request that data products affected by data events or data waiting to be reprocessed be excluded.
The DSDS is the central hub of the DMAC through which all Exabyte data cartridges must pass from one pipeline stage to another. If one stage of the pipeline goes down, other stages can continue processing data until no more input tapes are available. If the DSDS goes down, the entire DMAC stalls. To provide a DSDS with a high level of availability to the DMAC and scientific users, the database is mirrored on two workstations, and DSDS applications are designed to run on either workstation. During normal DMAC operations, an operator on either DSDS workstation can perform any DSDS function and the results appear on both workstations in near real-time (within a few seconds). This is achieved using UNIX message queues and sockets (communications links). When an application reads from the database, the read is performed only on the local workstation. But if an application writes to the database, it first writes to the local database. If the local database is updated without error, then the application places a Structured Query Language (SQL) command or a bitmap update command in a buffer and gives the buffer to a routine that places it on a local message queue. A database mirror daemon removes the buffer and sends it over a socket to the other (remote) Operations Machine (OM), where a daemon receives the buffer from the socket and places it on another message queue on the remote OM.
On the remote OM, one of two daemons removes the buffer from the message queue. If the buffer is a bitmap update command, a daemon updates the bitmap. This daemon processes bitmap update commands from both the local and remote OM's, and sends a new bitmap to the Users' Machine when it receives a buffer coded to indicate that updates to the current bitmap are complete. If the command is an SQL command to the database, another daemon removes the command buffer from the queue and performs an SQL EXECUTE IMMEDIATE on the buffer. This process is repeated in the opposite direction, enabling either OM to perform any DSDS function and keep the other OM's database current.
When a socket goes down (such as when an OM fails), if an OM daemon tries to send a buffer to the remote OM and receives an error from the socket, the daemon places the buffer in one of two files on the local OM, depending on whether the message is an SQL message or a bitmap update message. When the remote OM comes back up, the remote database is brought up, and all sockets and daemons are reestablished. The DSDS operator on the remote OM then copies over these files and runs a recovery program that reads the files and places the buffers back on the message queue for processing. This permits one OM to keep processing by itself while the other OM is down, and permits the DSDS to weather most single points of failure (excluding power failures, which bring down the entire DMAC, but which are rare enough and of short enough duration to be no more than a temporary nuisance.