NOAO Save the bits archive -------------------------- These directories contain the software for the "Save the bits" archive. Save the bits is a pragmatic network data archive which transfers FITS data to a centralized archive machine from an arbitrary number of data acquisition computers (presumably at telescopes) using a unix print queue to handle the network communications and control concurrency issues. Any conforming FITS data extension or normal FITS image is acceptable for archiving. The data are multiplexed from any and all attached acquisition computers onto a single archive medium (e.g., exabyte tapes) as large FITS image extension files. A disk based FITS header catalog is written with each tape file. The tapes and the header catalog are cross-referenced via a unique running ID number into a separate index file. Recasting either the header catalog or the data tapes is designed to be straightforward since only the index need be regenerated in either case. The header files serve as easy input into a third party database such as maintained by the CADC. The online portion of save the bits is composed of three main pieces, bitf, bitmon and bitverify. Bitf is a lpd style input filter which stages the incoming FITS format images and extensions into a staging directory. When the staging directory is full enough, the files are concatenated into a FITS extension file on the current tape device. Bitf tracks how full the tape is, switches to an alternate drive when the tape fills up, and spawns bitverify to checksum verify each tape (and any copies). Other functions include reporting when the drives are in need of cleaning. Bitmon is the tool used by the archive manager to monitor the status of the archive and swap tapes as necessary. Utility programs include boottime, which reports the time of the last reboot of the computer (important since the tape drives automatically rewind when the computer is booted), lpcbits, which allows the non- privileged bits and daemon accounts to control the host level queue functions, and domes, which generates a report on the health of the archive queue running in each particular dome. Installation ------------ You will need to create two accounts, one to own the software and one to manage the online archive. We call the software account "bits" and the archive manager "stb" (for "save the bits") for simplicity I would recommend that you use the same names. The stb account must be added to the daemon group to allow the bitmon program (run by stb) to communicate with the bitf daemon using disk file based keyword=value status requests. After creating the accounts, log in as bits and unpack the tar file into a convenient subdirectory .../bits. The archive needs to know the absolute pathname to the spool directory, /bits/spool and other working directories such as /bits/stage, /bits/discards, and /bits/stb. We simplify this by making a symbolic link from /bits to /bits and referencing all paths relative to /bits. These paths may be specified in config.h, but again for simplicity I would recommend that you use the same symbolic link technique. The tar file contains SunOS Sparc binaries and this is the only platform that the archive software has been used on. There is no reason the software shouldn't run on any other BSD derived host, but there may be some slight compatibility issues. The bitf filter was also designed with a future migration to the Solaris lpsched mechanism, but this will require some moderate amount of porting effort. Three of the binaries require special ownership and unix setuid permissions. These are boottime which requires access to the unix kernel, lpcbits which must be able to reset the file permissions on the unix lpd queue lock file, and domes, but only if you intend to install a WWW page to display the archiving status from the domes - this is not integral to the mechanism. To set the ownership and permissions, log in as root, cd to /bits/src and type `make chown'. The archive working directories have to be owned by daemon, since they are written into by the bitf lpd filter itself. Log in as root, cd to /bits/src and type `make setup' - this should only be done the first time you setup the archive, since it will overwrite the archive status file /bits/spool/stb_status. The stb_status file contains the moment to moment status of the archive, while the /bits/lib/stb_drives and /bits/lib/stb_domes files allow specifying your particular drive and data acquisition setup. The format of stb_drives is: ndrives = 2 drive[0].device = /dev/nrsx0 drive[0].capacity = 1500000 drive[0].tapemark = 20 drive[2].device = /dev/nrsx1 drive[2].capacity = 700000 drive[2].tapemark = 800 The number of sequentially accessed drives is freely configurable at runtime and archiving duties will swap from one to the next as tape file. The maximum number of such drives is specified via config.h when the software is compiled. The unit of data that the archive uses is 2880 byte FITS records, so a tape capacity of 1,500,000 = 4.12Gb for an exabyte 8505 and 700,000 = 2.01GB for an exabyte 8200 (pick conservative round numbers) The size of the tapemark should also be conservatively set on the large size and corresponds to whatever the exabyte manuals suggest for your particular unix driver. You can see why we decided to go with the 8505 drives. The stb_domes file is used by bitmon and the domes command (we access this through mosaic) to keep tabs on what's going on in all the domes. It isn't integral to the archive. To activate the archive, add the appropriate printcap entries from the file /bits/lib/printcap into /etc/printcap. By default a unix lpd queue is active as soon as the printcap entry and spool directory are setup and before a lock file is created by the lpd daemon, so your archive should be on the air now. You can query the archive using either unix tools (lpc, lpq) or the bitmon command: status [] status display [loop delay] domes [] dome status [loop delay] swap [] swap tapes, only full ones by default queue [on|off] enable/disable queuing, or report status tape [on|off] enable/disable taping, or report status spooldir [on|off] enable/disable (or report) spool file count help list this information (or `?') clear clear the screen quit exit the program Try typing "bitmon status" at the unix prompt, or just type "bitmon" to stay in the program. The only real day to day control that is needed for the archive is the "swap" command which leads you through swapping tapes and mounting new ones. Rob Seaman NOAO/IRAF Group seaman@noao.edu