Figure 1: The OPUS Process Manager.
Figure 1: PS 2.3 Mb
Next: Programming in Glish
Previous: perl as an Astronomer-Friendly Language---the pgperl Experience
Table of Contents --- Search ---
PS reprint
J. Rose,1 T. H. Choo,2
M. A. Rose
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore,
MD 21218
1CSC: Computer Sciences Corporation, Inc.
2AURA: Association of Universities for Research in Astronomy, Inc.
There are four objects or entities which must be understood separately in the OPUS environment: observations, processes, paths and nodes.
Observations refer to the raw and processed data which start as exposures on the telescope and wind up as datasets in the science data archive. With the advent of the new OPUS pipeline it is now possible to define combinations of exposures which are to be processed together in an ``association".
Processes are the computer programs and tasks which partition the data, convert the telemetry into understandable science data, remove instrument signatures in the data, and package the products into standard FITS datasets. Processes can be written in any language, and OPUS can accommodate executable images, shell scripts, or even IRAF or IDL scripts as components of a pipeline. Note that a pipeline can be tailored to have any mix of processes which the current data load dictates, and there can be any number of these different pipelines.
Paths describe the list of physical directories on the disk farm where processes can find the input data and where they are to write the output datasets. The definition of the path appears in a simple text file which associates physical devices with symbolic path names. In OPUS any number of paths can be defined allowing the system to be reconfigured as additional hardware is acquired.
Nodes or queues define the machine in the cluster where processing is to take place. While OPUS is currently running in a single VMS cluster of workstations it is possible using DECnet across clusters, or NFS across a network to have both VMS and UNIX workstations defined in the OPUS pipeline. Once observation datasets are converted to standard FITS format, processing can cross operating system boundaries.
To a great extent these four components are independent of one another. Paths have no need to know which machine or node is being used, likewise processes are neither machine-specific nor path-specific. Any combination of processes and nodes operating in a single path defines an OPUS pipeline. As resources and constraints evolve, as new machines are added or as new bottlenecks are created by more elaborate calibration strategies, the pipelines can be easily redefined and the OPUS system reconfigured.
The Process Manager is first a tool to assist operations in configuring the OPUS environment. The File menu allows the user to bring up the process selection dialog where the definition of the pipeline can be established. As shown in Figure 1, this dialog allows one to select any combination of the three OPUS components: processes, nodes and paths. In this example six processes were selected, all of which were operating on the ``blue" path, and all submitted to the same node through a batch queue (``phloxbatch"). A mixture of machines could as easily have been selected, as well as a mixture of paths.
Figure 1: The OPUS Process Manager.
Figure 1: PS 2.3 Mb
Secondly, in addition to activating the processes in the system, the Process Manager has the capability of controlling those processes: to suspend processes, resume suspended processes, instruct processes to reinitialize their data structures, or halt processes.
The Process Manager display indicates for each process, the node (or queue) name, the name of the process, the name of the path, the time the process was started, the time it was reinitialized, its current status, and the amount of CPU it has consumed. As processes pick up observation datasets to work on, they post a ``message" on the blackboard giving the name of the observation. That message (a ``process status file") is picked up by the Process Manager and displayed on the main screen.
Just as each process posts information regarding what it is currently doing, there is equivalent information about the state of each observation dataset. The ``observation status files" indicate the name of the observation, when processing began, the path in which the observation is being processed, and what processing steps have been completed. As with process status files, observation status files are empty: all the information which the system requires is embedded in the name of the file.
Figure 2: The OPUS Observation Manager.
Figure 2: PS 2.0 Mb
The first entry in Figure 2 indicates that the observation Y2A50201T (which started pipeline processing on October 3, 1995 at 13:18:04) is currently being processed by the fifth component of the pipeline. Both the fourth and the sixth observations are being processed by the third pipeline step. This is possible by having instances of that process running on two separate computers. The seventh observation, Y2A50207T, is waiting for two different processes which can execute independently, an example of the parallelism possible in the OPUS system.
Each process in the system polls the blackboard directory for files with a particular status. For example the third processing step will poll for files with names like *-CCW*.* which indicates the first two steps have been completed and the observation is waiting for the third process. Upon finding a candidate, the process will attempt to rename the observation status file, replacing the ``W" (waiting) with a ``P" (processing).
When processing for that step is complete, the observation status file is again renamed. If processing was successful, the ``P" is changed to a ``C" (complete), and the next process in the sequence of steps will eventually recognize this observation as a candidate.
Certainly monitoring the status of observations as they progress through the processing pipeline is critical to understanding the state of the system. But the Observation Manager also has the capability to control the observations. One can halt observations at the next processing stage, remove observations from the pipeline, and copy datasets to a user area for further analysis.
To meet the challenges which the new HST instruments present to the ground system we have re-engineered the production data processing pipeline, converting from a single-processor, linearly-coupled system, to a more flexible and extensible environment. Providing the capability to run multiple instances of processes on multiple nodes in a cluster might solve the problem of handling many hundreds of exposures per day, but this solution itself presents the operations team with a different challenge: to determine what's going on? Two Motif applications have been developed to help make that determination and to assist operators in controlling the pipeline.
Nii, H. P. 1989, in Blackboard Architectures a Applications, eds. V. Jagannathan, R. Dodhiawala, & L. Baum (San Diego: Academic Press), xix