Installation and Configuration Guide
NOAO High-Performance Pipeline System (2008)

Derec Scott <dscott@noao.edu>
Francisco Valdes <fvaldes@noao.edu>
Francesco Pierfederici <fpierfed@noao.edu>


* Introduction

The NOAO High-Performance Pipeline System (NHPPS) is a lightweight
framework for event-based based pipelines. It is written in Python and,
as such, supports a wide variety of Operating Systems (e.g. Mac OS X,
Linux, Solaris etc.).  It requires Python 2.3.4 or later.  The other
class of source is csh scripts.  Some of the networking and configuration
assumes the pipeline accounts use csh/tcsh as the default login shell.
The distribution includes all the extra modules and packages you need and
there is no need to compile anything.  It comes with an example pipeline
and a script that demonstrates it.


* Obtaining NHPPS

NHPPS is available by anonymous ftp at

    ftp://iraf.noao.edu/misc/nhpps10/

Extract the contents of the compressed tar file in some directory.
Set the path to the directory in the environment variable NHPPS.
Source the file $NHPPS/config/Setup.csh.  In more advanced
usage the environment variables in this setup script can be set before
sourcing the file to something other than the defaults.  Of course,
you could also modify the file or use a customized copy.  The defaults
for the pipeline log and data directories are for the supplied examples
which are within the NHPPS tree.  In production these would be outside
the tree on some large data partition.  To run the example pipeline, source
the file $NHPPS_BASE_DIR/examples/cshrc.

Clearly these three steps can be defined in a shell file which is sourced
at login.  For the recommended csh shell this would be .cshrc and the
steps would be

    setenv NHPPS <some path>
    source $NHPPS/config/Setup.csh
    source $NHPPS/examples/cshrc

The pipeline is intended to be used in a networked environment with
multiple processing nodes, with a single node being a possible case.
As such the nodes must be able to communicate with each other and use
password-less ssh.  One gotcha is that the local host must also be able
to ssh to itself with no password.

Another aspect of this networked environment is use of Pyro for
communication between python processes.  Pyro is included in the
distribution but one should check if this name server can be started
and reached.  The script $NHPPS/config/configure.py should be
run to check the NHPPS installation.  If the network does not support
UDP broadcasting you will need to set PYRO_NS_HOSTNAME to point to the
machine running the (Pyro) Name Server.

For communicating with nodes it is also recommended that the following is
added to the beginning of .cshrc.  This allows most remote ssh commands
(e.g. ssh <node> ps) to run quickly without sourcing the full environment.
Pipeline scripts that need the full environment call "plexec" which then sets
SSH_TTY and sources .cshrc again so that the full environment is read where
things like the Setup.csh file are sourced.

if (! $?SSH_TTY) then
    # Setup MINIMAL environment for shells which are started via ssh.
    # It is recommended that the very first item in the ~/.cshrc file be this
    # check of the $SSH_TTY variable. If it is not set, then the NHPPS may be
    # executing a command remotely, and we wish to include in the path the path
    # to the plexec command (<PATH_TO_NHPPS>/bin).
    set path = (/usr/bin /bin <PATH_TO_NHPPS>/bin)
    
    # If you have other environment variables which you wish to set when the
    # shells SSH_TTY is not set, set them here.
    exit
endif

* Demonstration Example Pipeline

This is described in the file $NHPPS/doc/misc/Example.txt.  For
the impatient, if you have set NHPPS and sourced the two files
Setup.csh and examples/cshrc, you can type runexample1 and, hopefully, you
will see the sample pipeline start and run.

* Support and Caveats

For support questions, please contact Derec Scott at <dscott@noao.edu>

Currently we are providing NHPPS with minimal documentation and
experience with external users.  We will be glad to help within our
limited resources.  Those serious about using NHPPS in a production
system should plan to visit the NOAO Pipeline Team in Tucson for
learning about real production pipelines and many of the details
of the system and applications.