APAC Grid Environment variables

Design note

This issue is discussed in detail in this design note.

• ANG_design_note2.doc: environment variables design note (Word version)
• ANG_design_note2.pdf: environment variables design note (PDF version)

The following topics are based on the design note and are here for easy reference.

The variables and their definitions

HOME

Defined on all grid accessible gateways VMs and compute nodes within a site, and containing on each a (potentially different) string which names a common home directory for each user (with an account at that site)

• $HOME is distinct and private for each user or VO.
• Management policy of the contents is determined by the user
• Subject to site specific quota and backup policies. For VO accounts sites will ensure that contents are preserved for at least 5 working days.
• accessible via a GridFTP server for data staging

USER_SCRATCH

Defined on all grid accessible gateway VMs and compute nodes within a site and containing on each a (potentially different) string which names a common temporary file space for that user or VO.

• $USER_SCRATCH is intended to be a large pooled storage resource, operated without backup and subject to site specific retention policy. Sites without that capability will map $USER_SCRATCH to a subdirectory within $HOME
• $USER_SCRATCH is distinct and a site can ensure that it is created to be private for each user or VO
• Sites will ensure that contents are preserved for at least 5 working days from last write
• accessible via a GridFTP server for data staging

NODE_SCRATCH

Defined on all grid accessible compute nodes within a site

• $NODE_SCRATCH is intended to access local storage devices dedicated to a node and hence is not visible to gateway VMs or head nodes. Because the storage is local it is potentially faster than the other available spaces and should be used preferably by I/O intensive applications.
• $NODE_SCRATCH is distinct and private to each compute node and each user or VO
• Though the variable is defined by the site the associated directory may need to be created by the user's job.
• The user may also need to clean up the directory when their job is finished with it
• not necessarily accessible to GridFTP or any other network service

Example uses

Creating independent file spaces shared by a job

To isolate job data from other job submissions, users will usually need to generate a unique identifier (say MYJOBID) for each job invocation from within their scripts or work flow, so that files related to a single job can be created and referenced within directories named as:

JOB_SCRATCH = $USER_SCRATCH/$MYJOBID or $HOME/$MYJOBID

Sites will also provide local node storage, visible after a job is scheduled onto the node, which will still need user management to avoid file conflicts, so that files for a job should be kept in:

LOCAL_SCRATCH = $NODE_SCRATCH/$MYJOBID or LOCAL_SCRATCH = $NODE_SCRATCH/$MYJOBID/$$

The first definition would access a shared file space in the case of multiple processes on a multi-cpu node with the same MYJOBID. The second definition would access separate space for each process.

In a work flow outside of Globus

An application may want to stage data or from a site in a workflow involving GridFTP, for example. GridFTP does not expand environment variables itself so the application will need to be able to query a registry to find the value of one of the above environment variables before using it. This would apply to the $HOME and $USER_SCRATCH variables only since the $NODE_SCRATCH space would probably not be accessible via GridFTP.

From within a job

Jobs that arrive on a compute node whether through GRAM 2 or GRAM 4 will be able to use the above variables in the same way at each site. Globus defines other variables as well which may be useful. Here’s an example script which uses $HOME, $USER_SCRATCH and $NODE_SCRATCH.

#!/bin/sh
# script to test APAC Grid variables
# files are staged in the workflow before this script runs

# define a directory specific to this job and process
LOCAL_SCRATCH=$NODE_SCRATCH/$myjobid/$$
# make the directory
mkdir -p $LOCAL_SCRATCH
# copy files already staged in to this directory
cp $USER_SCRATCH/$myjobid/input $LOCAL_SCRATCH
# this will do faster I/O than to USER_SCRATCH if it is an issue
# process the data
/usr/bin/bzip2 $LOCAL_SCRATCH/input
# copy the output to an area from which it can be staged out
cp $LOCAL_SCRATCH/input.bz2 $USER_SCRATCH/$myjobid
# remove the temporary directory and the contents
rm -rf $LOCAL_SCRATCH

When submitting to Globus 2

Globus 2 provides a set of variables that can be used in job descriptions which include $HOME and $LOGNAME. $HOME is the same as $HOME above and $LOGNAME is the user name the job is running under.

The $SCRATCH_DIRECTORY variable is available only when the RSL attribute scratch_dir is set. It is set to the path of a temporary directory which is created in the directory specified by scratch_dir. If scratch_dir is a relative path then the directory is created relative to a site specified location. Using a relative path is preferred since the site can specify the appropriate parent directory with the correct permissions for the user. This variable is like $USER_SCRATCH except that it is specific to a job and is deleted after a job completes.

Here are some example RSL parameters you could see in a Globus 2 job.

(scratch_dir = .)
(file_stage_in = ($(GLOBUSRUN_GASS_URL)/input
                    $(SCRATCH_DIRECTORY)/input)
(file_stage_out = ($(SCRATCH_DIRECTORY)/input.bz2
                    $(GLOBUSRUN_GASS_URL)/input.bz2))

This example also shows the use of another variable in Globus 2 – GLOBUSRUN_GASS_URL. This refers to a GASS server running on a client machine. Files are staged to and from there to the $SCRATCH_DIRECTORY.

Note that $USER_SCRATCH and $NODE_SCRATCH can not be used in a Globus 2 job description.

Here’s a script that runs on a compute node using the Globus 2 variables as well as $NODE_SCRATCH.

#!/bin/sh
# script to test APAC Grid variables with Globus 2

# define a directory specific to this job and process
LOCAL_SCRATCH=$NODE_SCRATCH/$myjobid/$$
# make the directory
mkdir -p $LOCAL_SCRATCH
# copy files already staged in to this directory
cp $SCRATCH_DIRECTORY/input $LOCAL_SCRATCH
# this will do faster I/O than to USER_SCRATCH if it is an issue
# process the data
/usr/bin/bzip2 $LOCAL_SCRATCH/input
# copy the output to an area from which it can be staged out
cp $LOCAL_SCRATCH/input.bz2 $SCRATCH_DIRECTORY
# remove the temporary directory and the contents
rm -rf $LOCAL_SCRATCH

When submitting to Globus 4

Globus 4 expands and uses only certain variables supplied to a job description. They are

$GLOBUS_USER_HOME

this is the same as $HOME

$GLOBUS_SCRATCH_DIR

this is defined to be the same as $USER_SCRATCH

$GLOBUS_USER_NAME

the user name the job is running under.

Here’s part of an example Globus 4 job description using $GLOBUS_SCRATCH_DIR.

<directory>${GLOBUS_SCRATCH_DIR}</directory>
<stdout>${GLOBUS_SCRATCH_DIR}/stdout.$myjobid</stdout>
<stderr>${GLOBUS_SCRATCH_DIR}/stderr.$myjobid</stderr>
<fileStageOut>
  <transfer>
     <sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/stdout.$myjobid</sourceUrl>         
     <destinationUrl>gsiftp://store/home/900/sjm900/stdout.$myjobid
     </destinationUrl>
  </transfer>
  <transfer>
     <sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/stderr.$myjobid</sourceUrl>
     <destinationUrl>gsiftp://store/home/900/sjm900/stderr.$myjobid
     </destinationUrl>
  </transfer>
</fileStageOut>
<fileCleanUp>
  <deletion> <file> file:///${GLOBUS_SCRATCH_DIR}/stdout.$myjobid</file>
  </deletion>
  <deletion> <file> file:///${GLOBUS_SCRATCH_DIR}/stderr.$myjobid</file> 
  </deletion>
</fileCleanUp>

Here’s a script that runs on a compute node using Globus 4 variables. It is generated as part of a workflow script.

myjobid=$$
input=input.$myjobid
output=$input.bz2
input_path=\${GLOBUS_SCRATCH_DIR}/$input
output_path=\${GLOBUS_SCRATCH_DIR}/$output
stdout_path=\${GLOBUS_SCRATCH_DIR}/stdout.$myjobid
stderr_path=\${GLOBUS_SCRATCH_DIR}/stderr.$myjobid

cat >$script <<EOF
#!/bin/sh
# script to test APAC Grid variables

LOCAL_SCRATCH=\$NODE_SCRATCH/$myjobid/\$\$
mkdir -p \$LOCAL_SCRATCH
cp $input_path \$LOCAL_SCRATCH
# this will do faster I/O than to GLOBUS_SCRATCH_DIR if it is an issue
/usr/bin/bzip2 \$LOCAL_SCRATCH/$input
cp \$LOCAL_SCRATCH/$output $output_path
# remove the temporary directory and the contents
rm -rf $LOCAL_SCRATCH
EOF

Implementation

Providing the space

Because of assumptions within Globus it may be necessary to use symbolic links to provide the right functionality.

For example at the National Facility $USER_SCRATCH depends on a user’s project. Globus does not know about a user’s project so the following can be done to get around it...

ln –s /short/$PROJECT/$LOGNAME $HOME/.globus/scratch
GLOBUS_SCRATCH_DIR=$HOME/.globus/scratch

A site may need to make sure these directories are created and with the correct permissions as well as any required symbolic links. This can be automated to be part of a user account creation.

Defining the APAC grid variables

The APAC grid variables can be defined in a system’s environment so that they are available whether a user logs in interactively or is running a job submitted through Globus. An example implementation of this would be

USER_SCRATCH=/short/$PROJECT/$LOGNAME
NODE_SCRATCH=$PBS_JOBFS

A site which is using /tmp for local storage could define

NODE_SCRATCH=/tmp/$LOGNAME

A site whose local storage is actually shared between compute nodes could use

NODE_SCRATCH=/tmp/$LOGNAME/`hostname`

to ensure that the variable refers to a per node space.

Another way would be to implement them in Globus’ site specification job submission script pbs.pm. Here’s a script fragment setting variables available outside of PBS and other setting variables available within a PBS job.

# add some more environment variables
    @environment = $description->environment();
    push @environment,['USER_SCRATCH',
         '/short/'.$description->project()."/".$ENV{'LOGNAME'}];

    foreach my $tuple (@environment)

   # add environment variables that depend on PBS
   $pbs_job_script->print("export NODE_SCRATCH=\$PBS_JOBFS\n");

Globus 2 variables

Most of the Globus 2 variables do not need configuring. However, a site should provide a value for -scratch_dir_base in the file $GLOBUS_LOCATION/etc/globus-job-manager.conf

e.g.

-scratch-dir-base $(HOME)/.globus/scratch

Globus 4 variables

$GLOBUS_SCRATCH_DIR is defined in $GLOBUS_LOCATION/etc/gram-service-PBS/jndi-config.xml (for example). The definition of this can contain ${GLOBUS_USER_HOME} but no other variable apparently.

Note that the GLOBUS* variables are expanded from a job’s description but not provided to a job’s environment when it runs on a compute node. This is a bug in Globus 4.0.1. The variables can be set in pbs.pm as they are for the APAC grid variables. The bug is supposed to be fixed in Globus 4.0.2.

-- StephenMcMahon - 19 May 2006