APAC Grid Accounting

This page moved to new wiki site 4/12/2007

Purpose - Maintain records of APAC grid Usage, viewable by system and project administrators. http://www.vpac.org/grid/ There are three distinct aspects -

• At run time, that is, while a job is running, a site needs to be able to contact the owner.
• After a job is completed, some sites are required to keep records of who ran the job.
• Long term, tha APACGrid must be able to provide detailed records of quota usage.

Responsible Group - VPAC, SAPAC

Contact - David Bannon - dbannon@vpac.org - 03 9925 4743

Results - Data is stored in a database.

• see the Grid Operations Centre.
• Individual sites can obtain reports on their usage using a web service -

  wget -O usage.txt http://vpac.org:8080/gridsiteusage?site=ac3\&date=2007-01-07

  Don't forget to drop the '/' if using a browser instead of wget.

---

Technology

• On NG1 gateway hosts and non-VDT NG2 gateway hosts, the 'pbs.pm' program writes records containing PBS Job-Id's with corresponding User DN's to a file whose contents are emailed daily to the GOC; note that a special local version of gram-service.jar was required on non-VDT NG2 gateway hosts to place the DN in a GLOBUS_USER_SUBJECT environment variable.
• On VDT-based NG2 gateway hosts, those records are written in a database, and extracted at houry intervals for emailing to the GOC; they are also logged locally for diagnostic purposes.
• On each Cluster head Node, a 'cron' job is executed daily to email appropriate PBS accounting records to the GOC.
• Data is then collated in a mysql database and displayed on the GOC web page.

Emailing Accounting Data from Cluster Head Node

Requirements

• Usage emails are sent from the cluster (eg) management node as root once a day.
• The subject line should look like this Subject: brecca vpac 20060730, being the cluster name, site name and date.
• The emails should contain data that looks like PBS accounting log files, preferably only those lines that include Grid_ng1_ (or ng2, ngdev, ngportal etc).

Sites need to send the appropriate PBS accounting files to grid_pulse each day. For example, they may choose to install something like this on their management (or head) node :

In roots (or another user who can read the PBS accouning files) crontab there will be an entry like this :

3 1 * * *   /usr/local/sbin/send_grid_usage;

Note that the gateways have a randomised delay between 1 and 10 to ensure that all sites don't hit the GOC at the same.

and the script, send_grid_usage, will look like this (on brecca at VPAC) :

#! /bin/bash
#
# This script emails the grid usage report for yesterday
# to the grid operations centre.
# It should be called from crontab daily
# Please note the email subject, its is in the format of :
#   <cluster_name> <site_Name> <date>
# Please keep this consistent, we need it like that.

YESTERDAY=`date --date=yesterday +%Y%m%d`

# echo "Yesterday is $YESTERDAY"

cd /usr/spool/PBS/server_priv/accounting;
grep  Grid_ "$YESTERDAY" | mail -s "brecca vpac $YESTERDAY" grid_pulse@vpac.org

I'm sure there are other ways of achieving the same result.

A subject line will typically look like this -

Subject: brecca vpac 20061213

A data line will typically look like this -

12/13/2006 23:59:59;E;750442.brecca-m.vpac.org;user=grid-admin group=vpac jobname=Grid_ngtest_ queue=sque ctime=1166014798 qtime=1166014798 etime=1166014798 start=1166014798 exec_host=node082/1+node082/0 Resource_List.ncpus=1 Resource_List.nodect=2 Resource_List.nodes=2 Resource_List.walltime=00:01:00 session=351 end=1166014799 Exit_status=0 resources_used.cput=00:00:00 resources_used.mem=3868kb resources_used.vmem=8508kb resources_used.walltime=00:00:01

The data collected will be inserted into a data base and the contents displayed on the GOC pages.

Debugging and Internal Details

If you use the recommended install scripts then you may never need to read the remainder of this page. However, if you are choosing to "roll your own" in some way, or have having problems that require some deep debugging, read on ! There are three parts to the system, each is needed and provided by the standard install. Generally, things will work better if you follow the recommended methods. Note that all three messages must come from Root !

• Grid_Pulse Message, as root, from the gateway to the GOC describing its health. Note that if the Grid_pulse messages have not been received, the data from the other two systems will not appear.
• JobID Message is sent from the gateway and maps pbs jobid to the users DN.
• The Usage Message is sent from the PBS Management node that provides accounting information for relevent jobs completed in the previous 24 hours.

Grid-Pulse Message

Grid_Pulse sends a message, as root or as grid-admin, from the gateway to the GOC describing its health. Note that if the Grid_pulse messages have not been received, the data from the other two systems will not appear. The subject line must have the " FQHN-of-gateway Periodic System Check " for example a message should contain " Subject: ng1.ivec.org Periodic System Check " in its header.

The instructions for building a node include all thats necessary to make grid_pulse work. Sometimes. Othertimes you may have problems with reverse DNS lookup, if its not working the message will be rejected (but that normal behaviour for email now). Make sure your outward going email handling systems don't decide to rewrite the From line, apparently JCU experienced this.

The Content of the grid-pulse message will contain a number of lines detailing info or services that are working or not. The last line is critical, it determines the state the node is shown on on the GOC, it can be "OK or "Not OK".

Info:    CRL problem ..  /etc/grid-security/certificates/ff94d436.0
Service: syslog is: OK
Summary: ng1.ivec.org is: Not OK

note Machines are reported to be 'OK', "Warn', 'Down'. A machine in 'Warn' may be in that state because it believes a service should be running that is not. In many cases all that needs to be done is to knock out this service * chkconfig servicename off*.

JobID Message

A message is sent from the gateway that maps pbs jobid to the users DN. This message must come from root and must have a subject " FQHN JobID date " for example " Subject: ng1.vpac.org JobID 20070518 ", note that 'JobID' is a literal, not the an actual JobID. This is the JobID message. Contents of the JobID Message will look like this -

Dec 14 23:02:40 ng1 Job-DN: 18614.rufous /C=AU/O=APACGrid/OU=ac3/CN=Youzhen Cheng GTest
Dec 14 23:03:06 ng1 Job-DN: 18615.rufous /C=AU/O=APACGrid/OU=ac3/CN=Youzhen Cheng GTest

Usage Message

A message is sent from the PBS Management node that provides accounting information for relevent jobs completed in the previous 24 hours. This message must come from root and have a subject that says " Cluster-Name site date ", for example - " Subject: wexstan vpac 20070516 ". This is the Usage message. Note that the 'site' term must be in an agreed and short format. Generally this is easy, Australian 'PACs have four or less letter acronym and that is used. Canterbury NZ uses "NZ-Cant" This name is also used in the (VPAC, internal) sites.conf file to map FQDNs to and is the site name as displayed as a heading on the GOC.

Contents of the Usage Message will any number of lines like the following, its a standard PBS accounting log entry.

05/16/2007 20:12:29;E;195289.edda-m.vpac.org;user=markus group=vpac jobname=Grid_ng2_ queue=sque ctime=1179309348 qtime=1179309348 etime=1179309348 start=1179309698 exec_host=edda024/1+edda024/0+edda016/1+edda008/3 Resource_List.nodect=4 Resource_List.nodes=4 Resource_List.walltime=00:10:00 session=31528 end=1179310349 Exit_status=271 resources_used.cput=00:42:29 resources_used.mem=254128kb resources_used.vmem=396592kb resources_used.walltime=00:10:50

Necessary data in these lines include -

• Date - first 10 characters and must be followed by a space.
• PBSJobID - always appears after ";E;" and terminated by next ";" - hey, don't blame me, its how PBS does it! Must correspond to the entry in the JobID message.
• resources_used.walltime - used to calculate usage.
• queue - stored but not used for any particular purpose.
• jobname - note must be in the format shown, starting with e.g. "Grid_ng2_", the APACGrid pbs.pm will make it so. If you do it yourself, make sure its "Grid_[gatewayname]_". The gateway name must be the same as appears as the first part of the FQHN in the Subject of both grid_pulse and JobID? messages. The trailing underscore is necessary.
• Exit_status - stored but not used for any particular purpose.
• account - optional, stored if provided. Used to over ride an associated project.
• user - the name of the user as whom the job was run as.
• either Resource_List.ncpus (Resource_List.ncpus=2) or a valid, traditional exec_host line (exec_host=edda024/1+edda024/0). Systems tries for Resource_List.ncpus first, if that is not one or greater looks to exec_host, that must be in the traditional comma separated format, NOT the ANUSF invented compact form like exec_host=ac10/cpus=4-5/mems=2

-- DavidBannon - 20 May 2007

This page moved to new wiki site 4/12/2007