PBS Log Tailing with globus 4

Problem

Globus 4 has moved from a model where each job polls qstat for pbs info to tailing the pbs logs to generate job notifications. The problem with that for us is the gateway machine doesn't have any pbs logs, and exporting them via NFS a) defeats the whole purpose of the gateway machine, and b) only works for one cluster anyway. So.

Proposed Solution

PBS-Telltail is a perl script that I wrote to fix the problem. Here's how it works: A perl daemon runs on the cluster tailing the pbs logs and whenever one of the interesting events happens, it sends it off to the gateway machine.

PBS-Logmaker is also a perl daemon that runs on the gateway, listening for incoming messages, whenever one is recieved, it writes it to the local pbs server_logs directory, emulating actual pbs entries.

Possible alternatives

• Provide no notification mechanism
• Use NFS and work out how to run 3 seperate Scheduler-Event-Generator processes
• hack up globus to use qstat again instead of log tailing

Pros

• Low privilege reader - The cluster process only reads a file and sends. No privileges, and no incoming connections to clusters.
• Low privilege writer - the gateway end only needs to be able to write to one directory and nothing else
• Decoupling/No Globus required on clusters - this app has no requirement on globus, and is completely independent of it.
• no NFS required!
• Log messages are written quickly, making the globus system nice and responsive

Cons

• No built in security. Firewalling would make it safe, but takes extra work. Immediate implications are that anyone could write text to the gateway pbs logfiles.
• DBannon hates perl, prefers buffer overrun friendly languages.

Boring bit / How it works

The only hard part with this script was the fact that pbs doesn't work in standard logrotate fashion, so it's wise to periodically check for a new pbs log file, stop tailing the old one and start tailing the new one. Otherwise it's very simple, it detaches itself to run in daemon mode and is very easy to write an init script for.

-- DamonSmith? - 05 Jul 2005