TWiki> Main Web>GridAdminInstructions>ChangeControl>ChangeNotesTest (30 Jun 2008, DanielCox)EditAttach

Change Notes

Dynamic list of change notes for ChangeControl policy.

ALERT! NOTE: This is not working as intended yet. Daniel 25/6/08 (will fix this week, I am trying a few options).

  • WORKFLOW variables are not being expanded. Even after setting the plugin order.
  • This worked at SAPAC
  • Location of set WORKFLOW = makes no difference
  • Expandvariables otherwise seems to work even with included topics

See: VarSEARCH, FormattedSearch, TablePlugin

Topic Updated State
| ChangeNote200806-002 | 30 Jun 2008 - 17:10 | ---+!! arcs.org.au host downtime

Description

A Xen server running on the VPAC network requires downtime for a switchover. The current physical host has reached its limit in terms of available memory and therefore a new server has been allocated to replace it with more resources. This host is used primarily for VPAC servers (including our corporate web site and staff services), but also hosts the arcs.org.au virtual machine.

Estimated Duration

Approximately 1 hour - should be much less.

Systems/Services Affected

The following services are hosted on arcs.org.au, and will be unavailable during the downtime:

  • rt.arcs.org.au
  • wiki.arcs.org.au
  • pfc.org.au
  • goc.arcs.org.au
  • gocdev.arcs.org.au
  • www.arcs.org.au
  • ARCS Jabber server

DNS and CA services are hosted on a difference machine.

Staff Responsible

AndyBotting, SamMorrison

Detailed Instructions

The existing server will be shut down and disconnected from the fibre channel storage unit. The new server will be installed and reconnected to the fibre channel unit, and booted up. The Xen Dom0 is installed on the fibre channel unit, so no transfer of data is required, limiting the required down time.

Testing Procedures

Boot the Dom0 and start all virtual machines. Ping all the virtual machines and test all services hosted by them. In terms of ARCS infrastructure, visiting the web sites hosted, and connecting to the Jabber server.

Back-out Procedures

If anything should go wrong and it cannot be fixed within the allocated time frame, the old server can be reinstated.

Review

25/06/08 sent to techstaff list. Change date to Monday 7th July 8am to avoid Questnet conference.

30/06/08 discussed at systems services meeting.

Schedule/Notification

8am EST 7 July 08

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200806-003 | 07 Jul 2008 - 17:06 | ---+!! Cronjob for SRB Zone Sync

Description

This note describes the steps necessary to implement a cronjob for SRB user federation on a ARCS SRB server.

Estimated Duration

5 minutes

Systems/Services Affected

SRB. at iVEC, eRSA, UQ, TPAC, ANUSF

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon

Detailed Instructions

as root:

run 

   vim /etc/logrotate.d/srbzonesync
and paste 
   /usr/srb/data/log/zonesync.log {
      daily
      rotate 8
      compress
      missingok
   }
save and exit vim.

as srb server user:

run 

   crontab -e
and add line 
   30 */2 * * * cd ~ && /usr/bin/Szonesync.pl -u >> /usr/srb/data/log/zonesync.log 2>&1

Testing Procedures

Monitor /usr/srb/data/log/zonesync.log for a number of days to ensure proper logging.

Back-out Procedures

as srb server user:

run 

   crontab -e
and remove line 
   30 */2 * * * cd ~ && /usr/bin/Szonesync.pl -u >> /usr/srb/data/log/zonesync.log 2>&1

as root:

remove file 

   /etc/logrotate.d/srbzonesync

Review

Looks good to me. -- StephenMcMahon - 30 Jun 2008

Looks fine. -- PaulineMak - 1 Jul 2008

Good to me too. -- ShundeZhang - 01 Jul 2008

Schedule/Notification

no downtime required.

Tracking completion

iVEC: done.

UQ: Done.

SAPAC: Done.

TPAC: Done.

ANU: Done.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

-- FlorianGoessmann - 30 Jun 2008 | | ChangeNote200806-004 | 12 Sep 2008 - 11:40 | ---+!! Gridpulse and MDS updates including SIP

Description

Upgrade gridpulse (to do certificate checks and use correct ARCS email address).

Upgrade MDS with SIP to extract software for the grid from APAC Software Map.

Estimated Duration

Less than 1 hour per site. A time frame of less than 2 weeks will be given for all sites to apply the change.

Making more software information available and correcting existing information will be an ongoing task.

Systems/Services Affected

MDS at all sites - Few problems expected, additional information will be provided by MDS about software.

ALERT! Warning: It is possible during the upgrade, that a client tool - such as GGC or Grisu may read incorrect information from the central index. The index is updated every 10min :-

  • this could be because of upgrade errors (unlikely)
  • or because the current static software information is being merged with new dynamic data from the software map. More software may be listed as available but may not be 100% correct yet. Sites will aim to have all software published correctly soon after the upgrade. MdsScoreCard will be updated to measure success of software information.

Staff Responsible

Daniel/Gerson, Leigh, Jason, Andy, Joel, Will, Darran, Youzhen, Vlad, Andrey

Detailed Instructions

Upgrade RPMs and MIP as described in Systems Trac ReleaseNotes/080401

Once software is being published correctly, the static information in /usr/local/mip/modules/default/SubCluster/softwareInfoData/localSoftware.xml can be removed. Test and double check that this does not affect the MIP information being published - minor changes to Software Map may still be required.

5/5/08 Will

  • Check if /opt/vdt/globus/etc/globus_wsrf_mds_index/server-config.wsdd is owned by daemon, otherwise the file will need correct ownership for globus to start successfully.
    • chown daemon:daemon /opt/vdt/globus/etc/globus_wsrf_mds_index/server-config.wsdd
    • not needed for eRSA 11/8/08

11/8/08 Daniel

  • slightly changed SIP download example to make it quiet and prevent mail from cron to root user

Testing Procedures

Check SIP, see: Trac notes

Check MIP output: /usr/local/mip/mip

Check via Globus WSRF (from a machine with Globus client tools and your proxy credentials), eg: wsrf-query -s https://ng2.sapac.edu.au:8443/wsrf/services/DefaultIndexService "//*[local-name()='Site']"

Back-out Procedures

Backup /usr/local/mip first.

Try and work out issues with Gerson, if data is not completely wrong.

Otherwise if MIP is not working at all or providing completely wrong data it will end up in the central index after approx 10min and may then affect users. Restore the backup and try again on a development machine first.

Review

Release notes, RPMs reviewed by Darran and Gerson in April 08.

30/06/08 Send to techstaff@, discussed at systems services meeting.

1/07/08 Darran - suggest adding warnings. Daniel - actually there could be issues if SIP not working. Instructions to be improved.

14/07/08 Daniel - instructions fixed to describe using wget and cron instead.

17/07/08 Vlad - notified that the preuninstall scriptlet would disable MIP, globus code. With step 4. (mip-globus-config), the procedure is now ready to roll.

5/08/08 Will - file ownership not set correctly after MIP rpms upgrade, added extra info in detailed instruction.

Schedule/Notification

31/7/08 Developers list because of possible risk of disruption to GGC and Grisu users.

Notification will not be sent per site, but once on approving this change.

A "good news" announcement will be sent once all sites are updated and have some software information being published.

Tracking completion

All sites should upgrade the software, but there may be a few exceptions (NZ) who are not using the software map.

Site Date Completed or Comments
eRSA Completed 11/08/08
TPAC Completed 08/2008
VPAC New gridpulse installed 2/07/08
AC3  
CSIRO HPSC Change completed, tested and working 01/08/08
ESSCC Completed 5/08/08
QUT  
UQ Completed 5/08/08
JCU Completed 5/08/08
IVEC Completed 12/08/08
ANU Completed 11/08/08
Canterbury  
Aukland  

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200806-005 | 28 Jul 2008 - 15:14 | ---+!! MyProxy Changeover

Description

Move and upgrade MyProxy service (from v3.6 to v4.2) onto a new machine, change to ARCS domain/host name.

NOTE: Backwards Compatible

Estimated Duration

4 days (over a weekend) - no actual downtime expected.

DNS TTL 30min

Systems/Services Affected

Grid submission hosts and tools (Grisu/Grix) run by end users.

User might upload a proxy to old server, DNS records then change and when they go to download the proxy it's not there (as on the old box).

Staff Responsible

SamMorrison, ANU (Jonathan)

Detailed Instructions

Stage 1
  • DONE Set up myproxy2.arcs.org.au
  • DONE Test
  • DONE Set up a cron job to copy all proxies from the old machine onto the new machine
  • DONE Change APAC DNS records shown below
  • DONE Change ARCS DNS records to point myproxy.arcs.org.au, myproxy1.arcs.org.au to -> myproxy2.arcs.org.au
  • DONE Test

Stage 2 - To happen at a later date

  • set up myproxy1.arcs.org.au at Queensland ?
  • Test
  • Set up unison to sync the data dirs
  • Test
  • Change ARCS DNS records as shown below

DNS Changes

INITIAL ARCS RECORDS (Stage 1) 

myproxy         A          202.158.218.233
myproxy1       A          202.158.218.233
myproxy2       A          202.158.218.233

Final ARCS DNS RECORDS (stage 2) 

myproxy         A          IP ADDRESS 1
                     A          202.158.218.233
myproxy1       A          IP ADDRESS 1
myproxy2       A          202.158.218.233

APAC DNS RECORDS 

myproxy       CNAME    myproxy.arcs.org.au
myproxy1     CNAME    myproxy1.arcs.org.au
myproxy2     CNAME    myproxy2.arcs.org.au

Things that Grid Admins need to do

Can be changed any time and are not absolutely necessary yet. A separate Change Note will be created before the old addresses are eventually removed.

Command line clients

Command line client - myproxy-logon etc. uses an environment variable to determine the server to speak to. You can check this by doing: 
env | grep MYPROXY
This needs to be myproxy.arcs.org.au and is normally set by scripts in /opt/vdt/post-setup loaded by /etc/profile.d/vdt_setup.{sh,csh}.

Other tools that use MyProxy

Need to change the host in their code. Already done for Grisu, but not Grix (30/6/08).

Testing Procedures

Try and create/get a myproxy proxy using command line tools.

Run the following on command line :-

  • ALERT! make sure myproxy env is set as stated above 

myproxy-init
myproxy-logon
myproxy-info

Back-out Procedures

  • Leave the old myproxy machines running after the change
  • Change the DNS records back

Review

30/06/08 Daniel - minor changes to explanation, would be nice to have actual IP addresses listed above and details of upgrade versions inc host ...

15/07/08 Joel - Tested with new myproxy server (myproxy2.arcs.org.au), everything works fine

17/07/08 Daniel - confirm possible problems with DNS changing on a user once created a proxy

Schedule/Notification

30/06/08 Announcement to techstaff@ and developers@ by Sam

Tracking completion

10/06/08 Successfully tested creating a proxy on old myproxy, copying it to new one and downloading

17/07/08 Ticket 1585 - problems with CSIRO firewall and new address!

State of Play

myproxy2.arcs.org.au given the old myproxy servers IP address to address firewall issues. New server running fine, need to start thinking about stage 2 and building another myproxy server.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200807-001 | 14 Jul 2008 - 20:52 | ---+!! TITLE

Description

This note describes the steps necessary to add timestamps to the output of Szonesync.pl.

Estimated Duration

30 seconds

Systems/Services Affected

SRB. at iVEC, eRSA, UQ, TPAC, ANUSF

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon

Detailed Instructions

as root:

run 

vi /usr/bin/Szonesync.pl

and add two lines (line 3 and 4) to function runCmd (at the bottom of file) so that it looks like 

sub runCmd {
    my($option, $cmd) = @_;
    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
    printf "[%4d-%02d-%02d %02d:%02d:%02d]",$year+1900,$mon+1,$mday,$hour,$min,$sec;
    print "running: $cmd\n";
    $cmdOutput=`$cmd`;

Testing Procedures

as srb server user:

run 

Szonesync.pl -u
and you'll see something like 
[2008-07-09 11:58:58]running: Spullmeta -F GET_CHANGED_USER_INFO 1997-01-01 > ngdev2.its.utas.edu.au.GET_CHANGED_USER_INFO 2> ngdev2.its.utas.edu.au.Spullmeta.stderr
[2008-07-09 11:58:58]running: Spullmeta -F GET_CHANGED_USER_INFO 1997-01-01 > ngdev2.its.utas.edu.au.GET_CHANGED_USER_INFO.orig
Timestamps are added to the beginning of lines where Spullmeta, Spushmeta or /usr/bin/ZoneUserSync.py is executed.

Back-out Procedures

as root:

run 

vi /usr/bin/Szonesync.pl

and delete the following lines from function runCmd (at the bottom of file) 

    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
    printf "[%4d-%02d-%02d %02d:%02d:%02d]",$year+1900,$mon+1,$mday,$hour,$min,$sec;

Review

tested. works. -- FlorianGoessmann - 09 Jul 2008 Looks good to me. Don't have a test server so will wait for the downtime to implement. StephenMcMahon - 09 Jul 2008

It works. -- KaiLu - 11 Jul 2008

Schedule/Notification

no downtime required.

Tracking completion

SAPAC: Done.

UQ: Done.

ANU: Done. StephenMcMahon - 11 Jul 2008

iVEC: done. FlorianGoessmann

TPAC: done. PaulineMak - 14 Jul 2008

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200807-002 | 25 Jul 2008 - 15:16 | ---+!! New GTest Cert for Inca test

Description

This note describes the steps necessary to add a new GTest Cert to inca user. The new GTest cert is owned by Will, who is now the main Inca person. Gerson's GTest cert will continue to be used but as a backup.

Estimated Duration

30 seconds

Systems/Services Affected

SRB. at iVEC, eRSA, UQ, TPAC, ANUSF

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon

Detailed Instructions

as srb server user: 
echo \"/C=AU/O=APACGrid/OU=QCIF/CN=William Hsu GTest\" inca@<YOUR DOMAIN> >> /etc/grid-security/grid-mapfile.srb

run 

Sinit
SmodifyUser addDN inca <YOUR_DOMAIN> '/C=AU/O=APACGrid/OU=QCIF/CN=William Hsu GTest'

Testing Procedures

as srb server user:

run 

SgetU -D inca
and you will see two certs like this 
-----------------------------------------------------------------
user_name: inca
domain_desc: ngspare.sapac.edu.au
distin_name: /C=AU/O=APACGrid/OU=QCIF/CN=William Hsu GTest
auth_scheme: GSI_AUTH
zone_id: ngspare.sapac.edu.au
-----------------------------------------------------------------
user_name: inca
domain_desc: ngspare.sapac.edu.au
distin_name: /C=AU/O=APACGrid/OU=SAPAC/CN=Gerson Galang GTest
auth_scheme: GSI_AUTH
zone_id: ngspare.sapac.edu.au

Back-out Procedures

as srb server user: delete line 
"/C=AU/O=APACGrid/OU=QCIF/CN=William Hsu GTest" inca@srb.ivec.org
from /etc/grid-security/grid-mapfile.srb

run 

SmodifyUser delDN inca <YOUR_DOMAIN> '/C=AU/O=APACGrid/OU=QCIF/CN=William Hsu GTest'

Review

all good. -- FlorianGoessmann - 18 Jul 2008

Schedule/Notification

Tracking completion

iVEC: done. -- FlorianGoessmann - 18 Jul 2008

TPAC: done. -- PaulineMak - 18 Jul 2008

UQ: Done. -- KaiLu - 18 Jul 2008

ANU: Done. -- StephenMcMahon - 24 Jul 2008

SAPAC: Done. -- ShundeZhang - 25 Jul 2008

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200807-003 | 27 Nov 2008 - 14:38 | ---+!! Cron Job for SRB Log Rotation

Description

This note describes the steps necessary to create a cron job for SRB log rotation

Estimated Duration

5 minutes

Systems/Services Affected

SRB. at iVEC, eRSA, UQ, TPAC, ANUSF

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon

Detailed Instructions

As root:

run 

   cd /usr/bin
   wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/SlogsCleanup.py?format=raw -O SlogsCleanup.py
   chmod +x SlogsCleanup.py

Make sure that you have installed the python interpreter on your system

As SRB server user:

Run 

   crontab -e

and add line 

   15 5 1 * * /usr/bin/SlogsCleanup.py -k 5 -d >> /usr/srb/data/log/logsCleanup.log 2>&1

The cron job will be run at 5:15 AM on the first of every month

Testing Procedures

Check if the script can be excuted

As SRB server user:

run 

   /usr/bin/SlogsCleanup.py -l

It works if a list of current SRB logs files are displayed (sorted by date)

Back-out Procedures

as srb server user:

run 

   crontab -e

and remove line 

   15 5 1 * * /usr/bin/SlogsCleanup.py -k 5 -d >> /usr/srb/data/log/logsCleanup.log 2>&1

as root:

remove file 

   rm -rf /usr/bin/SlogsCleanup.py

Review

looks good. -- FlorianGoessmann - 22 Jul 2008

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200808-002 | 07 Aug 2008 - 10:11 | ---+!! RT username and email address changes

Description

Two issues have arisen with the ARCS RT system over time:

  1. Current usernames are not consistent and it is not easily identifiable who owns a username, especially when assigning ownership of a ticket to another user.
  2. When emailing a comment to an RT ticket, the email address sending the email must match that of the registered user. This a problem with institution email addresses used for registration, then email sent using the @arcs.org.au email alias.

To fix this, each user account (for ARCS staff) will need to have the username changed to the ARCS current standard of firstname.lastname, and the email address of that user to firstname.lastname@arcs.org.au. This will most likely be done by hand.

Estimated Duration

It is anticipated that the change should take no longer than 1 hour to complete. This will include time for testing. During this time, the service will remain unaffected, but users will need to be notified that if their original username does not work, then to try the new ARCS standard username.

Systems/Services Affected

ARCS RT System: http://rt.arcs.org.au

Staff Responsible

AndyBotting

Detailed Instructions

Take a backup of the database, then one by one, change each of the ARCS staff usernames to firstname.lastname and email address to firstname.lastname@arcs.org.au.

Testing Procedures

Ask RT users to:

  1. log in using new username.
  2. test commenting on tickets via email, using their ARCS email address

Back-out Procedures

Restore the RT database from a backup taken prior to changes.

Review

  • 06/08/2008 Daniel - no need to wait an extra week since there will be no actual downtime or affect on ARCS users. Change procedure to cover ARCS staff only

Schedule/Notification

  • Sent an initial email on Thu 7th August to the Techstaff list. No need to notify other lists since only ARCS technical staff will be changed.
  • Make an announcement at the Monday 11th meeting
  • Send out an email on the morning of Tuesday 12 August, prior to the change
  • Send out an email confirming the change has taken place

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200808-003 | 10 Sep 2008 - 11:48 | ---+!! MyProxy? Change

Description

MyProxy? Cert CN Change and reverse DNS change

Estimated Duration

30 secs for cert change, up to 24 hours for DNS propagation

Systems/Services Affected

All myproxy clients - biggest user of myproxy is INCA

Staff Responsible

SamMorrison

Detailed Instructions

Change the certificate CN of myproxy2.arcs.org.au to myproxy.arcs.org.au

change the reverse DNS for IP 202.158.218.205 from myproxy2.arcs.org.au to myproxy.arcs.org.au

This is to try and fix an issue where clients complain that the forward lookup of myproxy aka. myproxy.arcs.org.au and the certificate (myproxy2.arcs.org.au) don't match. (It shouldn't care and only check the reverse lookup(currently matches the certificate)

See also http://wiki.arcs.org.au/bin/view/Main/ChangeNote200806-005

Testing Procedures

Using myproxy clients

see http://wiki.arcs.org.au/bin/view/Main/ChangeNote200806-005#Testing_Procedures

Back-out Procedures

Change DNS Back although this could take another 24hours to propergate Backup old cert/key

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200808-005 | 12 Sep 2008 - 14:09 | ---+!! TITLE Host OS upgrade for CSIRO grid gateway apacgridgw.hpsc.csiro.au

Description

Current operating system on host apacgridgw.hpsc.csiro.au is to be upgraded from CentOS 4.4 to CentOS 5.2
This will be a complete wipe of the current system and a fresh install of the new OS.
All VM's will be backed up (simple tar backup of the given volume) to cherax.hpsc.csiro.au and can then be extracted to new logical volumes once the host OS is setup.
Current host setup is currently being backed up using rsync (the whole thing, just in case) to cherax.hpsc.csiro.au.

Estimated Duration

1 Day (just to be safe in case anything goes wrong)
Services themselves should not be down for more than a couple of hours.

Systems/Services Affected

  • CSIRO grid gateway apacgridgw.hpsc.csiro.au
  • ng2.hpsc.csiro.au
  • ngdata.hpsc.csiro.au
  • nggums.hpsc.csiro.au
  • ngportal.hpsc.csiro.au
  • ng2dev.hpsc.csiro.au

Logs show little recent use, by: Terry Rankine, Will Hsu and Megan Hough/Monash.

Staff Responsible

Joel Ludbey-Bruhwel

Detailed Instructions

Stage 1
  • Backup current VM's
    • sudo tar -cvzpf /some/dir/backup.tar.gz --same-owner --atime-preserve /mounted/volume/

  • Backup current state of host

Stage 2

  • Wipe current host
  • Install CentOS 5.2 CentOS 5.2 host install
  • Configure network and authentication settings
    • /etc/hosts, /etc/resolv.conf, /etc/nsswitch.conf, /etc/sysconfig/network-scripts/ifcfg-eth0, /etc/krb5.conf, /etc/pam.d/system-auth
  • Restore all VM's

Stage 3

  • Test that VM's are up and running as expected
  • Install any third party software needed/desired by the host

Testing Procedures

  • ng2, submit some test jobs from globus client Test Suite
  • nggums, ensure that https://nggums.hpsc.csiro.au:8443/gums/ is up, attempt to map a known user, perform an update of members database.
  • ngdata, again submit a few test jobs from globus client Test Suite
  • ngportal, at this point in time this is not being used or working to my understanding. Possibly could be removed but will leave it be for the time being.
  • ng2dev, currently not doing anything important. Provided that it boot's it should be fine.

Back-out Procedures

  • Wipe the new install of CentOS5.2 and revert back to previous version (4.4)
  • Restore backups of the host using rsync backups
  • Restore all VM's

Review

26/08/08 Daniel - a suggestion to reduce risk and prevent relying on a single backup
  • backup as suggested
  • install new OS root volume into an LV on the existing disks (without clobbering anything)
  • setup grub to easily boot between old and new OS

Schedule/Notification

Developers list 28/8/08 11:40AM

Tracking completion

29/08/08 - Host upgraded successfully.
Issues starting ng2, ngdata and nggums.
04/09/08 - Finally got one of the VM's to boot, seemed to be some issue with kernel libraries on the host (????, not sure what) and needed to create custom ramdisk for VM's. Sam's notes helped but there were some slight differences to get stuff to work here.
VM's however, do not have any network access
08/09/08 - Finally solved all networking issues, xen was bridging a connection to the wrong interface, this has now been fixed.
udev doesn't start, but this is an expected issue.
Also noticed that on odd occassions, when starting a VM it will get about halfway through the boot process and then throws up a kernel panic.
No consistency with this, it appears to be random. Restarting the VM usually fixes the issue, looking around on the internet this is apparently a 'known' issue with running centOS4 VM's under a centOS5 host.
This shouldn't be a problem provided that ng2, ngdata and nggums are not restarted.
If this issue does occur, simply kill the consol, run "xm destroy guest_name" and re-create the guest. Should then boot without issue.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200808-006 | 10 Sep 2008 - 12:21 | ---+!! New ng2 for ESSCC cluster Savanna

Description

ESSCC has now commissioned the new cluster Savanna. The old cluster will be turned off, hopefully by the end of 2008 to free up UPS capacity. This is a good opportunity to build a new ng2 to interact with new cluster.

References: ARCS Ticket 1694.

Estimated Duration

Less than 1 hour to rename and test, but allocating 1 Day to be safe in case anything else goes wrong.

Systems/Services Affected

ng2.esscc.uq.edu.au will be offline for a short period and then be connected to Savanna instead of ESS.

This upgrade will prevent jobs from being sent via Grid to the old cluster ESS. Recent logs show only test jobs, so we don't expect this to cause any problems for users - but it will provide a benefit for those that need to use the new cluster.

If there is a need, ARCS could setup the old gateway with a different name to access ESS.

Staff Responsible

Will Hsu

Detailed Instructions

Build one CentOS 5 VM, configured to use new ESSCC authentication and home filesystem.

Boot up new ng2 under different dns name/ip and install PBSpro client and vdt software.

Modify nggums to use new ESSCC authentication.

Shutdown old ng2. Keep a backup!

Rename new ng2 to old ng2 according to this How to Upgrade NG2 document.

Configure globus, MDS/MIP.

Test job submission.

Testing Procedures

submit some test jobs from globus client to ng2 Test Suite

Back-out Procedures

Shutdown new ng2

Revert changes to nggums

Boot up old ng2

Review

2/09/08 Daniel - discuss options with Ryan and ask for input

10/09/08 Daniel

  • home file system should have nothing to do with GUMS?
  • CentOS4 is currently approved OS but we expect that the instructions work properly with CentOS5. This will be a good chance to test on a machine that is not currently receiving any real jobs
  • please try installing a new ng2 with a different name, and then rename following these instructions http://projects.arcs.org.au/trac/systems/wiki/HowTo/UpgradeNg2. To confirm that this process works and provides less down time

Schedule/Notification

Separate email discussion with Ryan, CC Terry and Darran on 2/09/08. They are happy to start using the new cluster - code will need to be recompiled for different architecture.

10/09/08 Update ticket 1694. Send a message to developers list.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200809-002 | 27 Nov 2008 - 15:09 | ---+!! Improving performance of MCAT with Vacuum and Indexing

Description

This note describes the steps necessary to run vacuum and index for first time and implement a cronjob to run weekly.

Estimated Duration

5 - 10 Minutes

Systems/Services Affected

SRB. at iVEC, eRSA, UQ, TPAC, ANUSF, ac3, VPAC

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins

Detailed Instructions

Login as user srb,

Run 

mkdir  /usr/srb/bin/VacuumIndexFiles
cd /usr/srb/bin/VacuumIndexFiles
wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/Vacuum_Index_First?format=raw -O Vacuum_Index_First
wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/Vacuum_Index_Cron?format=raw -O Vacuum_Index_Cron
psql -h <Srb server name> -d MCAT -U srb -f /usr/srb/bin/VacuumIndexFiles/Vacuum_Index_First >> /usr/srb/data/log/vacuum_index_first.log 2>&1

Above command will take a while depends on size of your MCAT database 

crontab -e

add this line to end of crontab( run every sunday at 1:15am). 

15 1 * * 0 psql -h <Srb server name> -d MCAT -U srb -f /usr/srb/bin/VacuumIndexFiles/Vacuum_Index_Cron >> /usr/srb/data/log/vacuum_index_cron.log 2>&1

Testing Procedures

Check /usr/srb/data/log/vacuum_index_first.log for any errors and check /usr/srb/data/log/vacuum_index_cron.log next monday morning.

Back-out Procedures

as srb user:

run 

crontab -e

and remove line 

15 1 * * 0 psql -h <Srb server name> -d MCAT -U srb -f /usr/srb/bin/VacuumIndexFiles/Vacuum_Index_Cron >> /usr/srb/data/log/vacuum_index_cron.log 2>&1

remove directory 

rm -rf  /usr/srb/bin/VacuumIndexFiles

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200809-003 | 19 Aug 2009 - 13:22 | ---+!! Configuring SRB for Automatic Account Creation

Description

This note describes the steps necessary to enable automatic account creation

Estimated Duration

30 minutes

Systems/Services Affected

SRB at iVEC, eRSA, UQ, TPAC, ANUSF, CSIRO, AC3, VPAC

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins, GarethWilliams

Detailed Instructions

As the srb user, run: 

    cd /usr/srb/bin/
    mkdir autoUserCreationScripts
    cd autoUserCreationScripts
    wget "http://projects.gridaus.org.au/trac/systems/browser/trunk/dataFabricScripts/addDomain.sh?format=raw" -O addDomain.sh
    wget "http://projects.gridaus.org.au/trac/systems/browser/trunk/dataFabricScripts/addGroup.sh?format=raw" -O addGroup.sh
    wget "http://projects.gridaus.org.au/trac/systems/browser/trunk/dataFabricScripts/addUser.sh?format=raw" -O addUser.sh
    wget "http://projects.gridaus.org.au/trac/systems/browser/trunk/dataFabricScripts/dnToUserDomain.py?format=raw" -O dnToUserDomain.py
    chmod 700 *

Then edit the file: 

    /usr/srb/bin/autoUserCreationScripts/dnToUserDomain.py

and change line 25 to match your institution's OU and your server's domain 

    domains = {
        'TPAC':'srb.tpac.org.au',
    }

Then edit the file: 

    /usr/srb/data/shibConfig

And replace the entire file with the following lines: 

# The full path to the dn to user/domain script
dnToUserDomainScript=/usr/srb/bin/autoUserCreationScripts/dnToUserDomain.py
# The full path to the add user script
addUserScript=/usr/srb/bin/autoUserCreationScripts/addUser.sh 
# The full path to the add group script
addGroupScript=/usr/srb/bin/autoUserCreationScripts/addGroup.sh
# The full path to the add domain script
addDomainScript=/usr/srb/bin/autoUserCreationScripts/addDomain.sh
# The OID of the certificate extension
shibCertOid=1.2.3.4.5.6.7.8.9
# The group separator in the certificate extension
shibGroupSeparator=;

As root, run: 

cd /etc/grid-security
wget http://www.arcs.org.au/slcs/arcs-slcs-ca.tar.gz -O - | tar xvz
cd arcs-slcs-ca
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.0?format=raw" -O 663bbd41.0
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.namespaces?format=raw" -O 663bbd41.namespaces
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.signing_policy?format=raw" -O 663bbd41.signing_policy
chown root:root *
chmod 644 *
cp * /etc/grid-security/certificates

To make the certificates are retained between certificate updates, edit the file /opt/vdt/vdt/etc/vdt-update-certs.conf and add the following lines: 

include=/etc/grid-security/arcs-slcs-ca/1ed4795f.0
include=/etc/grid-security/arcs-slcs-ca/1ed4795f.namespaces
include=/etc/grid-security/arcs-slcs-ca/1ed4795f.signing_policy
include=/etc/grid-security/arcs-slcs-ca/663bbd41.0
include=/etc/grid-security/arcs-slcs-ca/663bbd41.namespaces
include=/etc/grid-security/arcs-slcs-ca/663bbd41.signing_policy

Restart the SRB server.

Testing Procedures

Create a new SRB connection to your SRB server using Shibboleth as the authentication method. This should create a valid connection. If a connection is not created, check your SRB log (/usr/srb/data/logs) for error messages.

Note: Shibboleth authentication will NOT work if you are behind a proxy - this is because the SLCS-client java library must be repackaged with your proxy username and password.

Back-out Procedures

As root, remove the installed certificates: 

    cd /etc/grid-security/certificates
    rm -f 1ed4795f.*
    rm -f 663bbd41.*
    rm -rf /etc/grid-security/certificates/arcs-slcs-ca

Then remove the user creation scripts directory: 

    rm -rf /usr/srb/bin/autoUserCreationScripts/

Edit the file: 

    /opt/vdt/vdt/etc/vdt-update-certs.conf

And remove these lines: 

    include=/etc/grid-security/arcs-slcs-ca/1ed4795f.0
    include=/etc/grid-security/arcs-slcs-ca/1ed4795f.namespaces
    include=/etc/grid-security/arcs-slcs-ca/1ed4795f.signing_policy
    include=/etc/grid-security/arcs-slcs-ca/663bbd41.0
    include=/etc/grid-security/arcs-slcs-ca/663bbd41.namespaces
    include=/etc/grid-security/arcs-slcs-ca/663bbd41.signing_policy

Restart the SRB server

Review

-- FlorianGoessmann - 19 Sep 2008

Done for ANU. -- StephenMcMahon - 17 Nov 2008

Schedule/Notification

Tracking completion

  • UQ: Done. Not yet tested because our Idp does not work now. -- KaiLu - 19 Sep 2008
  • ANU: Also done and not tested for the same reason as Kai. -- StephenMcMahon - 22 Sep 2008
  • VPAC: Done. Not yet tested. -- GrahamJenkins - 22 Sep 2008
  • ac3: Done. Not tested for same reason. -- SridharReddapani - 22 Sep 2008.
  • CSIRO: Done. -- GarethWilliams - pre 27 Nov 2008

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200811-001 | 27 Nov 2008 - 15:10 | ---+!! slcs1 certificates for automatically created accounts

Description

Estimated Duration

30 minutes

Systems/Services Affected

SRB at iVEC, eRSA, UQ, TPAC, ANUSF, CSIRO, AC3, VPAC

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins, GarethWilliams

Detailed Instruction

If you have not installed the script, please see this change note

http://wiki.arcs.org.au/bin/view/Main/ChangeNote200809-003

and follow the instructions before applying this change note.

As the srb user and move the existing dnToUserDomain.py: 

cd /usr/srb/bin/autoUserCreationScripts
mv dnToUserDomain.py dnToUserDomain.py.orig

Then download the new file and make sure it has executable permission for the srb user: 

wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/dnToUserDomain.py?format=raw -O dnToUserDomain.py
chmod 700 dnToUserDomain.py

Edit line 80 of dnToUserDomain.py to include IdPs that your server will accept (ac3 will need to add AC3 and ANSTO, while VPAC will need to add VPAC and ARCS OpenIdP), e.g.

Where the first value is 'O' part of the slcs1 DN, and the the second value is the name of your SRB domain. 

    domains = {
         'VPAC':'srb.vpac.org',
        'ARCS OpenIdP':'srb.vpac.org'
    }

Since we're only using the slcs1 certificates, we can remove the slcstest CA certificates.

Become the root user: 

cd /etc/grid-security/certificates/
rm 663bbd41.0
rm 663bbd41.namespaces
rm 663bbd41.signing_policy
cd /etc/grid-security/arcs-slcs-ca/
rm 663bbd41.0
rm 663bbd41.namespaces
rm 663bbd41.signing_policy

Then edit 

/opt/vdt/vdt/etc/vdt-update-certs.conf
and remove the following lines: 

include=/etc/grid-security/arcs-slcs-ca/663bbd41.0
include=/etc/grid-security/arcs-slcs-ca/663bbd41.namespaces
include=/etc/grid-security/arcs-slcs-ca/663bbd41.signing_policy

Restart the SRB server.

Testing Procedures

Attempt to connect to your SRB server using Hermes 1.0rc8. This version uses slcs1 certificates.

Back-out Procedures

Get the previous version of the script and overwrite the dnToUserDomain.py script.

As the srb user: 

cd /usr/srb/bin/autoUserCreationScripts
wget 'http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/dnToUserDomain.py?rev=718&format=raw' -O dnToUserDomain.py

and change line 25 to match your institution's OU and your server's domain 

    domains = {
        'TPAC':'srb.tpac.org.au',
    }

Then add the slcstest certificates.

As the root user: 

cd /etc/grid-certificates/arcs-slcs-ca/
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.0?format=raw" -O 663bbd41.0
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.namespaces?format=raw" -O 663bbd41.namespaces
wget "http://projects.arcs.org.au/trac/slcs-client/attachment/wiki/WikiStart/663bbd41.signing_policy?format=raw" -O 663bbd41.signing_policy

Then edit the file 

/opt/vdt/vdt/etc/vdt-update-certs.conf
and insert the following lines: 

include=/etc/grid-security/arcs-slcs-ca/663bbd41.0
include=/etc/grid-security/arcs-slcs-ca/663bbd41.namespaces
include=/etc/grid-security/arcs-slcs-ca/663bbd41.signing_policy

Restart the SRB server.

Review

Schedule/Notification

Tracking completion

  • ac3/Intersect: Done. SridharReddapani - 17/11/2008
  • TPAC: Done. -- PaulineMak - 17/11/2008
  • VPAC: Done. -- GrahamJenkins - 17/11/2008 (both srb.vpac.org and srbdev.vpac.org)
  • ANU: Done. -- StephenMcMahon - 17 Nov 2008. Highlighted an issue with the NCI National Facility IDP but that is now working.
  • eResearch SA: Done. -- ShundeZhang - 20 Nov 2008 (both srb.sapac.edu.au and ngspare.sapac.edu.au)
  • CSIRO: Done -- GarethWilliams - pre 27 Nov 2008

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200811-002 | 02 Dec 2008 - 10:32 | ---+!! Installing data fabric usage scripts

Description

This will install data fabric usage scripts at individual sites.

Estimated Duration

10 minutes

Systems/Services Affected

SRB at iVEC, eRSA, UQ, TPAC, ANUSF, CSIRO, AC3, VPAC

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins, GarethWilliams

Detailed Instructions

As the root user:

Install postgres module for Python, run: 

  yum install postgresql-python.i386

As the srb user, run:

Create and enter the directory: 

  mkdir /usr/srb/bin/usageScripts
  cd /usr/srb/bin/usageScripts
Download the required scripts: 
  wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/usageScripts/usageFromMCAT.py?format=raw -O usageFromMCAT.py
  wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/usageScripts/UsageStatsUpload.sh?format=raw -O UsageStatsUpload.sh
Make these scripts executable: 
  chmod 700 *

Make a directory locally to store XML output files generated by the scripts, e.g. 

  mkdir xml

If you have folders under /projects please make sure these folders are owned by a group. Without this setting, nothing will be collected by the usage script for that folder. You can change the ownership of a folder like so: 

Schmod o <groupname> groups <folder name>

Setup the cron jobs crontab -e, then add the following line: 

  1 0 * * * /usr/srb/bin/usageScripts/UsageStatsUpload.sh -d /usr/srb/bin/usageScripts/xml/

Testing Procedures

To do a quick test to see if it's working, run: 

   /usr/srb/bin/usageScripts/UsageStatsUpload.sh -d /usr/srb/bin/usageScripts/xml/

You should see XML files in the local directory /usr/srb/bin/usageScripts/xml/ and also on the data fabric, under /<zone name>/projects/dataFabricStats/.

Note that the full access permission for the XML files under this collection has been granted to srbAdmin@srb.hpcu.uq.edu.au (the user that will read the XML files using the processing scripts) automatically when the script is run.

Back-out Procedures

as the srb user:

run 

   crontab -e

and remove line 

   1 0 * * * /usr/srb/bin/usageScripts/UsageStatsUpload.sh -d /usr/srb/bin/usageScripts/xml/

remove directory 

   rm -rf /usr/srb/bin/usageScripts
   rm -rf /usr/srb/data/log/useLog

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200811-003 | 02 Dec 2008 - 12:16 | ---+!! Davis for SRB

Description

Davis is ready for beta test. please install it on your production box and use it!

Estimated Duration

15-30 mins

Systems/Services Affected

All

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins, GarethWilliams

Detailed Instructions

Ensure your https port (443) is opened to the public. See http://projects.arcs.org.au/trac/davis/wiki/HowTo/Install#Jetty6standalone and http://projects.arcs.org.au/trac/davis/wiki/HowTo/Configuration

Testing Procedures

See http://projects.arcs.org.au/trac/davis/wiki#HowtoUse

Back-out Procedures

Stop Davis; Delete davis home folder, /etc/init.d/jetty.sh, /etc/default/jetty

Review

Had to chmod +x /etc/init.d/davis

Location of jetty.xml was not specified. Exact fields to change were not specified either.

-- StephenMcMahon - 27 Nov 2008

I've modified the doc. -- ShundeZhang - 02 Dec 2008

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200811-004 | 02 Dec 2008 - 10:41 | ---+!! Fixing bugs in dnToUserDomain.py script

Description

Update dnToUserDomain.py script with some bug fixes

Estimated Duration

10 minutes

Systems/Services Affected

SRB at iVEC, eRSA, UQ, TPAC, ANUSF, CSIRO, AC3, VPAC

Staff Responsible

FlorianGoessmann, ShundeZhang, KaiLu, PaulineMak, StephenMcMahon, SridharReddapani, GrahamJenkins, GarethWilliams

Detailed Instructions

If you have not installed the script, please see this change note

http://wiki.arcs.org.au/bin/view/Main/ChangeNote200809-003

and follow the instructions before applying this change note.

As the srb user: 

cd /usr/srb/bin/autoUserCreationScripts/
mv dnToUserDomain.py dnToUserDomain.py.old
wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/dnToUserDomain.py?format=raw -O dnToUserDomain.py
chmod 700 dnToUserDomain.py
wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/DNCONFIG.py?format=raw -O DNCONFIG.py

Then modify DNCONFIG.py to match your institution's OU and your server's domain. 

domains = {
    'TPAC':'srb.tpac.org.au',
 }

Testing Procedures

Try logging in with Hermes, using both shibboleth and GSI. You should be able to log in with your existing account(s). Then try running dnToUserDomain.py with other DNs, either using slcs or APACGrid certificates, e.g. 

./dnToUserDomain.py '/DC=au/DC=org/DC=arcs/DC=slcs/O=TPAC/CN=Pauline Mak 78ISwLgFjnpuw14IZ4X8XYbIp-4'

or 

./dnToUserDomain.py '/C=AU/O=APACGrid/OU=TPAC/CN=Pauline Mak'

Make sure your DN matches something in the DNCONFIG.py file. The script should print out something with a username@domain. If the username is too long, then it will be shortened to something like a.b.surname@domain.

Back-out Procedures

As the srb user: 

cd /usr/srb/bin/autoUserCreationScripts/
mv dnToUserDomain.py.old dnToUserDomain.py
rm DNCONFIG.py

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200812-002 | 22 Dec 2008 - 14:27 | ---+!! TWiki Upgrade from 4.1 to 4.2.4

Description

A cross site scripting vulnerability have been discovered, and a TWiki upgrade is required for security reasons.

Estimated Duration

One day

Systems/Services Affected

All ARCS hosted TWiki services, including the ARCS Wiki, Exec Wiki and all other webs created for research groups.

Staff Responsible

AndyBotting, LevLafayette

Detailed Instructions

A copy of the 4.2.4 TWiki code has been installed, parallel with the existing 4.1 TWiki code, and data has been syncronised between the two via a manual method. The new site will be available as http://wikitest.arcs.org.au from now, up to the change-over time. During this time, it is encouraged that ARCS staff test the site as much as possible.

Once the change-over commenced, the existing TWiki will be marked as read-only, to prevent updates while the data is being migrated to the new TWiki. Once completed, the Apache will be configured to use the new TWiki directory.

Testing Procedures

As much testing will be done as possible before the switch, but due to the complex nature of TWiki installations, it is expected that some minor issues may arise, which will be worked through.

Back-out Procedures

If any major issues arise during the change-over, we can switch back to the old code-base as it should remain untouched.

Review

Schedule/Notification

An email was sent out on 16th Dec 2008 to all registered TWiki users outlining the upgrade procedure.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200903-001 | 01 Apr 2009 - 03:28 | ---+!! Apache configuration change on MARCS IdPs

Description

We are experiencing some problems when the SLCS server opens a connection to an IdP to resolve the Shibboleth Artifact. It has been determined that a minor change to the Apache configuration on the Idp will stop this error from occurring. There will be a small performance hit, but the reliability of the IdP is deemed more important that the performance. We will also take this opportunity to replace the schacCountryOfResidence with the l (locality) attribute.

Estimated Duration

10 minutes.

Systems/Services Affected

Authentication services provided by the IdPs.

Staff Responsible

DarranCarey, GarthDenley, LeighGordon, WillHsu, JoelLB, SamMorrison, PaulWarren, SimonYin

Detailed Instructions

Make a backup copy of /etc/httpd/conf.d/ssl.conf before proceeding

The file /etc/httpd/conf.d/ssl.conf should have any existing entries of the form: 

SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)

commented out, and a new line should be added: 

SSLSessionCache         none

The Apache server will need to be restarted for the configuration change to take affect.

Make a backup copy of /usr/local/shibboleth-idp/etc/resolver.ldap.xml and /usr/local/shibboleth-idp/etc/arps/arp.site.xml before proceeding

Edit /usr/local/shibboleth-idp/etc/resolver.ldap.xml on the IdP, and in the 

<StaticDataConnector id="staticLibraryEPE">
section change: 

<Attribute name="schacCountryOfResidence">

to 

<Attribute name="l">

and comment out or remove the section: 

<Attribute name="urn:mace:terena.org:schacCountryOfResidence">
        <AnyValue release="permit"/> 
</Attribute>

Tomcat will need to be restarted.

Testing Procedures

Verify that you can authenticate against an IdP by visiting https://slcs1.arcs.org.au/SLCS/login

Back-out Procedures

If for some reason you cannot authenticate against an IdP, restore the file /etc/httpd/conf.d/ssl.conf, and restart the Apache server. Perform the testing procedures again.

Review

Schedule/Notification

The changes are scheduled for Monday the 23rd of March, An email notification will be sent to all affected users on Monday 16th of March, informing them of the change.

Tracking completion

PaulWarren - completed for NCI NF

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200903-002 | 03 Apr 2009 - 09:47 | ---+!! Migrating ARCS mail from VPAC to Google Apps

Description

ARCS has decided to move it's mail to a Google App domain. This means that mail intended for ARCS staff will be stored in a separate mailbox, as opposed to forwarding the mail to the staff member's institution email account.

Estimated Duration

The process should take anywhere from a few hours to a few days.

Systems/Services Affected

All systems relying on mail:

  • Staff email
  • Mailing lists (mailman)
  • Help desk tickets (RT)

Staff Responsible

Detailed Instructions

VPAC is currently the MX host for arcs.org.au, and this will change to Google. While this is good for staff mailboxes, it makes it is inconvenient for other systems which rely on mail to @arcs.org.au. This includes the mailing lists and RT system.

To overcome this, we need to create Google mailing lists on the Google App domain to forward mail from the @arcs.org.au domain to @lists.arcs.org.au domain, which will be hosted at VPAC to process mailing lists and RT messages. The mail server at VPAC will then be configured to handle mail from @lists.arcs.org.au.

For information about configuring mail clients for POP/IMAP and SMTP to their Google Apps mail, you can see the documentation by Google at http://mail.google.com/support/a/arcs.org.au/bin/topic.py?topic=12773

Basic connection information:

Incoming Mail (IMAP) Server

Outgoing Mail (SMTP) Server

  • Server: smtp.gmail.com
  • Username: firstname.lastname@arcs.org.au
  • Use Authentication: Yes
  • Use STARTTLS: Yes (some clients call this SSL)
  • Port: 465 or 587

Testing Procedures

Once completed, it will take some time for the DNS MX records to be propagated. Once it is done, we will need to test staff email addresses, mailing lists and RT queues.

Back-out Procedures

DNS records will be reverted back to VPAC and the system will carry on as it was before.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200904-001 | 11 Jun 2009 - 14:13 | ---+!! Migrate ARCS Plone Service to VPAC

Description

Migration of an existing system to new infrastructure. Purpose is to strategically position the Plone infrastructure in a more centrally connected MARC and leverage the systems provided by the Data Services team.

Estimated Duration

One hour. Which is the time taken for DNS entry to expire.

Systems/Services Affected

www.arcs.org.au

Staff Responsible

RussellSim (Plone Migration)

AndyBotting (DNS change)

Detailed Instructions

  1. Turn off Zope service on zope9.cluster on the JCU cluster
  2. Turn off Zeo service on zeo2.cluster on the JCU cluster
  3. Change Zeo Database to READ-ONLY
  4. Turn on Zeo service on zeo2.cluster on the JCU cluster
  5. Turn on Zope service on zope9.cluster on the JCU cluster
  6. Copy Data file from zeo2.cluster to db-vpac.arcs.org.au at VPAC.
  7. Turn on Zeo service at VPAC
  8. Turn on Zope service (plone.arcs.org.au) at VPAC
  9. Change ARCS website cache service url in portal_squidtool
  10. Adjust DNS entry from: 
www.arcs.org.au.   3600   IN   CNAME   plone.jcu.edu.au.
to 
www.arcs.org.au.   3600   IN   CNAME   plone.arcs.org.au.

Test

Testing Procedures

  1. Local data access.
  2. Check DNS entry.

Back-out Procedures

  1. Point DNS entry back at JCU and
  2. Turn infrastructure back on at JCU.
  3. Test Plone service is operating within expected paramaters at JCU.
  4. Wait for DNS entry to expire.
  5. Test DNS entry.

Review

Schedule/Notification

15th June, Notification sent on 8th June

Tracking completion

RT Ticket

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-001 | 07 May 2009 - 12:58 | ---+!! Migration of critical grid infrastructure virtual machines at VPAC

Description

Currently we have 9 virtual machines being hosted on hardware which is out of warranty. With newer hardware set up and running, we are now in a position to migrate these virtual machines over.

The machines in question are:

  • grid.vpac.org
  • hudson.vpac.org (for Grisu/Grix builds)
  • ng2.vpac.org
  • ng2dev.vpac.org
  • myproxy2.arcs.org.au
  • nggums.vpac.org
  • ngportal.vpac.org
  • ngportaldev.vpac.org
  • vomrs.arcs.org.au

Estimated Duration

2 hours for the whole procedure, but only about 20 minutes per machine.

Systems/Services Affected

Most of these virtual machines are infrastructure for the grid computing at VPAC, but vomrs.arcs.org.au and myproxy2.arcs.org.au are critical for grid jobs ARCS wide.

During the migration, these machines in order will be taken down for up to 20 minutes, affecting all grid jobs.

Staff Responsible

AndyBotting, SamMorrison

Detailed Instructions

The migration procedure is:

  1. Create a new VM container on either arcs-vh01.vpac.org or arcs-vh02.vpac.org, using the virt-install command.
  2. Shut down the VM running on xen-d.vpac.org
  3. Using SCP, copy the VM image from the old box, to the new
  4. While the copying is taking place, get the MAC address out of the Xen config file
  5. Using virsh edit _vmname_ on the new VM host, replace the MAC address with the original
  6. Once the copy has completed, start the VM on the new host.
  7. Rinse and repeat for each of the VM's.

Testing Procedures

Boot each VM and watch the console for messages. Once it's done, SSH in and check everything is sane. Run some test grid jobs.

Back-out Procedures

Boot the VM's again on the original Xen host

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-002 | 14 May 2009 - 20:08 | ---+!! db-1.arcs.org.au database server rebuild

Description

Needs to be consistent with db-2 (currently one has 32bit OS and the other 64bit)

Estimated Duration

2 hr

Systems/Services Affected

mysql database service

Staff Responsible

Gareth Williams, Darran Carey

Detailed Instructions

Make both an explicit backup of the mysql databases files and and an extra dump intend to restore from the files - falling back to the dump, falling back to the slave, falling back to an old backup...

Phase 1 - preparation. Things to note/backup (there is some documentation at http://projects.arcs.org.au/trac/systems/wiki/DataServices/Database - in particular the mysql install and backup sections

  • copy/check script /usr/local/bin/mysql_backup.sh (Gareth)
  • copy/check sudo settings (Gareth/Darran)
  • copy/check iptables settings (Darran)
  • copy custom mysqld service script (though I'm going to replace it) (Gareth)
  • copy/note network/host settings (Darran)
  • plan desired disk/OS configuration (Darran with Andy/Jim/Gareth)
  • clean out old binary logs to reduce clutter (Gareth)

Phase 2 - outage pre-shutdown

  • optionally lock out clients (not sure how... maybe break firewall - must be a better way) (Gareth/Darran)
    • start a root mysql client and run "FLUSH TABLES WITH READ LOCK;" - leave mysql session open to preserve lock until service stopped
  • run a full backup and copy to pbstore (Darran)
  • stop mysqld service
  • make a binary copy of /data/mysql (root tar) and copy it to pbstore (Darran)

Phase 3 - rebuild and restore

  • rebuild host with 64 bit OS and disk configuration like db-2 (Darran)
  • setup networking and restore ssh access (Darran)
  • setup arcs-admin user and add ssh authorized_keys (from db-2 maybe with mods) (Darran)
  • setup sudo for arcs-admin (Darran)
  • copy full mysql backup/dump to a scratch area (Darran)
  • copy binary backup to a scratch area (Darran)
  • unpack binary backup to new area and sets up service (install mysql-server with yum if necessary) (Gareth)
  • restore other iptables setup (for mysql access) (Darran)
  • check, check, check, including replication (Gareth)
  • re-setup backup as per documentation on projects site (Gareth)
  • notify clients (Gareth)

No need to further document alternative mysql restore strategies at this point.

Testing Procedures

can connect to database server; databases are present and useable

Back-out Procedures

No provision is made for full restore to current state. Once the reinstall is started we are committed to building a new working platform onto which to restore the service.

Review

Schedule/Notification

clients notified 05 May 2009 - only specific database clients and directly involved staff outage to commence 06 May 2009 12pm EST rescheduled for 14 May 10am EST

Tracking completion

  • prior to start, bcees, arcs_odl and irodsUsage were moved to db-2
  • service locked at 9.55am EST
  • phases 1 and 2 complete at about 10.10am EST
  • base rebuild done and admin access restored at 11.53am EST
  • disk reconfig and physical move within rack complete Darran's work at 1.50-pm EST
  • Sridhar back in to set up postgresql at that time.
  • mysql configured for new location and started by about 3pm EST but firewall needs fixes
  • update sent to clients at 4.47pm EST
  • firewall fixed around 7pm EST - replication recommenced
  • final notification sent at 8pm EST

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-003 | 18 May 2009 - 09:43 | ---+!! TITLE

Description

Change slcs1.arcs.org.au entityID

Estimated Duration

1 hour

Systems/Services Affected

All IdP's that can access SLCS

Staff Responsible

SamMorrison

Detailed Instructions

Change the shibboleth entityID from urn:mace:federation.org.au:testfed:slcs1.arcs.org.au to https://slcs1.arcs.org.au/shibboleth

IdP Instructions

Make a copy of any files to be changed

*Once the changes are complete you don't need to restart tomcat or any other service

Shibboleth 1.3

Edit $SHIBBOLETH_HOME/etc/arps/arp.site.xml

Add 

<Rule>
     <Description>SLCS 1 New</Description>
         <Target>
              <Requester matchFunction="urn:mace:shibboleth:arp:matchFunction:exactShar">https://slcs1.arcs.org.au/shibboleth</Requester>
        </Target>
        <Attribute name="urn:mace:federation.org.au:attribute:auEduPersonSharedToken">
            <AnyValue release="permit"/>
        </Attribute>
    </Rule>

Shibboleth2

Edit /etc/shibboleth/attribute-map.xml (could be different at each site) 

  <AttributeFilterPolicy>

    <PolicyRequirementRule xsi:type="basic:OR">
      <basic:Rule xsi:type="basic:AttributeRequesterString" value="urn:mace:federation.org.au:testfed:slcs1.arcs.org.au" />
      <basic:Rule xsi:type="basic:AttributeRequesterString" value="https://slcs1.arcs.org.au/shibboleth" />
      <basic:Rule xsi:type="basic:AttributeRequesterString" value="https://services.arcs.org.au/shibboleth" />
    </PolicyRequirementRule>

    <AttributeRule attributeID="auEduPersonSharedToken">
      <PermitValueRule xsi:type="basic:ANY" />
    </AttributeRule>
  </AttributeFilterPolicy>

Testing Procedures

One user of each IdP needs to visit https://slcs1.arcs.org.au/SLCS/login

Back-out Procedures

Change name back

Review

Schedule/Notification

Tracking completion

VPAC Done
Canterbury Done
IVEC Done
TPAC Done
NCI NF ?Not used
eRSA Done
ARCS Done
ANSTO Done
CSIRO Done
ac3 Done

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-004 | 29 May 2009 - 15:12 | ---+!! Update iRODS User Records

Description

Update the iRODS user records so that shared tokens (where present) are stored in the 'user_info' field. This will make it possible for Shibboleth logins to be performed using Davis.

Estimated Duration

45 minutes.

Systems/Services Affected

arcs-df.ivec.org, arcs-df.vpac.org, arcs-df.ac3.edu.au, arcs-df.hpcu.uq.edu.au

Staff Responsible

Shunde Zhang, Graham Jenkins.

Detailed Instructions

  • Do: iadmin lu graham.jenkins .. check that 'user_info' field contains: VI8SEdbk_8Ph3E7M1O8jdORVTF4
  • Do: iadmin lu graham.jenkins1 .. check that 'user_info' field is empty.
  • Do: export irodsConfigDir=/data/rods/iRODS/server/config
  • Do: /data/rods/iRODS/server/bin/cmd/createUser "/C=AU/O=APACGrid/OU=VPAC/CN=Jane Doe" .. check result
  • Do: 
/data/rods/iRODS/server/bin/cmd/createUser "/DC=au/DC=org/DC=arcs/DC=slcs/O=TPAC/CN=Jane Doe VI8SEdbkx8Ph3E7M1O8jdABRA99"
.. check result
  • Do: iadmin rmuser jane.doe ; iadmin rmuser jane.doe1

Back-out Procedures

  • Change non-empty 'user-info' fields so that they contain: 1
  • Change empty ones so that they contain: 0
  • Do: cp /data/rods/iRODS/server/bin/cmd/createUser.PRE /data/rods/iRODS/server/bin/cmd/createUser
  • Test as above

Review

Schedule/Notification

Implementation date to be advised.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-005 | 26 May 2009 - 13:22 | ---+!! ARCS Mail server migration

Description

The ARCS mail server will be migrated from its current host, to a new virtual machine.

This is for two reasons:

  1. The current host has a outdated OS, and upgrading is not an option.
  2. The current host has a hardware fault
  3. The version of Mailman installed is outdated, and a new version is required for fixing some issues highlighted by the AuScope group.

Estimated Duration

9am - 5pm, Saturday 6th June

Systems/Services Affected

ARCS services affected by this change are:
  • RT
  • Mailing lists

Staff Responsible

Detailed Instructions

  1. The current postfix configuration will be transferred from isis.vpac.org to lists.vpac.org.
  2. A new version of mailman will be installed and configured, with the archives transferred.
  3. The RT mailgate tool will be installed for delivering mail into RT.

Testing Procedures

Mail will be sent to RT and the postfix mailing lists to ensure they are accepting and delivering mail correctly.

RT Queues:

  • Auth
  • CertAlert?
  • Collaboration
  • CollabSprints?
  • Customer
  • Data
  • DataSprints?
  • General
  • Grisu
  • Grix
  • Inca
  • Mailman
  • Phone
  • Systems
  • Test
  • VideoCollaboration?
  • WebContent?
  • Wiki

Mailing lists:

  • Allstaff
  • Arcs_admin
  • Auscope
  • Auscope-dev
  • Auscope-geosciml
  • Auscope-geosciml-testbed
  • Cloudadvice
  • Community
  • DataMINX?
  • Developers
  • Exec
  • Exec_conf
  • Fullmoon
  • Fullmoon-dev
  • Macddap-ref-group
  • Managers
  • Systems
  • Techstaff

Back-out Procedures

If we cannot complete the migration in the current timeframe, the old mail server will be re-instated.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-006 | 26 May 2009 - 13:20 | ---+!! wiki.arcs.org.au upgrade

Description

The TWiki software hosting wiki.arcs.org.au will be upgraded to the latest version.

A cross-site scripting vulnerability has been discovered. For more information, visit: http://twiki.org/cgi-bin/view/Codev/SecurityAlert-CVE-2009-1339

To protect from this, TWiki will be upgraded to the current version, 4.3.1.

Estimated Duration

9-10am, Tuesday 2nd June, 2009 (1 hour)

Systems/Services Affected

http://wiki.arcs.org.au

Staff Responsible

AndyBotting

Detailed Instructions

Data will be migrated from the arcs.org.au VM to the hosting.arcs.org.au VM. Once migrated, the upgrade tarball will be extracted over the top of the exisiting TWiki installation, and tested.

If the testing proves that the upgrade was a success, then the DNS records will be switched to point to the new host.

Testing Procedures

Simple tests to ensure the the TWiki is working correctly are:
  • Logging in
  • Modify some pages
  • Test TWiki configure script

Back-out Procedures

The current TWiki install will be kept running, so at any time we can switch back to the old version.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-007 | 28 May 2009 - 11:19 | ---+!! SLCS Delegation Service (SDS)

Description

Update SLCS software on slcs1.arcs.org.au to enable the SDS

Estimated Duration

1 hour

Systems/Services Affected

All services that rely on SLCS. DataFabric, Grid.

Staff Responsible

SamMorrison

Detailed Instructions

Redeploy the SLCS WAR file with update code. Restart Tomcat server. All this done at 8am

Testing Procedures

Use test client on plonedev.arcs.org.au

Back-out Procedures

Save a backup of old WAR file. Redploy old one if something wrong.

Review

Schedule/Notification

8am Friday. - install WAR and restart tomcat

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-008 | 09 Jun 2009 - 13:15 | ---+!! Running Davis as unprivileged davis user

Description

Running Davis as unprivileged davis user with Apache.

Estimated Duration

1 hours

Systems/Services Affected

ARCS Data Fabric

Staff Responsible

Shunde Zhang

Detailed Instructions

Stop Davis

Install Apache and Apache SSL module, configure it properly

Modify $DAVIS_HOME/etc/jetty.xml to enable ajp interface (port 8009) and disable SSL interface 

  <Call name="addConnector">
    <Arg>
       <New class="org.mortbay.jetty.ajp.Ajp13SocketConnector">
         <Set name="port">8009</Set>
         <Set name="ThreadPool">
           <New class="org.mortbay.thread.BoundedThreadPool">
             <Set name="minThreads">50</Set>
             <Set name="maxThreads">500</Set>
             <Set name="MaxIdleTimeMs">0</Set>
         </New>
         </Set>
         <Set name="statsOn">false</Set>
       </New>
    </Arg>
  </Call>
Add the following to /etc/httpd/conf.d/ssl.conf 
ProxyPass /ARCS/ ajp://localhost:8009/ARCS/ flushpackets=on
RewriteEngine On
RewriteRule ^/$ /ARCS/home [R]

create user davis, change owner of davis directory to davis user.

Modify /etc/default/davis to use davis user to start Davis. 

JETTY_HOME=/opt/davis/davis
JAVA_OPTIONS="-server -Xms1024m -Xmx1562m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC"
JAVA_HOME=/usr/java/jdk1.6.0_12
JETTY_USER=davis
JETTY_PID=/var/run/davis.pid
Restart Apache.

Copy dojoroot to /var/www/html

Start Davis.

Testing Procedures

Test Davis.

Back-out Procedures

Stop Apache, use Jetty for SSL

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-009 | 28 May 2009 - 11:29 | ---+!! GSI patch for iRODS

Description

Patch iRODS to fix a bug with GSI authentication.

Estimated Duration

10 mins

Systems/Services Affected

All

Staff Responsible

Shunde Zhang

Detailed Instructions

1. Replace the existing $IRODS_HOME/server/api/src/rsGsiAuthRequest.c with https://projects.arcs.org.au/trac/davis/attachment/wiki/ReleaseNotes/0.7.0/rsGsiAuthRequest.c

2. Go to $IRODS_HOME and run make

3. restart iRODS.

Testing Procedures

Use Davis 0.7.0 for test.

Back-out Procedures

You can back up your old rsGsiAuthRequest.c before making the change, or get it from iRODS 2.0.1 tarball.

Go to $IRODS_HOME and run make

restart iRODS.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-010 | 05 Jun 2009 - 18:02 | ---+!! Upgrade TDS to use Jetty 6.1.18 and Apache

Description

Update Jetty to 6.1.18 with a few bug fixes and also security updates.

Estimated Duration

1 hour

Systems/Services Affected

OPeNDAP? (THREDDS Data Server) at TPAC, QCIF, iVEC, eRSA and Intersect

Also indirectly for the ARCS OPeNDAP? Digital Library

Staff Responsible

Pauline Mal, Kai Lu, Florian Goessmann, Shunde Zhang, Sridhar Reddapani

Detailed Instructions

Stop TDS: 

/etc/init.d/tds stop

Download and expand Jetty 6.1.18: 

wget http://dist.codehaus.org/jetty/jetty-6.1.18/jetty-6.1.18.zip
unzip jetty-6.1.18.zip -d /opt/jetty-6.1.18

As root, make symbolic link to the latest version in /opt and set that as your $JETTY_HOME: 

ln -s /opt/jetty-6.1.18 /opt/tds
export JETTY_HOME=/opt/tds
usermod -d /opt/tds jetty

Copy across your previous installation: 

cd /opt/jetty-6.1.15/webapps
cp -r thredds/ /opt/jetty-6.1.18/webapps/thredds/
cd /opt/jetty-6.1.15/content
mkdir /opt/jetty-6.1.18/content/
cp -r thredds/ /opt/jetty-6.1.18/content/thredds/

Change ownership of files back to the jetty user: 

chown -R jetty:jetty /opt/jetty-6.1.18/

Create a new environment variable file in /etc/default/tds with the following content symbolic link: 

JETTY_HOME=/opt/tds
JAVA_HOME=/usr/java/jdk1.6.0_13/jre
JAVA_OPTIONS="-server -Xmx1536m -Xms512m"
JETTY_PID=/var/run/tds.pid
JETTY_USER="jetty"

Backup and copy the start/stop script to /etc/init.d 

mv /etc/init.d/tds ~/
cp /opt/jetty-6.1.18/bin/jetty.sh /etc/init.d/tds
chmod u+x /etc/init.d/tds

Modify the start/stop script to point to the new environment file. 

##################################################
# See if there's a default configuration file
##################################################
if [ -f /etc/default/jetty6 ] ; then
  . /etc/default/jetty6
elif [ -f /etc/default/tds ] ; then
  . /etc/default/tds
fi

Make sure these variables are picked up when running the script. Running 

/sbin/service/tds check
, you should get something like the following: 

JETTY_HOME     =  /opt/tds
JETTY_CONF     =  
JETTY_RUN      =  /var/run
JETTY_PID      =  /var/run/tds.pid
JETTY_PORT     =  
JETTY_LOGS     =  
CONFIGS        =  /opt/tds/etc/jetty-logging.xml /opt/tds/etc/jetty.xml
JAVA_OPTIONS   =  -server -Xmx1536m -Xms512m -Djetty.home=/opt/tds -Djava.io.tmpdir=/tmp
JAVA           =  /usr/java/jdk1.6.0_13/jre/bin/java
CLASSPATH      =  
RUN_CMD        =  /usr/java/jdk1.6.0_13/jre/bin/java -server -Xmx1536m -Xms512m -Djetty.home=/opt/tds -Djava.io.tmpdir=/tmp -jar /opt/tds/start.jar  /opt/tds/etc/jetty-logging.xml /opt/tds/etc/jetty.xml

Modify /opt/jetty-6.1.18/etc/jetty.xml to make sure you only have one"SelectChannelConnector" the server to only listen to localhost on 8008 like so: 

    <Call name="addConnector">
      <Arg>
          <New class="org.mortbay.jetty.nio.SelectChannelConnector">
            <Set name="host">localhost</Set>
            <Set name="port">8008</Set>
            <Set name="maxIdleTime">30000</Set>
            <Set name="Acceptors">2</Set>
            <Set name="statsOn">false</Set>
            <Set name="confidentialPort">8443</Set>
            <Set name="lowResourcesConnections">5000</Set>
            <Set name="lowResourcesMaxIdleTime">5000</Set>
          </New>
      </Arg>
    </Call>

All sites apart from iVEC will not have Apache installed, so install it using yum 

yum install httpd

Modify the Apache config for ajp:

Now configure the THREDDS server to connect to Apache. Create the file /etc/httpd/conf.d/tds.conf and add the following content: 

<VirtualHost *:80>
  ServerName opendap-your.host.name.arcs.org.au
  DocumentRoot "/var/www/html"

  RewriteEngine On
  RewriteRule ^/$ /thredds/ [R]
  ProxyPass /thredds/ http://localhost:8008/thredds/
</VirtualHost>

Now to run the server, start Jetty first, the Apache: 

/sbin/service tds start
/sbin/service httpd start

Testing Procedures

This set of instruction has been tested on irodsdev.vpac.org.

When you start the server, you should get the following message: 

Redirecting stderr/stdout to /opt/jetty-6.1.18/logs/2009_05_27.stderrout.log

(note that the log is in the jetty-6.1.18 directory). Also make sure that it is running as jetty by running ps. You should get something like this: 

jetty    24612 67.2 16.7 1738880 87936 ?       Sl   13:38   0:02 /usr/java/jdk1.6.0_13/jre/bin/java -server -Xmx1536m -Xms512m -Djetty.home=/opt/tds -Djava.io.tmpdir=/tmp -jar /opt/tds/start.jar /opt/tds/etc/jetty-logging.xml /opt/tds/etc/jetty.xml

Browse to: http://opendap-your.host.name.arcs.org.au/thredds/catalog.html

And if you browse to 

http://opendap-your.host.name.arcs.org.au/
it should redirect you to http://opendap-your.host.name.arcs.org.au/thredds/catalog.html

You should still have the same catalog as the previous installation.

IF AND ONLY IF you're satisfied with the new installation, remove the old /opt/jetty-6.1.15 directory and ~/tds.

Back-out Procedures

To back out, stop TDS if is is not running, as root,

Stop Apache and Jetty (if both are running) 

/sbin/service httpd stop
/sbin/service tds stop


rm /etc/init.d/tds

And remove jetty 6.1.18 

rm /opt/tds
rm -rf /opt/jetty-6.1.18/

Use the old startup script 

mv ~/tds /etc/init.d/tds

And restart the old server: 

/etc/init.d/tds start

Review

Schedule/Notification

Tracking completion

TPAC - done. Note that I had to change the Apache config to: 

<VirtualHost *:80>
  ServerName opendap-tpac.arcs.org.au
  DocumentRoot "/var/www/html"

  RewriteEngine On
  ProxyPreserveHost On
  ProxyRequests Off
  ProxyPass /thredds http://localhost:8008/thredds
  RewriteRule ^/$ /thredds/ [R]
</VirtualHost>

QCIF- done.
AC3 - done.
iVEC - done.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-011 | 28 May 2009 - 12:34 | ---+!! Upgrade Davis to 0.7.0

Description

Upgrade current Davis to 0.7.0

Estimated Duration

20 mins

Systems/Services Affected

All

Staff Responsible

Shunde Zhang

Detailed Instructions

Download 0.7.0 tarball from google code

extract it somewhere

replace all jars in $DAVIS_HOME/webapps/root/WEB-INF/lib with those from the tarball

replace jetty jars (in $DAVIS_HOME/lib) with those from the tarball

update $DAVIS_HOME/webapps/root/WEB-INF/web.xml according to the one in the tarball

restart davis

Testing Procedures

You know what to do.

Back-out Procedures

Better to back up old Davis before making the change.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-012 | 08 Aug 2009 - 10:07 | ---+!! Libtar and GSI Patches for iRODS

Description

Install iRODS-specific version of libtar on all ARCS-DF Production servers. libtar will enable them to process ibun requests from clients. The iRODS-specific version is used to that files larger than 2Gb can be handled.

Also install a GSI patch required for Davis 0.7.

Estimated Duration

60 minutes on each server

Systems/Services Affected

  • Phase 1: arcs-df.vpac.org .. no service outage.
  • Phase 2: arcs-df.ivec.org .. 60 minutes service outage.
  • Phase 3: Other Production servers.

Staff Responsible

Graham Jenkins.

Detailed Instructions

Ref: https://www.irods.org/index.php/Mounted_iRODS_Collection

Copy the attached libtar-1.2.11_64bit.tar.gz file to directory: /var/tmp on your server. Also copy the attached rsGsiAuthRequest.c file to the same directory.

Then proceed as shown hereunder. It's necessary to copy header files into the $LT_HOME/compat directory because the iRODS configuration utilities expect to find them there. 

service irods stop
su - rods
mkdir $IRODS_HOME/../libtar
cd $IRODS_HOME/../libtar
export LT_HOME=`pwd`
mkdir /tmp/build
cd /tmp/build
tar xf /var/tmp/libtar-1.2.11_64bit.tar.gz
cd libtar-1.2.11
./configure --prefix=$LT_HOME
make
make install
mkdir $LT_HOME/compat
cp compat/compat.h $LT_HOME/compat
cp config.h $LT_HOME/compat
cp $LT_HOME/include/* $LT_HOME/compat 
cd $IRODS_HOME

vi config/config.mk config/config.mk.in 
  .. uncomment the line: TAR_STRUCT_FILE=1
  .. set the parameter tarDir to the value of $LT_HOME  (e.g. /opt/rods/libtar)

cp server/api/src/rsGsiAuthRequest.c server/api/src/rsGsiAuthRequest.c.ORI
cp /var/tmp/rsGsiAuthRequest.c server/api/src/
make clean; make
exit
service irods start

Testing Procedures

Proceed as follows: 

su - rods
imkdir Test
iput /etc/group Test
iput /etc/passwd Test
ibun -R arcs-df.ivec.org -cDtar Test.tar Test  .. Use the name of your own resource!
ils -l Test.tar  .. check create-time and length
imkdir Test2
ibun -R arcs-df.ivec.org -x Test.tar Test2       .. Use the name of your own resource!
ils -l Test2 .. check files
irm -rf Test Test2 Test.tar

Back-out Procedures 

service irods stop
su - rods
cd $IRODS_HOME

vi config/config.mk config/config.mk.in 
  .. comment the line: TAR_STRUCT_FILE=1

cp server/api/src/rsGsiAuthRequest.c.ORI server/api/src/rsGsiAuthRequest.c
make clean; make
exit
service irods start

Review

Phase 1 installation performed on arcs-df.vpac.org 20090529; tarfile generated by 'ibun'command contains mutilated filenames. This installation has been left in place pending resolution.

Phase 2 installation performed on arcs-df.ivac.org 20090529 .. rsGsiAuthRequest.c patch Only. Generic libtar installation installed during initial build has been left in place pending satisfactory completion of Phase 1.

Phase 3 has not been done yet.

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200905-013 | 28 May 2009 - 15:34 | ---+!! Upgrade of ng2.sf.utas.edu.au

Description

Migrate the existing ng2.sf.utas.edu.au to CentOS 5.3 and VDT 1.10.1v.

Estimated Duration

1 hour

Systems/Services Affected

Grid services at TPAC.

Staff Responsible

LeighGordon

Detailed Instructions

The existing ng2.sf.utas.edu.au will be shutdown. The new installation will be renamed and tested.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated hour, then this installation will be turned off, and the previous production installation will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-001 | 02 Jun 2009 - 10:50 | ---+!! Updating user information stored in usage statistics data base

Description

An additional script needs to be installed in order to fetch the sharedToken of each user from the Table r_user_main of ICAT (Postgres) and write them to Table users of usage statistics DB – irodsUsage (MySQL)

This will make it possible to get usage statistics of individual user to be displayed while logging into services.arcs.org.au.

Estimated Duration

15 Mins

Systems/Services Affected

Data Fabric machine in iVEC, data base server in db2.arcs.org.au

Staff Responsible

KaiLu, Gareth Williams

Detailed Instructions

A backup of usage statistics DB - irodsUsage at the machine db-2.arcs.org.au is required in advance

As DB user irodsUsage, connecting to DB - irodsUsage located at db-2.arcs.org.au:

run 

   ALTER TABLE users ADD COLUMN sharedToken VARCHAR(255) DEFAULT '' AFTER username;

As user rods at the machine arcs-df.ivec.org:

run 

   cd /opt/rods/iRODS/server/bin/usageScripts/DBScripts
   wget http://projects.arcs.org.au/trac/systems/browser/trunk/dataFabricScripts/iRODS/utils/dbOps.py?format=raw -O dbOps.py
   chmod +x dbOps.py
   Modify the data bases connection information in the script

Run 

   crontab -e

and add line 

   45 1 * * * python /opt/rods/iRODS/server/bin/usageScripts/DBScripts/dbOps.py >>  /opt/rods/iRODS/server/log/useLog-DataFabric 2>&1

The cron job will be run at 1:45 AM every day

Testing Procedures

To do a quick test to see if it's working, run: 

python /opt/rods/iRODS/server/bin/usageScripts/DBScripts/dbOps.py >>  /opt/rods/iRODS/server/log/useLog-DataFabric 2>&1

As DB user irodsUsage, connecting to DB - irodsUsage located at db-2.arcs.org.au:

Run 

   select username, sharedToken from users;

They should be the same as shown in the Table r_user_main of ICAT

Back-out Procedures

As user rods at the machine arcs-df.ivec.org:

Run 

   crontab -e

and remove line 

   45 1 * * * /opt/rods/iRODS/server/bin/usageScripts/DBScripts/dbOps >>  /opt/rods/iRODS/server/log/useLog-DataFabric 2>&1

remove file 

   rm -rf /opt/rods/iRODS/server/bin/usageScripts/DBScripts/dbOps.py

As DB user irodsUsage, connecting to DB - irodsUsage located at db-2.arcs.org.au:

run 

   ALTER TABLE users DROP COLUMN sharedToken;

or

Make a recovery from the latest backup of usage statistics DB - irodsUsage at the machine db-2.arcs.org.au

Review

Schedule/Notification

Tracking completion

Done. -- KaiLu - 01 Jun 2009

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-002 | 03 Jun 2009 - 08:21 | ---+!! Upgrade and Migration of ARCS Sakai

Description

Upgrade from Sakai 2.4 to 2.5, migration to new hardware, retirement of existing virtual machine.

Resolves multiple problems with existing setup:

  • Existing setup uses private IP, problematic because:
    • Cannot use mail-archive tool.
    • Automatic monitoring (eg. NAGIOS) not practical.
  • OpenVZ? less flexible and tied to host configuration.
  • Content is currently proxied by an unreliable and improperly supported intermediate host.
  • Existing host has limited storage options.
  • New hardware:
    • is more powerful;
    • has more resources dedicated to the Sakai VM;
    • has remote management capabilities;
    • has redundant power supplies;
    • is located in air-conditioned facilities;
    • offers far more (and much faster) storage; and
    • has many other benefits.

Proposed Date

Wednesday, 10th June 2009, 10am-2pm AEST.

Estimated Duration

Four hours

Systems/Services Affected

All sites hosted on the ARCS Sakai will be completely unavailable during the period of the outage.

Level of Impact

3D

  • 3: downtime more than 2 hours but less than 24 hours
  • D: multiple services at multiple sites OR significant loss of functionality affecting more than 10 people

Staff Responsible

  • Denny (via ARCS email, Jabber)- contact for Sakai-level issues.
  • Garth Denley (via ARCS email, Jabber)- contact for system-level issues.
  • Andy Botting- for DNS record change only.

Detailed Instructions

  • Update DNS records (A and MX) for the new Sakai host. (Andy)
  • Disable access to both the old and new Sakai instances.
    • On both hosts: service shutdown tomcat
  • Firewall off new ARCS Sakai to allow selected access only.
    • will be done using iptables rules
    • connecting users will have packet rejected.
    • if practical, may be redirected to a downtime page- although this may prove difficult.

  • On dhpc05 (old Sakai): 

    TODAY=`date +%F`
    mysqldump -h localhost -u sakai -p sakai > sakai-$TODAY.sql
    scp sakai-$TODAY.sql denny@sakai-new.arcs.org.au:~/storage
    cd /home/tomcat/sakai
    tar czf ~/sakai-files.tar.gz db
    scp sakai-files.tar.gz denny@sakai-new.arcs.org.au:~/storage

  • On sakai-new (new Sakai):

    • Run content conversion tool to fix up the site descriptions 

    ./fixd < sakai-xxxx.sql > sakai-xxxx-fixed.sql

    • Both the tar.gz and sql files on both the old and new hosts should be checked to ensure nothing went wrong in the transfer: 
    md5sum sakai-xxx.sql
    md5sum sakai-files.tar.gz

    • The MD5 sum reported should be the same on each host.

    • Create database for Sakai and import the database from the dump file, and also extract the binary content as well 

    mysql -u root -p
    create database sakai_prod default character set utf8;
    grant all on sakai_prod.* to sakai_prod@localhost identified by 'xxxx';
    grant all on sakai_prod.* to sakai_prod@127.0.0.1 identified by 'xxxx';

    mysql -h localhost -u sakai_prod -p sakai_prod < sakai-xxxxx
    mkdir -p /storage/sakai/db
    cd /storage/sakai
    tar xzf /storage/denny/sakai-files.tar.gz

    • Run conversion script to update the database structure 

    cd /home/denny/sakai-2-5-x/reference/docs/conversion
    mysql -u sakai_prod -p sakai_prod
    source sakai_2_4_0-2_5_0_mysql_conversion.sql
    source sakai_2_5_0-2_5_2_mysql_conversion.sql

    • Update sakai.properties (if needed) 
    bodyPath
    bodyVolumes
    username@javax.sql.BaseDataSource=sakai_prod
    password@javax.sql.BaseDataSource=xxxx
    url@javax.sql.BaseDataSource=

    • Update hostname (sakai-new -> sakai) 
    system-config-network
    • Other references to hostname 
    find . -type f -exec grep sakai /dev/null {} \;
    • and clean up anything found.
  • Reboot new server to ensure new hostname is being used properly, all services come up automatically, etc.
  • Test. Additional volunteers have been sought. This step will be performed in the last hour of the four hour period.
  • Remove firewall rules to allow world-wide access.

Testing Procedures

  • Login with accounts created on old Sakai server
  • Check/compare list of usernames on both server
  • Test Reset Password feature
  • Compare contents of the existing Sakai server and the new server visually
    • Check site descriptions in Home tool
    • Check content of Resources tool
    • Check announcement contents
    • Check other tools contents, e.g. Site Stats
  • Upload file & create folder using Resources tool with various size: small, slightly less than 20 MB, above 20 MB (should be rejected, because the limit is 20 MB)
  • Upload file & create folder using WebDAV? with different Operating System
  • Test Email Archive tool: sending a mail to @sakai.arcs.org.au, with various email format

Back-out Procedures

As the original data remains untouched on the old server, backing out is not a particular onerous process:

  • DNS records returned to point at old VM.
  • New server reconfigured to use temporary name (sakai-new) so as to not interfere with old system.
  • Old system brought back up.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-003 | 15 Jun 2009 - 11:15 | ---+!! Update and migration of ARCS projects.arcs.org.au

Description

ARCS is migrating all hosted sites that use Trac and Subversion to    
Trac 0.11.4
Subversion 1.4.2
OS (Xen Guest) 32-bit CentOS 5.3
OS (Xen Host) 64-bit CentOS 5.3
The reason for this change is to    
* Transfer from AC3 to Intersect owned host machines  
* Increase storage space for Subversion repositories  
* Partition the administrative interface to allow greater autonomy for project leaders  

Estimated Duration

2 hours

Systems/Services Affected

http://projects.arcs.org.au https://projects.arcs.org.au

This covers all Trac and Subversion sites, as well as the vdt mirror and ARCS/MARCS yum repositories

Staff Responsible

Simon Yin - simon.yin@arcs.org.au, Tel: 0410609094

Detailed Instructions

Migration of the ARCS Trac/Subversion site is summarised by the following steps:

  1. On the old system, dump the PostgreSQL? database to a backup file
  2. On the old system, dump the Trac wiki/attachments directory to a backup directory
  3. On the new system, initialise the new Trac and Subversion environment
  4. Restore the PostgreSQL? database and wiki/attachments directory to the new system
  5. Synchronise a new Subversion repository using the old system as the source.

The mirror and repositories are tranferred with rsync. Note the root of the server filesystem on the new server is /var/www. This is changed from the old system where the root of the server filesystem was /repository

Server certificates for Apache (from /etc/pki/tls/certs/ and /etc/pki/tls/private/ ) and for Globus (from /etc/grid-security/ ) are also transferred to the new server.

Links to detailed instructions for each individual site: https://projects.arcs.org.au/trac/systems/wiki/HowTo/UpgradeAndMigrateTracSubversion

Testing Procedures

An Internet accessible IP address, 203.202.139.98, is configured on the new system for system testing.

The following scenarios are tested for each Trac/Subversion site:

  1. Windows XP, IE8 user
    1. Connect anonymously to http://203.202.139.98/trac/project-under-test
      1. View tickets
    2. Register as "TestUser"
    3. (https only) Connect as "TestUser" with no extra permissions to https://203.202.139.98/trac/project-under-test
      1. Create a new ticket
    4. (https only) Connect as user with full admin permissions to https://203.202.139.98/trac/project-under-test
      1. Change the password of "TestUser" (and then login again as "TestUser" to check the password)
    5. Connect to http://203.202.139.98/svn/project-under-test
  2. Repeat the tests in 1. for a Mac OSX, Safari user
  3. Repeat the tests in 1. for a Linux, Firefox 3.0.x user
  4. (https only) Linux, Subversion from command line (if necessary, check out from a known sub-directory of the project-under-test to save time)
    1. svn co https://203.202.139.98/svn/project-under-test

Where tests listed above have used the http protocol, repeat them with the https protocol.

For the mirror and yum repositories, simply confirm that a web browser can browse http://203.202.139.98/mirror/vdt/ and http://203.202.139.98/dist/

Once all tests are passed, the IP address of projects.arcs.org.au (203.202.30.92) will replace the test IP address and the system deemed as accepted into production.

All references in Trac configuration files to srb-dev.ac3.edu.au (the temporary URL) are then replaced with projects.arcs.org.au

Back-out Procedures

The old server and host machine are unaffected by these changes - moving the service IP 203.202.30.92 back to the old system will revert to the previous system configuration.

Review

Schedule/Notification

The following notice will be posted on the Trac front page of each project:

Scheduled downtime for this site: Monday, 22 June 2009 at 5:00pm AEST
Service will be restored on Monday, 22 June 2009 at 7:00pm AEST

Notification by email:

For ARCS internal projects:  
  techstaff@arcs.org.au, developers@arcs.org.au
For Fullmoon  
  simon.cox@csiro.au, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For common-grid-libs  
  aron.abrook@qut.edu.au
For commons-vfs-grid  
  manish.saroha@its.monash.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For grisu  
  y.halytskyy@auckland.ac.nz
For grix  
  aron.abrook@qut.edu.au
For gsub  
  sam.moskwa@sapac.edu.au
For jmoleditor  
  Joachim.Mai@ac3.edu.au, vvv900@gmail.com, Martin.Thompson@unsw.edu.au
For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For access  
  martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For pht3d  
  Henning.Prommer@csiro.au, Aaron.McDonough@csiro.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org
For sggc  
  Terry.Rankine@csiro.au
For slcs-client  
  chi.nguyen@melco.mq.edu.au

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-004 | 17 Jun 2009 - 16:50 | ---+!! Upgrade Davis

Description

Upgrade Davis

Estimated Duration

1 mins

Systems/Services Affected

Data Fabric

Staff Responsible

Shunde Zhang, Florian Goessmann

Detailed Instructions

  • Download davis tar ball from google code
  • Extract it to /opt/davis
  • Modify web.xml
  • Stop davis service
  • Change soft link from last version to current version
  • Start Davis

Testing Procedures

  • Try litmus.
  • Try WebDAV? clients.

Back-out Procedures

Stop service, change soft link back to the last working version. Restart service.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-005 | 19 Jun 2009 - 09:15 | ---+!! iRODS 'gethostbyname' Problem Resolution

Description

It has been found that all iRODS Production servers can, under heavy network traffic conditions, log a message of the following form:

WARNING: gethostbyname of df.arcs.org.au is taking 2 sec. This could severely affect interactivity of your Rods system

To overcome this, the name of partner hosts will be added to the /etc/hosts file in each server.

Estimated Duration

This Change will take 30 minutes, including testing and backout steps.

Systems/Services Affected

The /etc/hosts file will be updated on each of the following systems:
arcs-df.ivec.org
arcs-df.vpac.org
arcs-df.ac3.edu.au
arcs-df.hpcu.uq.edu.au

Staff Responsible

Graham Jenkins

Detailed Instructions

Save the attached hostbyname test script in /tmp on each machine, and make it executable. Then do: 

time /tmp/hostbyname arcs-df.ivec.org arcs-df.vpac.org  arcs-df.ac3.edu.au arcs-df.hpcu.uq.edu.au
and note the 'real' time.

Then add the following lines to /etc/hosts on each of the above systems, commenting the entry for the system itself: 

# Entries added 20090619 to improve 'gethostbyname' performance
#192.65.130.247 arcs-df.ivec.org
202.158.218.58  arcs-df.vpac.org
203.202.30.74   arcs-df.ac3.edu.au
130.102.163.138 arcs-df.hpcu.uq.edu.au

Testing Procedures

Repeat the test described above, ensure that it produces the same output and that the 'real' time is significantly less than before.

Back-out Procedures

Remove the entries added to /etc/hosts and repeat the test, ensuring that it produces the same results.

Review

Schedule/Notification

It is proposed that this Change should be performed at 9am Friday June 19.

Tracking completion

Test results ("before" and "after" real times) were as follows: 

arcs-df.ivec.org:        0.84, 0.09
arcs-df.vpac.org:        0.77, 0.07
arcs-df.ac3.edu.au:      1.54, 0.10
arcs-df.hpcu.uq.edu.au:  1.06, 0.08

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-006 | 26 Jun 2009 - 10:54 | ---+!! Upgrade Davis 0.7.2a

Description

Upgrade Davis

Estimated Duration

1 mins

Systems/Services Affected

Data Fabric

Staff Responsible

Shunde Zhang, Florian Goessmann

Detailed Instructions

  • Download davis tar ball from google code
  • Extract it to /opt/davis
  • Modify web.xml
  • Stop davis service
  • Change soft link from last version to current version(0.7.2a)
  • Start Davis

Testing Procedures

  • Try litmus.
  • Try WebDAV? clients.

Back-out Procedures

Stop service, change soft link back to the last working version. Restart service.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-007 | 23 Jun 2009 - 23:11 | ---+!! Upgrade of nggums.sf.utas.edu.au

Description

Migrate the existing nggums.sf.utas.edu.au to CentOS? 5.3 and VDT 1.10.1y.

Estimated Duration

1 hour.

Systems/Services Affected

Grid services at TPAC.

Staff Responsible

LeighGordon

Detailed Instructions

The existing nggums.sf.utas.edu.au will be shutdown. The new installation will be renamed and tested.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite? located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated hour, then this installation will be turned off, and the previous production installation will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-008 | 08 Aug 2009 - 10:10 | ---+!! Adding rules for IMOS collections

Description

Files under IMOS collections will need to be stored at different resources.

Estimated Duration

1 hour

Systems/Services Affected

Data Fabric on all production boxes:

arcs-df.hpcu.uq.edu.au, arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.eresearchsa.edu.au, arcs-df.vpac.org

Staff Responsible

Pauline Mak

Detailed Instructions

Modify core.dvm to map $objPath to the appropriate data structure in the acSetRescSchemeForCreate rule.

Note that: $IRODS_HOME = /opt/iRODS-2.0v/iRODS

Open up 

$IRODS_HOME/server/config/reConfigs/core.dvm

Insert in line 17: 

objPath||rei->doinp->objPath

Resulting in line 16-18 of: 

objPath||rei->doi->objPath
objPath||rei->doinp->objPath
rescName||rei->doi->rescName

Download the imos rules file 

cd $IRODS_HOME/server/config/reConfigs
wget http://projects.arcs.org.au/trac/systems/export/993/trunk/dataFabricScripts/iRODS/Rules/imos.irb -O imos.irb

Now add the IMOS rules into the server config $IRODS_HOME/server/config/server.config 

reRuleSet   imos,arcs,core

Testing Procedures

Login as the rods user, and modify the ~/.irods/.irodsEnv file and comment out the default resource setting.

Put a file in the following directory on the data fabric. Create a dummy text file, then 

icd /ARCS/projects/IMOS/archive/eMII/
iput testForIMOSRules
ils -L

The result should say the file is now on emii.resource.tpac.org.au.

Remove the file using irm and uncomment the default resource setting in ~/.irods/.irodsEnv.

Repeat for all hosts.

Back-out Procedures

Remove imos from the server configuration file: $IRODS_HOME/server/config/server.config 

reRuleSet   arcs,core

Remove the imos rules file: 

rm $IRODS_HOME/server/config/reConfigs/imos.irb

Revert $IRODS_HOME/server/config/reConfigs/core.dvm to previous state by removing line 17. Resulting in the following line 16-18: 

objPath||rei->doi->objPath
rescName||rei->doi->rescName
destRescName||rei->doi->destRescName

Review

Schedule/Notification

Enable IMOS rules after the outage on 27th June.

Tracking completion

iVEC - applied on 29th June TPAC - applied on 3rd July VPAC - applied on 3rd July UQ - applied on 3rd July eRSA - applied on 3rd July AC3 - applied on 3rd July

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-009 | 01 Jul 2009 - 16:46 | ---+!! Upgrade of grid virtual machines at iVEC

Description

Migrate the existing ng2, ngdata, and nggums virtual machines to CentOS 5.3 and VDT 1.10.1 on new hardware.

Estimated Duration

2 hours.

Systems/Services Affected

All ARCS grid services in WA.

Staff Responsible

DarranCarey

Detailed Instructions

Shutdown the existing ng2, ngdata and nggums virtual machines. Rename the new ng2, ngdata and nggums installations and test the grid workflow.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-010 | 18 Sep 2009 - 10:17 | ---+!! Deploy TDS 4.0 stable Deploy TDS 4.0 Stable and modify file system permission

Description

TDS 4.0 stable will be replacing the current alpha version. Additionally, TDS will now be access data managed by the DF. Underlying file permission has to be changed to accommodate this.

Estimated Duration

2 hours

Systems/Services Affected

ARCS OPeNDAP? Network

arcs-df.ivec.org, arcs-df.sf.utas.edu.au, arcs-df.eresearchsa.edu.au, arcs-df.intersect.org, arcs-df.hpcu.uq.edu.au

Staff Responsible

Pauline Mak, Shunde Zhang

Detailed Instructions

Stop the TDS server. 

/sbin/service tds stop

Disable robots

To disable robots from touching all files, create a file in /var/www/html/robots.txt and put in the following text" 

User-agent: *
Disallow: /

Modify Permission

As the rods user, create the directory structure on your local resource for the OPeNDAP? directory /ARCS/projects/IMOS/opendap/ e.g. 

icd /ARCS/projects/IMOS/opendap/
iput -R irodsdev.vpac.org test.txt
irm test.txt

Then, as rods user, get the script from subversion: 

cd $IRODS_HOME/server/bin/local
wget http://projects.arcs.org.au/trac/systems/export/1095/trunk/dataFabricScripts/iRODS/utils/opendap_acl.sh
chmod 700 opendap_acl.sh

You will need to replace the value of vaultDir and user.

To your vault, use 

iadmin lr <local resource name>

Then find out the name of the TDS user by looking at /etc/default/tds and look for "JETTY_USER".

For example, on irodsdev.vpac.org, the values would be: 

vaultDir=/data/Vault
user=jetty

Run the script: 

$IRODS_HOME/server/bin/local/opendap_acl.sh

To test that this is working, become the jetty user (or whatever this user is set to on your system). You should be able to cd into the opendap directory, however, you should not be able to list or view any of the directories above opendap. Check that the ACL is set, you should get something like the following using getfacl: 

-bash-3.2$ getfacl --access /data/Vault/ARCS/projects/IMOS/opendap
getfacl: Removing leading '/' from absolute path names
# file: data/Vault/ARCS/projects/IMOS/opendap
# owner: rods
# group: rods
user::rwx
user:jetty:r-x
group::---
mask::r-x
other::---

As the rods user, modify the cronjob such that the script is executed hourly: 

0 * * * * /opt/iRODS-2.1v/iRODS/server/bin/local/opendap_acl.sh >/dev/null 2>&1 || :

Installing new TDS

Download the latest version of TDS. 

wget ftp://ftp.unidata.ucar.edu/pub/thredds/4.0/thredds.war

Then, as root: 

cp -r /opt/tds/webapps/thredds ~/thredds_old
rm -rf /opt/tds/webapps/thredds/
unzip thredds.war -d /opt/tds/webapps/thredds/
chown jetty:jetty -R /opt/tds/

As the jetty user, configure your TDS server to point to the new OPeNDAP? directory by modifying catalog.xml. This file is in 

/opt/tds/content/thredds

If you are using the default catalog, then remove all XML tags between <catalog> and </catalog> and replace it with the following: 

<catalogRef name="IMOS" xlink:title="Integrated Marine Observation System" xlink:href="imos.xml" />

Otherwise, sure the above tag is somewhere in between the catalog element.

Modify the name attribute in the element to read: 

<your institution's name>/ARCS OPeNDAP server

Save and close the file.

Create a new file in the same directory (/opt/tds/content/thredds) with the name imos.xml

And put in the following 

 <-- Regularly Gridded Datasets -->
  <service name="regGriddedServices" base="" serviceType="compound">
    <service name="dapService" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="httpService" serviceType="HTTPServer" base="/thredds/fileServer/" /> <!-- direct file download -->
    <service name="wcsService" serviceType="WCS" base="/thredds/wcs/" /> <!-- OGC Web Coverage Service -->
    <service name="wmsService" serviceType="WMS" base="/thredds/wms/" /> <!-- OGC Web Map Service -->
    <service name="ncss" serviceType="NetcdfSubset (Experimental)" base="/thredds/ncss/grid/" /> <!-- NetCDF Subset service -->
   </service>

   <!-- Irregularly Gridded Dataset -->
   <service name="irregGriddedServices" base="" serviceType="compound">    <service name="dapService" serviceType="OpenDAP" base="/thredds/dodsC/" />    <service name="httpService" serviceType="HTTPServer" base="/thredds/fileServer/" /> <!-- direct file download -->
    <service name="wmsService" serviceType="WMS" base="/thredds/wms/" /> <!-- OGC Web Map Service -->
    <service name="ncss" serviceType="NetcdfSubset (Experimental)" base="/thredds/ncss/grid/" /> <!-- NetCDF Subset service -->

   <!-- Station Datasets -->
   <service name="station" base="" serviceType="compound">
    <service name="dapService" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="httpService" serviceType="Driect Download" base="/thredds/fileServer/" /> <!-- direct file download -->
   </service>

   <!-- Trajectory Datasets -->
   <service name="trajectory" base="" serviceType="compound">
    <service name="dapService" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="httpService" serviceType="HTTPServer" base="/thredds/fileServer/" /> <!-- direct file download -->
   </service>


    <!-- Files that are not served by OPeNDAP, just plain HTTP.  E.g. Matlab Scripts -->
    <service name="httpOnly" serviceType="HTTPServer" base="/thredds/fileServer/" />

   <!-- Now configure the dataset directory - note that IMOS may request this to be changed from time to time to include extra metadata about datasets -->
          When datasets are moved to the OPeNDAP directory, you may want to ask your eMII/IMOS contact whether this dataset belongs to any of
          the above service categories (regGriddedServices, etc) and configure the "serviceName" value accordingly .  -->

        <dataset name="IMOS - VPAC">
                <datasetScan name="IMOS OPeNDAP directory" path="IMOS" location="/data/Vault/ARCS/projects/IMOS/opendap/" serviceName="basic">
                    <filter>
                        <include wildcard="*.nc" />
                        <include wildcard="*.hdf" />
                    </filter>
                </datasetScan>
        </dataset>

Modify the THREDDS server to run with the following options (in /etc/default/tds) 

JAVA_OPTIONS="-server -Xmx1536m -Xms512m -Djava.awt.headless=true"

Then start the server 

/sbin/service tds start

Testing Procedures

Browse to your TDS installation:

http://hostname/thredds

The footer should be updated with the latest version name: "THREDDS Data Server [Version 4.0.26 - 20090831.2140] Documentation"

TDS 4.0 is now installed!

Back-out Procedures

To reinstate the old version: 

service tds stop
rm  /opt/jetty/webapps/thredds.war
rm -rf /opt/jetty/webapps/thredds
mv ~/thredds_old /opt/jetty/webapps/thredds

Review

Schedule/Notification

TPAC - Done

QCIF - Done

eRSA - Done

Intersect - Done[Sridhar Reddapani]

iVEC - Done

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200906-011 | 08 Aug 2009 - 10:09 | ---+!! Rules Update Script for iRODS

Description

The updateRules.sh script executes hourly on each ARCS Production and Development iRODS server to download whatever version of the relevant arcs.irb (Rules) file is currently in the SVN repository. The file is downloaded with a temporary name, and edited so as insert the local default resource before being moved into place.

An imos.irb Rules file has now been manually deployed to Production and Development servers, and there is a requirement for versions of this to be automatically updated on all such servers when it changes.

The updateRules.sh script has therefore been extended so as to accommodate this requirement.

It is anticipated that the script will need to handle additional Rules files in the future, so the extended version has the capability of updating itself from the SVN repository.

This Change covers the deployment of the extended script to all Development (Phase 1) and Production (Phase 2) servers.

The notification address shown in the createUser.config file on each server will also be updated (to arcs-data@arcs.org.au) during each Phase. This will ensure that all iRODS server administrators are notified when new user identities are created.

Estimated Duration

The maximum duration for each Phase (including testing and backout steps) is one hour. There should be no interruption to service as seen by users.

Systems/Services Affected

Phase 1: irodsdev.vpac.org, data-dev.eresearchsa.edu.au, ngdata-dev.hpcu.uq.edu.au
Phase 2: arcs-df.vpac.org, arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.hpcu.uq.edu.au, arcs-df.tpac.org.au, arcs-df.eresearchsa.edu.au

Staff Responsible

Graham Jenkins

Detailed Instructions

On each machine listed, do crontab -e and comment the updateRules.sh line; take note of the machine's default resource-name as shown on that line. Then perform the following steps (as the 'rods' user): 

cd $IRODS_HOME/server/bin/local
mv updateRules.sh /var/tmp
wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/Rules/updateRules.sh
chmod a+xr updateRules.sh 

cp $IRODS_HOME/server/config/createUser.config /var/tmp
vi $IRODS_HOME/server/config/createUser.config
# Change address in the line which starts in 'A' to:  arcs-data@arcs.org.au

Also (Phase 1 only) .. update http://projects.gridaus.org.au/trac/systems/wiki/DataServices/iRODS_Server to reflect procedure in this Change.

Testing Procedures

On each machine listed, perform the following steps (as the 'rods' user): 

cp updateRules.sh /tmp
echo "##" >> updateRules.sh
$IRODS_HOME/server/bin/local/updateRules.sh irodsdev.vpac.org  # <= Substitute Machine's Default Resource-Name
cmp /tmp/updateRules.sh updateRules.sh  # Should be the same

cd $IRODS_HOME/server/config/reConfigs
cp *.irb /var/tmp
echo "##">>arcs.irb
echo "##">>imos.irb
$IRODS_HOME/server/bin/local/updateRules.sh irodsdev.vpac.org # <= Substitute Machine's Default Resource-Name
cmp /var/tmp/arcs.irb arcs.irb  # Should be the same
cmp /var/tmp/imos.irb imos.irb  # Should be the same

If the tests are successful, do crontab -e and uncomment the line containing: updateRules.sh

Back-out Procedures

Perform the following steps: 

cd /var/tmp
cp arcs.irb              $IRODS_HOME/server/config/reConfigs/
cp imos.irb              $IRODS_HOME/server/config/reConfigs/
cp updateRules.sh $IRODS_HOME/server/bin/local

Then do crontab -e and uncomment the line containing: updateRules.sh

Review

Schedule/Notification

Phase 1: 1:30 pm Wednesday July 1.
Phase 2: 9:00 am Friday July 3.

Tracking completion

Phase 1 completed: 2:15 pm Wednesday July 1.
Phase 2 completed: 10:05 am Friday July 3.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-001 | 09 Jul 2009 - 12:59 | ---+!! TITLE

Description

Moving Master ICAT from arcs-df.ivec.org to arcs-df.vpac.org and moving ICAT to Pgpool

Estimated Duration

8 Hours

Systems/Services Affected

Data Fabric

Staff Responsible

Sridhar Reddapani, Pauline Mak

Detailed Instructions

  • 1. Install Pgpool on arcs-df.vpac.org under /opt/PgPool
    • Download latest Pgpool from http://pgfoundry.org/projects/pgpool/ to /tmp
    • mkdir /opt/PgPool [as root]
    • chown -R rods:rods /opt/PgPool [as root]
    • su - rods
    • cd /opt/PgPool/
    • mkdir pgpool
    • mkdir logs
    • cd /tmp
    • tar -xvf pgpool-II-2.2.tar.gz
    • cd pgpool-II-2.2
    • ./configure --prefix=/opt/PgPool/pgpool
    • make
    • make install

  • 3. Stop iRODS on all servers, stop Davis on arcs-df.ivec.org
    • service davis stop
    • service irods stop
    • service postgres stop [if Running]
    • service davis stop [ on arcs-df.ivec.org]

  • 4. Setup password less ssh access between arcs-df.ivec.org and arcs-df.vpac.org for user rods

  • 5. Take backup of ICAT from arcs-df.ivec.org and copy it to arcs-df.vpac.org, run below commands on arcs-df.ivec.org
    • Copy whole /opt/iRODS-2.0v/Postgres/pgsql/data directory to off site and DVD?
    • service postgres start
    • pg_dump ICAT > ICAT_ivec_DUMP
    • Write ICAT_ivec_DUMP file to DVD and check CheckSum? is OK
    • scp ICAT_ivec_DUMP rods@arcs-df.vpac.org:/tmp/

  • 6. Change local postgres port to 5431 on arcs-df.vpac.org and make change shared_buffers[25% of RAM] and effective_cache_size [50% of RAM] on both postgres servers
    • edit .../Postgres/pgsql/data/postgresql.conf
      • change port = 5431 [ on arcs-df.vpac.org]
      • shared_buffers = 25% of RAM [ on both arcs-df.vpac.org & arcs-df.ivec.org]
      • effective_cache_size = 50% of RAM [on both arcs-df.vpac.org & arcs-df.ivec.org]

  • 7. Start postgres, drop ICAT and restore ICAT from dump from arcs-df.vpac.org
    • service postgres start
    • dropdb -p 5431 ICAT
    • createdb -p 5431 ICAT
    • psql -p 5431 ICAT < /tmp/ICAT_ivec_DUMP

  • 8. Install cronolog on arcs-df.vpac.org

  • 9. Configure Pgpool and add local postgres[5431] on arcs-df.vpac.org and postgres on arcs-df.ivec.org to Pgpool as documented here http://projects.arcs.org.au/trac/systems/wiki/DataServices/Database/PgpoolConfiguration
    • Edit /opt/PgPool/pgpool/etc/pgpool.conf as
      • listen_addresses = '*'
      • port = 5432
      • pcp_port = 9898
      • logdir = '/opt/PgPool/logs'
      • pid_file_name = '/opt/PgPool/logs/pgpool.pid'
      • replication_mode = true
      • load_balance_mode = true
      • insert_lock = false
      • num_init_children = 200
      • max_pool = 1
      • backend_hostname0 = 'arcs-df.vpac.org'
      • backend_port0 = 5431
      • backend_weight0 = 1
      • backend_hostname1 = 'arcs-df.ivec.org'
      • backend_port1 = 5432
      • backend_weight1 = 0
    • Edit /opt/PgPool/pgpool/etc/pool_hba.conf and give access to all irods server
    • Edit /opt/PgPool/pgpool/etc/pcp.conf and add rods md5 password to it.

  • 10. Make sure you have right access to other servers in pg_hba.conf on both servers

  • 11. Start postgres on both servers and then start pgpool -- check logs for any errors

  • 12. Config irods arcs-df.vpac.org to talk to pgpool for Master ICAT and start iRODS
    • edit /opt/iRODS-2.0v/iRODS/server/config/server.config as
      • icatHost arcs-df.vpac.org
    • service irods start

  • 13. Config irods arcs-df.ivec.org to talk to pgpool for Master ICAT and local postgres for Slave ICAT then start iRODS
    • edit /opt/iRODS-2.0v/iRODS/server/config/server.config as
      • icatHost arcs-df.vpac.org
      • slaveIcatHost arcs-df.ivec.org
    • service irods start

  • 14. By now we finished basic configuration of Pgpool, Check everything is working as expected and proceed to configure Pgpool Online Recovery

  • 15. Run these to stop everything
    • service irods stop [ on arcs-df.ivec.org]
    • service irods stop [ on arcs-df.vpac.org]
    • pgpool stop [ on arcs-df.ivec.org]
    • service postgres stop [ on arcs-df.vpac.org]
    • service postgres stop [ on arcs-df.ivec.org]

  • 16. Enable WAL archiving on both servers and start postgres, Test WAL archiving is working..
    • mkdir /opt/iRODS-2.0v/Postgres/pgsql_wal_archive/
    • touch /opt/iRODS-2.0v/Postgres/backup_in_progress
    • edit /opt/iRODS/Postgres/pgsql/data/postgresql.conf as
      • archive_mode = on
      • archive_command = 'test ! -f /opt/iRODS-2.0v/Postgres/backup_in_progress || rsync -a %p /opt/iRODS-2.0v/Postgres/pgsql_wal_archive/%f'
      • archive_timeout = 5
    • service postgres start
    • After 5 minutes check whether you have WAL files in /opt/iRODS-2.0v/Postgres/pgsql_wal_archive/ if so WAL archiving is OK
    • rm /opt/iRODS-2.0v/Postgres/backup_in_progress
    • edit /opt/iRODS/Postgres/pgsql/data/postgresql.conf as
      • archive_timeout = 0

  • 17. Install Pgpool function on arcs-df.vpac.org and copy untared pgpool directory to arcs-df.ivec.org
    • cd /tmp/pgpool-II-2.2/sql/pgpool-recovery/
    • make install
    • psql -p 5431 -f pgpool-recovery.sql template1
    • scp -r /tmp/pgpool-II-2.2/ rods@arcs-df.ivec.org:/tmp/

  • 18. Install Pgpool function on arcs-df.ivec.org
    • cd /tmp/pgpool-II-2.2/sql/pgpool-recovery/
    • make install
    • psql -p 5432 -f pgpool-recovery.sql template1

  • 19. Edit /opt/PgPool/pgpool/etc/pgpool.conf as..
    • health_check_timeout = 90
    • health_check_period = 30
    • failover_command = 'echo WARNING from ARCS DF | mail -s "One of the Pgpool node on arcs-df.vpac.org is DOWN!" arcs-data@arcs.org.au'
    • failback_command = 'echo NOTIFICATION from ARCS DF | mail -s "Pgpool on arcs-df.vpac.org is now HEALTHY" arcs-data@arcs.org.au'
    • backend_data_directory1 = '/opt/iRODS-2.0v/Postgres/pgsql/data/'
    • backend_data_directory2 = '/opt/iRODS-2.0v/Postgres/pgsql/data/'
    • recovery_user = 'rods'
    • recovery_password = 'rods-password'
    • recovery_1st_stage_command = 'copy_base_backup'
    • recovery_2nd_stage_command = 'pgpool_recovery_pitr'

  • 20. Download attached copy_base_backup, pgpool_recovery_pitr and pgpool_remote_start files and make (any required) changes to copy_base_backup and pgpool_remote_start

  • 21. Testing Online Recovery
    • Start postgres and pgpool on arcs-df.vpac.org
      • service postgres start
      • pgpool -d -n 2>&1 | cronolog --hardlink=/opt/PgPool/logs/pgpool.log '/opt/PgPool/logs/%Y-%m-%d-pgpool.log' &
    • watch log file while doing recovery on another window
      • tail -f opt/PgPool/logs/pgpool.log
    • Run recovery command
      • pcp_recovery_node -d 20 localhost 9898 rods <rods-password> 1
    • If recovery is done, postgres will be started on arcs-df-ivec.org, if not TROUBLESHOOT
    • Stop pgpool and postgres
      • pgpool stop
      • service postgres stop [ on both servers]

  • 22. If everything OK by now, CONGRATULATIONS... Start everything as below sequence
    • service postgres start [ on arcs-df.vpac.org]
    • service postgres start [ on arcs-df.ivec.org]
    • pgpool -d -n 2>&1 | cronolog --hardlink=/opt/PgPool/logs/pgpool.log '/opt/PgPool/logs/%Y-%m-%d-pgpool.log' & [ on arcs-df.vpac.org]
    • service irods start [ on arcs-df.vpac.org]
    • service irods start [ on arcs-df.ivec.org]

  • 23. Configure all other DF servers talk to pgpool for Master ICAT and start iRODS
    • Edit /opt/iRODS-2.0v/iRODS/server/config/server.config as
      • icatHost arcs-df.vpac.org
    • service irods start

  • 25. Check all rules are in place on arcs-df.vpac.org.

Testing Procedures

  • 1. Try accessing DF for both Reading and Writing.
  • 2. Check Automatic User creation is functional.
  • 3. Check all rules are functional.

Back-out Procedures

  • 1. Stop Everything in below sequence
    • service irods stop [ on all other irods servers]
    • service irods stop [ on arcs-df.ivec.org]
    • service irods stop [ on arcs-df.vpac.org]
    • pgpool stop [ on arcs-df.vpac.org]
    • service postgres stop [ on both arcs-df.vpac.org & arcs-df.ivec.org]

  • 2. Configure iRODS on arcs-df.ivec.org to talk to local postgres for Master ICAT and start iRODS
    • Edit /opt/iRODS-2.0v/iRODS/server/config/server.config as
      • icatHost arcs-df.ivec.org
    • service postgres start
    • service irods start

  • 3. Configure iRODS on all servers to talk to arcs-df.ivec.org for Master ICAT and start iRODS and start iRODS
    • Edit /opt/iRODS-2.0v/iRODS/server/config/server.config as
      • icatHost arcs-df.ivec.org
    • service irods start

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| ChangeNote200907-002 | 09 Jul 2009 - 09:20 | ---+!! Upgrade of grid virtual machines at CSIRO

Description

Migrate the existing ng2, ngdata, and nggums virtual machines to CentOS 5.3 and VDT 1.10.1 on new hardware

Estimated Duration

2 hours.

Systems/Services Affected

All ARCS grid services provided by CSIRO in Victoria.

Staff Responsible

JoelLudbey? -Bruhwel

Detailed Instructions

Shutdown the existing ng2, ngdata and nggums virtual machines. Rename the new ng2, ngdata and nggums installations and test the grid workflow.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite? located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-003 | 08 Aug 2009 - 10:08 | ---+!! Moving Davis from iVEC to VPAC

Description

This is a follow up step to the pgpool migration to VPAC (see http://wiki.arcs.org.au/bin/view/Main/ChangeNote200907-001). Davis will be installed at VPAC as it contains the master ICAT

Estimated Duration

1 hour

Systems/Services Affected

df.arcs.org.au, arcs-df.vpac.org

Staff Responsible

Pauline Mak, Andy Botting

Detailed Instructions

As root:

Stop Davis, as root: /etc/init.d/davis stop

Remove old version of davis 

mv /opt/davis ~/davis

Download, and install davis 

mkdir /opt/davis

wget http://code.arcs.org.au/hudson/job/davis/ws/trunk/dist/davis-0.7.2d.tar.gz

tar -zxvf davis-0.7.2.tar.gz
mv davis-0.7.2 /opt/davis/davis-0.7.2d
cd /opt/davis/
chown -R davis:davis /opt/davis/davis-0.7.2d
ln -s /opt/davis/davis-0.7.2d davis

Then edit the configuration file: 

/opt/davis/davis/webapps/root/WEB-INF/web.xml

Change the settings of the following parameters: 

    <init-param>
        <param-name>webdavis.Log.threshold</param-name>
        <param-value>WARNING</param-value>
    </init-param>
     <init-param>              
        <param-name>insecureConnection</param-name>
        <param-value>shib</param-value>
    </init-param>     
                <init-param>
                        <description>
                                server type, srb or irods
                        </description>
                        <param-name>server-type</param-name>
                        <param-value>irods</param-value>
                </init-param>
                <init-param>
                        <description>
                                default idp name
                        </description>
                        <param-name>default-idp</param-name>
                        <param-value>ARCS IdP</param-value>
                </init-param>
                <init-param>
                        <description>
                                server port of srb/irods
                        </description>
                        <param-name>server-port</param-name>
                        <param-value>1247</param-value>
                </init-param>
                <init-param>
                        <description>
                                server name of srb/irods
                        </description>
                        <param-name>server-name</param-name>
                        <param-value>localhost</param-value>
                </init-param>
                <init-param>
                        <description>
                                zone name of srb/irods
                        </description>
                        <param-name>zone-name</param-name>
                        <param-value>ARCS</param-value>
                </init-param>
             
                <init-param>
                        <description>
                                default resource of user
                        </description>
                        <param-name>default-resource</param-name>
                        <param-value></param-value>
                </init-param>
    <init-param>
        <param-name>anonymousCollections</param-name>
        <param-value>/ARCS/worldview,/ARCS/projects/IMOS/public</param-value>
    </init-param>
    <init-param>
        <param-name>shared-token-header-name</param-name>
        <param-value>auEduPersonSharedToken</param-value>
    </init-param>
    <init-param>
        <param-name>cn-header-name</param-name>
        <param-value>cn</param-value>
    </init-param>
    <init-param>
        <param-name>admin-cert-file</param-name>
        <param-value>/etc/grid-security/daviscert.pem</param-value>
    </init-param>
    <init-param>
        <param-name>admin-key-file</param-name>
        <param-value>/etc/grid-security/daviskey.pem</param-value>
    </init-param>

Remember to remove settings for proxy (proxy-host and proxy-port)! The other options can be left unchanged.

Make a copy of the irods and key PEM files in /etc/grid-security. 

cp irodscert.pem daviscert.pem
cp irodskey.pem daviskey.pem
chown davis:davis davis*

Now modify jetty to use the AJP connector (/opt/davis/davis/etc/jetty.xml). Comment out the existing addConnector element, and add in the following XML: 

  <Call name="addConnector">
    <Arg>
       <New class="org.mortbay.jetty.ajp.Ajp13SocketConnector">
         <Set name="port">8009</Set>
         <Set name="ThreadPool">
           <New class="org.mortbay.thread.BoundedThreadPool">
             <Set name="minThreads">50</Set>
             <Set name="maxThreads">500</Set>
             <Set name="MaxIdleTimeMs">0</Set>
         </New>
         </Set>
         <Set name="statsOn">false</Set>
       </New>
    </Arg>
  </Call>

In the same file, also modify the ThreadPool? element to optomise performance: 

      <New class="org.mortbay.thread.QueuedThreadPool">
        <Set name="minThreads">50</Set>
        <Set name="maxThreads">2000</Set>
        <Set name="lowThreads">100</Set>
        <Set name="SpawnOrShrinkAt">200</Set>
      </New>

Now install startup script at init.d. 

cp /opt/davis/davis/bin/jetty.sh /etc/init.d/davis
chmod +x /etc/init.d/davis

And change the script to point to the davis environment variable file. Make the following change to line 134-6: 

elif [ -f /etc/default/davis ] ; then
  . /etc/default/davis
fi

Then setup the environment variables in the file /etc/default/davis 

JETTY_HOME=/opt/davis/davis
JAVA_OPTIONS="-server -Xms1024m -Xmx1562m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC"
JAVA_HOME=/usr/java/default
JETTY_USER=davis
JETTY_PID=/var/run/davis.pid

Double check your settings by running /etc/init.d/davis check: 

Checking arguments to Jetty: 
JETTY_HOME     =  /opt/davis/davis
JETTY_CONF     =  
JETTY_RUN      =  /var/run
JETTY_PID      =  /var/run/davis.pid
JETTY_PORT     =  
JETTY_LOGS     =  
CONFIGS        =  /opt/davis/davis/etc/jetty-logging.xml /opt/davis/davis/etc/jetty.xml
JAVA_OPTIONS   =  -server -Xms1024m -Xmx1562m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC -Djetty.home=/opt/davis/davis -Djava.io.tmpdir=/tmp
JAVA           =  /usr/java/default
CLASSPATH      =  
RUN_CMD        =  /usr/java/jdk1.6.0_10/bin/java -server -Xms1024m -Xmx1562m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC -Djetty.home=/opt/davis/davis -Djava.io.tmpdir=/tmp -jar /opt/davis/davis/start.jar  /opt/davis/davis/etc/jetty-logging.xml /opt/davis/davis/etc/jetty.xml

Now configure Apache to use ajp - add the following line in /etc/httpd/conf/httpd.conf 

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

Then modify the SSL config file: /etc/httpd/conf.d/ssl.conf

Install IPSCABUNDLE: 

cd /etc/grid-security/
wget http://certs.ipsca.com/companyIPSipsCA/IPS-IPSCABUNDLE.crt

Then modify ssl.conf: 

SSLCertificateChainFile /etc/grid-security/IPS-IPSCABUNDLE.crt
chmod 744 IPS-IPSCABUNDLE.crt

Secure copy across the df.arcs.org.au IPCSCA certificate from iVEC to VPAC and put them in /etc/grid-security 

/etc/grid-security/df-hostcert.pem
/etc/grid-security/df-hostkey.pem

Then modify VPAC's apache's ssl.conf to point to this pair of files: 

SSLCertificateFile /etc/grid-security/df-hostcert.pem
SSLCertificateKeyFile  /etc/grid-security/df-hostkey.pem

And add the following to the end of the file (before the closing VirtualHost? tag): 

ProxyRequests Off
ProxyPreserveHost On

ProxyPass /ARCS/ ajp://localhost:8009/ARCS/ flushpackets=on
RewriteEngine On
RewriteRule ^/$ /ARCS/home [R]

Make sure the SLCS certificate is installed. If not, copy the /etc/grid-security/arcs-slcs-ca folder from iVEC across to VPAC.

Now modify httpd config (/etc/httpd/conf/httpd.conf) and set the following values: 

Timeout 1200
KeepAlive On
MaxRequestsPerChild  0

Download the latest dojokit and store it under Apache document root: 

cd /tmp
wget http://download.dojotoolkit.org/release-1.3.1/dojo-release-1.3.1.tar.gz
tar -zxvf dojo-release-1.3.1.tar.gz /var/www/html
cd /var/www/html
ln -s /var/www/html/dojo-release-1.3.1 dojoroot

Start davis and httpd 

service davis start
service httpd start

Testing

Browse to http://arcs-df.vpac.org/ARCS/home and see if you can login.

Upload/download files to see if it is working. Also check that anonymous access to public areas under IMOS and worldview are functional.

Also check connectivity through webdev using Linux, Windows and Mac using Litmus.

DNS Change

Once everything is in working order, switch DNS entry to point df.arcs.org.au to arcs-df.vpac.org

Testing Procedures

Browse to http://arcs-df.vpac.org/ARCS/home and see if you can login.

Upload/download files to see if it is working. Also check that anonymous access to public areas under IMOS and worldview are functional.

Check webdav using Litmus.

Back-out Procedures

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-004 | 02 Jul 2009 - 16:19 | ---+!! Upgrade of grid virtual machines at Intersect

Description

Migrate the existing ng2, ngdata, and nggums virtual machines to CentOS 5.3 and VDT 1.10.1 on new hardware.

Estimated Duration

2 hours.

Systems/Services Affected

All ARCS grid services in NSW.

Staff Responsible

SimonYin

Detailed Instructions

Shutdown the existing ng2, ngdata and nggums virtual machines. Rename the new ng2, ngdata and nggums installations and test the grid workflow.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite? located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-005 | 21 Jul 2009 - 15:40 | ---+!! Migrate remaining ARCS Plone services to VPAC.

Description

Migration of an existing system to new infrastructure. Purpose is to strategically position the Plone infrastructure in a more centrally connected MARC and leverage the systems provided by the Data Services team. This outage secificly affects the ANDS site (http://plone.jcu.edu.au/ands)

Estimated Duration

Two hour. Which is the time taken for DNS entry to expire, plus the time to copy the data to VPAC.

Systems/Services Affected

plone.arcs.org.au

plone.jcu.edu.au/ands

Staff Responsible

RussellSim (Plone Migration)

Systems Staff Member (Provisioning VM)

Systems Staff Member (Enabling DNS plone.arcs.org.au)

Detailed Instructions

  1. Turn off Zope service on zope4.cluster on the JCU cluster
  2. Turn off Zeo service on zeo1.cluster on the JCU cluster
  3. Change Zeo Database to READ-ONLY
  4. Turn on Zeo service on zeo1.cluster on the JCU cluster
  5. Turn on Zope service on zope4.cluster on the JCU cluster
  6. Copy Data file from zeo1.cluster to db-vpac.arcs.org.au at VPAC.
  7. Turn on Zeo service at VPAC
  8. Turn on Zope service (plone.arcs.org.au) at VPAC
  9. Change website cache service url in portal_squidtool
  10. Change SMTP server location
  11. Replace plone.jcu.edu.au/ands(.*) with a 403 redirect to plone.arcs.org.au/ands(.*)

Testing Procedures

  1. Local data access.
  2. Test Backup procedure.

Back-out Procedures

  1. Disable redirect from plone.jcu.edu.au
  2. Change Zeo Database to at JCU to RW

Review

Schedule/Notification

6:30am 29th Jul 2009

Tracking completion

RT Ticket

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-006 | 08 Aug 2009 - 10:08 | ---+!! iRODS Update to createUser Script

Description

It has been found that some educational institutions (e.g. University of Queensland) include a courtesy title in their LDAP CN fields. In consequence, a user identified as "Ms Jane Doe" will be allocated a username like "mrs.doe".

The createUser script has therefore been enhanced so as to swallow the first word found in a CN field if that word can be found (case-transparently) as a "T" record in the createUser.config file.

The updated createUser and createUser.config files can be found at: http://projects.gridaus.org.au/trac/systems/wiki/DataServices/iRODS_Server For convenience, createUser and createUser.table (containing extensions to createUser.config) have been copied to directory: tango.vpac.org:/tmp/GJ

The arcs.irb file used on Production systems will also be updated (with addition of double-quote characters) during this Change to match what was previously done on Development machines.

Estimated Duration

The estimated durations for each Phase (including testing and backout steps) are as follows. There should be no interruption to service as seen by users.

Phase 1: 30 mins
Phase 2: 60 mins

Systems/Services Affected

Phase 1: irodsdev.vpac.org, data-dev.eresearchsa.edu.au, ngdata-dev.hpcu.uq.edu.au
Phase 2: arcs-df.hpcu.uq.edu.au, arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.eresearchsa.edu.au, arcs-df.vpac.org

Staff Responsible

Graham Jenkins

Detailed Instructions

On each server, login as the 'rods' user and proceed as follows. 

cd /tmp
scp graham@tango.vpac.org:/tmp/GJ/* .
cat  createUser.table >>$IRODS_HOME/server/config/createUser.config
cp $IRODS_HOME/server/bin/cmd/createUser $IRODS_HOME/server/bin/cmd/createUser.20090707
cp createUser $IRODS_HOME/server/bin/cmd

For Phase 2 only, also use SVN update the arcs.irb file at: https://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/Rules/ so that the acGetUserByDN rule looks like this: 

acGetUserByDN(*arg,*OUT)||msiExecCmd(createUser,'"*arg"',null,null,null,*OUT)|nop

Testing Procedures

Perform the following test, and check that the name of the created user is printed as: jane.doe 

export irodsConfigDir=$IRODS_HOME/server/config
$IRODS_HOME/server/bin/cmd/createUser "/C=AU/O=APACGrid/OU=VPAC/CN=Ms Jane Doe"
iadmin rmuser jane.doe

Back-out Procedures

Perform the following, then repeat the above test. 

cp $IRODS_HOME/server/bin/cmd/createUser.20090707 $IRODS_HOME/server/bin/cmd/createUser

Review

Schedule/Notification

Phase 1: 3:00 pm Tuesday July 7. Update: 11:00 am Thursday July 9.
Phase 2: 4:00 pm Tuesday July 7. Update: 9:00 am Friday July 10.

Tracking completion

Phase 1: Completed 3:20 pm Tuesday July 7. Update Completed 11:30 am Thursday July 9.
Phase 2: Completed 4:50 pm Tuesday July 7. Update Completed 10:10 am Friday July 10.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-007 | 19 Aug 2009 - 12:12 | ---+!! Installing AWSTATS

Description

Install Awstats to analyze your Apache log.

Estimated Duration

20 mins

Systems/Services Affected

DF

Staff Responsible

Shunde Zhang

Detailed Instructions

Get awstats 
wget "http://prdownloads.sourceforge.net/awstats/awstats-6.9.tar.gz"
Extract it to /usr/local as root 
tar xfvz awstats-6.9.tar.gz
mv awstats-6.9 /usr/local/awstats
Execute configuration script. 
cd /usr/local/awstats/tools
perl awstats_configure.pl
And answer the questions 
----- AWStats awstats_configure 1.0 (build 1.8) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/httpd/conf/httpd.conf

-----> Check and complete web server config file '/etc/httpd/conf/httpd.conf'
  Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'
  Add 'Alias /awstatscss "/usr/local/awstats/wwwroot/css/"'
  Add 'Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"'
  Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'
  Add '<Directory>' directive
  AWStats directives added to Apache config file.

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> df.arcs.org.au

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> 

-----> Create config file '/etc/awstats/awstats.df.arcs.org.au.conf'
 Config file /etc/awstats/awstats.df.arcs.org.au.conf created.

-----> Restart Web server with '/sbin/service httpd restart'
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=df.arcs.org.au
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue... 


A SIMPLE config file has been created: /etc/awstats/awstats.df.arcs.org.au.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'df.arcs.org.au' with command:
> perl awstats.pl -update -config=df.arcs.org.au
You can also read your statistics for 'df.arcs.org.au' with URL:
> http://localhost/awstats/awstats.pl?config=df.arcs.org.au

Press ENTER to finish...
Add a cron job to update stats 
crontab -e
0 * * * * /usr/local/awstats/tools/awstats_updateall.pl now >/dev/null 2>&1 || :
Edit configuration file 
vi /etc/awstats/awstats.df.arcs.org.au.conf
LogFile="cat /var/log/httpd/ssl_request_log /var/log/httpd/ssl_request_log.1 |"
Edit /etc/httpd/conf.d/ssl.conf to use combineio format 
#CustomLog logs/ssl_request_log \
#          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

CustomLog logs/ssl_request_log \
          "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O"
Create a directory for awstats results 
mkdir /var/lib/awstats
Analyze Apache log for the first time 
/usr/local/awstats/tools/awstats_updateall.pl now
Apply basic authentication to awstats. Firstly, generate a htpasswd file. 
cd /etc/httpd/conf.d
htpasswd -c http_passwd arcs-data
password: davisguru
Modify httpd. 
vi /etc/httpd/conf/httpd.conf
<Directory "/usr/local/awstats/wwwroot">
    Options None
    AllowOverride None
    Order allow,deny
    Allow from all
    AuthType Basic
    AuthName "ARCS Data"
    AuthUserFile /etc/httpd/conf.d/http_passwd
    Require valid-user
</Directory>
Reload HTTPD 
/etc/init.d/httpd reload

Testing Procedures

Go to http://df.arcs.org.au/awstats/awstats.pl?config=df.arcs.org.au and see if you can get the result page.

Back-out Procedures

Remove /etc/awstats, /usr/local/awstats and delete relevant entries from httpd.conf

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-008 | 03 Sep 2009 - 10:52 | ---+!! Installing MAMS level-1 SP

Description

Install a SP (MAMS level-2) to get shibboleth authentication.

Estimated Duration

1 hour

Systems/Services Affected

ARCS DF

Staff Responsible

Shunde Zhang

Detailed Instructions

Register your SP on MAMS registry. Go to http://www.federation.org.au/FedManager/jsp/index.jsp and create a new Host, under that Host, create a new SP (required attributes are sharedToken and cn).

Install SP 2.1, get these RPMs 

wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/log4shib-1.0-1.i386.rpm  
wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/opensaml-2.1-1.i386.rpm  
wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/shibboleth-2.1-1.i386.rpm  
wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/xerces-c-2.8.0-1.i386.rpm  
wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/xml-security-c-1.4.0-1.i386.rpm  
wget http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/archive/2.1/RPMS/i386/RHE/5/xmltooling-1.1-1.i386.rpm
Install dependencies and the above RPMs 
yum install libicu unixODBC
rpm -ivh *
Get a pre-made configuration file and server cert. 
cd /etc/shibboleth
wget http://projects.arcs.org.au/trac/systems/attachment/wiki/HowTo/InstallSLCSServer2/shibboleth2.xml?format=raw -O /etc/shibboleth/shibboleth2.xml
wget http://projects.arcs.org.au/trac/systems/attachment/wiki/HowTo/InstallSLCSServer2/attribute-map.xml?format=raw -O /etc/shibboleth/attribute-map.xml
wget http://www.federation.org.au/twiki/pub/Federation/UpdateMetadata/www.federation.org.au.pem
Generate a certificate for back-channel communication. 
mkdir /etc/certs && cd /etc/certs
openssl req -newkey rsa:1024 -nodes -keyout mykey.pem -out newreq.pem
cat newreq.pem
On the SP machine, open up a browser (elinks) and go to this URL http://www.federation.org.au/CA/CA-sign.html

Open the file newreq.pem with a text editor and copy the section between: 

   -----BEGIN CERTIFICATE REQUEST-----

   ......                          

   -----END CERTIFICATE REQUEST-----
Make sure you also include the above lines when copying.

Paste the information into the form on the browser and hit Submit. Copy the resultant certificate into a file and call it mycert.pem.

Make sure that your SP user (or whatever user that you use for SP shibd daemon) has access to these certificate files

Note that if you do not run the browser from the same machine as your SP, you will not be able to obtain a certificate

Download the level-1 CA certificate from this URL: http://www.federation.org.au/level-1-ca.crt into /etc/certs directory

Check that at this point you have the three files:

  • Your SP x509 certificate: mycert.pem
  • Your SP key: mykey.pem
  • Level-1 CA x509 certificate: level-1-ca.crt

Modify shibboleth2.xml, change MY_DNS to df.arcs.org.au. And change the following section to enable metadata auto-update. 

        <!-- Chains together all your metadata sources. -->
        <MetadataProvider type="Chaining">
            <!-- Example of remotely supplied batch of signed metadata. -->

            <MetadataProvider type="XML" uri="https://www.federation.org.au/level-1/level-1-metadata.xml"
                 backingFilePath="level-1-metadata.xml" reloadInterval="7200">
               <SignatureMetadataFilter certificate="www.federation.org.au.pem"/>
            </MetadataProvider>


            <!-- Example of locally maintained metadata. -->
           <!--
            <MetadataProvider type="XML" file="level-1-metadata.xml"/>
            -->
        </MetadataProvider>
Restart httpd and start shibd 
/etc/init.d/shibd start
/etc/init.d/httpd start

Testing Procedures

Put a PHP test file in /var/www/html/secure to test. 
<?php
// Version: 2007-07-26
// Latest version of this script will be on https://aai-viewer.switch.ch/viewer.php?source
//
// Author: Lukas Haemmerle <lukas.haemmerle@switch.ch>
// Bug reports etc. please to aai@switch.ch

// Specify your attribute-map.xml file and make sure it is readable by the web server
$attribute_map_file = '/etc/shibboleth/attribute-map.xml';

//Set header
header('Content-type: text/html; charset=utf-8');

// Show source
if (isset($_REQUEST['source'])) {
    highlight_file(__FILE__);
    exit;
}
?><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
    <title>SWITCH Attribute Viewer</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <style type="text/css">
<!--
a
{
    color: #1B3E93;
    font-size: 14px;
    font-weight: bold;
    text-decoration: none;
}

a:hover
{
    color: #FE911B;
    text-decoration: underline;
}

body 
{
    background-color: white;
    font-family: Verdana, Arial, Helvetica, sans-serif;
}

h1
{
    font-family: Verdana, Arial, Helvetica, sans-serif;
    font-size: 18px;
    font-weight: bold;
    text-decoration: none;
}

.logo
{
    color: white;
    text-decoration: none;
}

.border-blue
{
    border-style: solid;
    border-width: 1px;
    border-color: rgb(0,43,127);
    font-family: Verdana, Arial, Helvetica, sans-serif;
    font-size: 13px;
}

.border-orange
{
    border-style: solid;
    border-width: 1px;
    border-color: rgb(226,140,5);
    font-family: Verdana, Arial, Helvetica, sans-serif;
    font-size: 13px;
}

.blue
{
    color: rgb(0,43,127);
}

.orange
{
    color: rgb(226,140,5);
}

td.top-left {
    background-image: url('http://www.switch.ch/aai/design/images/topleft.gif');
    height: 14px;
    width: 14px;
}

td.top-middle {
    background-image: url('http://www.switch.ch/aai/design/images/topcenter.gif');
    height: 14px;
}

td.top-right {
    background-image: url('http://www.switch.ch/aai/design/images/topright.gif');
    height: 14px;
    width: 14px;
}
td.middle-left {
    background-image: url('http://www.switch.ch/aai/design/images/middleleft.gif');
    width: 14px;
}

td.middle-right {
    background-image: url('http://www.switch.ch/aai/design/images/middleright.gif');
    width: 14px;
}

td.bottom-left {
    background-image: url('http://www.switch.ch/aai/design/images/bottomleft.gif');
    height: 14px;
    width: 14px;
}

td.bottom-middle {
    background-image: url('http://www.switch.ch/aai/design/images/bottomcenter.gif');
    height: 14px;
}

td.bottom-right {
    background-image: url('http://www.switch.ch/aai/design/images/bottomright.gif');
    height: 14px;
    width: 14px;
}
-->
</style>
</head>

<body>
<div align="center">

<table border="0" cellpadding="0" cellspacing="0">
<!-- top left corner + middle bar + right corner -->
<tr>
    <td class="top-left"></td>
    <td class="top-middle"></td>
    <td class="top-right"></td>
</tr>
<!-- left border + content + right border -->
<tr>
    <!-- left border -->
    <td class="middle-left"></td>
    <!-- content -->
    <td>
    <a class="logo" href="http://www.switch.ch/aai">
        <img alt="SWITCHaai logo" src="http://www.switch.ch/aai/design/images/switch-aai-logo.gif" style="padding-bottom: 7px" /></a>
    <h1>SWITCH Attribute Viewer</h1>

<!-- table content -->
<table width="100%">
<tr>
<td class="blue">Attributes</td><td class="orange">Values</td>
</tr>
<?php 

$attribute_map_lines = file($attribute_map_file);
$attribute_map = '';
foreach($attribute_map_lines as $line){
    $attribute_map .= $line;
}

$p = xml_parser_create();
xml_parse_into_struct($p, $attribute_map, $vals, $index);
xml_parser_free($p);

$HTTP_SHIB_HEADERS = array();
foreach ($vals as $element){
    if ($element['tag'] == 'ATTRIBUTE' && isset($element['attributes']['ID'])){
        $HTTP_SHIB_HEADERS[$element['attributes']['ID']] = $element['attributes']['ID'];
    }
}

// Dump all received Shibboleth attributes
$status = '00';
foreach ($_SERVER as $key => $value){
    // Do we have any variables defined in attribute map
    if (isset($HTTP_SHIB_HEADERS[$key])){
        $status[1] = '1';
        echo '<tr valign="top">';
        echo '<td class="border-blue" valign="top">'.$HTTP_SHIB_HEADERS[$key].'</td>';
        $clean_value = ereg_replace('\$','<br>',htmlspecialchars(stripslashes($value)));
        if (ereg(';', $clean_value)){
            $clean_value = ereg_replace(";",'</tt></li><li><tt>',$clean_value); 
            $clean_value = '<ul><li><tt>'.$clean_value.'</tt></li></ul>';
        }
        else {
            $clean_value = '<tt>'.$clean_value.'</tt>';
        }
        
        echo '<td class="border-orange">'.$clean_value.'</td>';
        echo '</tr>';
    }
    // or any attributes starting with Shib-
    elseif (eregi('Shib', $key) ) {
        $status[0] = '1';
        echo '<tr>';
        echo '<td class="border-blue" valign="top"><span style="color: grey; font-style:italic;">'.$key.'</span></td>';
        echo '<td class="border-orange"><span style="color: grey; font-style:italic;"><tt>'.wordwrap(htmlspecialchars($value), 70, "<br/>\n", true).'</tt></span></td>';
        echo "</tr>\n";
    }
}
if (isset($_REQUEST['assertions'])) {
    $counter = 1;
    foreach ($_SERVER as $key=>$value){
        
        // Check if it is an assertion
        if (ereg('Shib-Assertion-Count', $key)  || !eregi('Shib-Assertion', $key)) {
            continue;
        }
        
        // Download the assertion
        $value = ereg_replace('dieng.switch.ch','127.0.0.1',stripslashes($value));
        
        $assertion = '';
        $handle = fopen($value, 'rb');
        if ($handle){
            while (!feof($handle)) {
                $tmp = fread($handle, 8192);
                if (!$tmp){
                    break;
                }
                $assertion .= $tmp;
            }
            fclose($handle);
        }
        
        echo '<tr><td colspan="2" class="border-orange"><h4>Assertion '.$counter.':</h4>';
        $assertion = ereg_replace('<', "\n<", $assertion);
        $assertion = preg_replace('/>(.+)/', ">\n$1", $assertion);
        $assertion = preg_replace("/\s([\S]+)=\"([^\"]+)/", "\n$1=\"$2",$assertion);
        $elements = preg_split('/\n/',$assertion);
        echo '<pre>';
        $indent = -1;
        foreach ($elements as $element){
            if (ereg('</', $element) && ereg('/>', $element)){
                echo '';
            }
            else if (ereg('/>', $element)){
                $reduce_indent = true;
                echo '';
            }
            elseif (ereg('</', $element)){
                $reduce_indent = true;
            }
            elseif (ereg('<', $element)){
                $indent++;
            }
            else {
                echo '&nbsp;&nbsp;';
            }
            
            for($i = 0; $i < $indent; $i++)
                echo '    ';
            
            // Syntax highlighting
            $element = ereg_replace('<','&lt;',$element);
            $element = ereg_replace('>','&gt;',$element);
            
            $element = preg_replace('/(\w+)="(.+)"/', " <span style=\"color:green\">$1</span>=<span style=\"color:brown\">&quot;$2&quot;</span>", $element);
            $element = preg_replace('/(&lt;.+)/', "<span style=\"color:blue\">$1</span>", $element);
            $element = ereg_replace('span>&gt;', "span><span style=\"color:blue\">&gt;</span>", $element);
            
            
            if (!ereg('=', $element) && !ereg('&lt;', $element))
                echo  '<span style="color:black;">'.wordwrap($element."\n", 120, "\n", 1).'</span>';
            else
                echo  wordwrap($element."\n", 120, "\n", 1);
            
            if ($reduce_indent){
                $indent--;
                $reduce_indent = false;
            }
        }
        
        echo '</pre>';
        echo '</td></tr>';
        
        $counter++;
    }
}

// Check status
if ($status == '10' ) {
    echo '<tr>';
    echo '<td colspan=2><b>Valid Shibboleth session but no user attributes received!</b></td>';
    echo '</tr>';
    echo '<tr>';
    echo '<td colspan=2>Hint to Home Organization administrators:<br>Please verify your metadata and ARP files.</td>';
    echo '</tr>';
}

elseif ($status == '00') {
    echo '<tr>';
    echo '<td colspan=2><b>No valid Shibboleth session!</b></td>';
    echo '</tr>';
    echo '<tr>';
    echo '<td colspan=2>This web page is probably not protected with Shibboleth. Hint to Home Organization administrators:<br>Have a look at your web server or Shibboleth configuration</td>';
    echo '</tr>';
}

else {
?>
<tr>
    <td colspan="2" align="center">
<?php
    if (isset($_REQUEST['assertions'])) {
        echo '<a href=".">Hide Shibboleth assertions</a>'; 
    } 
    else {
        echo '<a href="?assertions">Show Shibboleth assertions</a>'; 
    }
    
    if (isset($_REQUEST['all_variables'])) {
        echo ' | <a href=".">Hide all HTTP variables</a>'; 
    }
    else {
        echo ' | <a href="?all_variables">Show all HTTP variables</a>'; 
    }
?>
 | <a href="?source">Show PHP source</a>
    </td>
</tr>
<?php
}
?>
</table>
<!-- end content -->
</td>
  <!-- right border -->
  <td class="middle-right"></td>
</tr>
<!-- bottom left corner + middle bar + right corner -->
<tr>
  <td class="bottom-left"></td>
  <td><img src="http://www.switch.ch/aai/design/images/bottomcenter.gif" height="14" width="100%" alt="bottomcenter" /></td>
  <td class="bottom-right"></td>
</tr>
</table>

<!-- all HTTP variables -->
<?php
    if (isset($_REQUEST['all_variables'])) {
        ?>
<p>&nbsp;</p>
    <table>
        <tr>
            <td><strong>HTTP Environment Variables</strong></td><td><strong>Raw Values</strong></td></tr>
        <?php
        ksort($_SERVER);
        foreach ($_SERVER as $key => $value) {
            if ( ereg('^Shib-', $key)  || isset($HTTP_SHIB_HEADERS[$key])) 
            { 
                $class= "border-orange"; 
            }
            else {
                $class= "border-blue";
            }
            echo '<tr valign="top">';
            echo '<td class="'.$class.'">'.$key.'</td>';
            if (is_array($value)){
                echo '<td class="'.$class.'">';
                if (!empty($value)){
                    echo '<ul>';
                    foreach($value as $item){
                        echo '<li><tt>'.wordwrap($item, 70, '<br>', true).'</tt></li>';
                    }
                    echo '</ul>';
                }
                echo '</td>';
            } else {
                echo '<td class="'.$class.'"><tt>'.wordwrap(htmlspecialchars(stripslashes($value)), 70, '<br>', true).'</tt></td>';
            }
            echo "</tr>\n";
        }
    ?>
    </table>
    <?php
    } 
?>
</div>
</body>
</html>
If the file is called viewer.php (You need to install PHP by "yum install php"), then go to https://df.arcs.org.au/secure/viewer.php. You should see the WAYF. If login successfully, you'll see an attribute list with your SharedToken? and common name, etc.

Back-out Procedures

Stop shibd, remove config from Apache, remove RPMs by "rpm -e"

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-009 | 18 Jul 2009 - 21:15 | ---+!! Upgrade of Grid VMs at eRSA

Description

Replace the existing ng2, ngdata, and nggums VMs with newer versions based on CentOS 5.3 and VDT 1.10.1 on Arcturus.

Estimated Duration

4 hours allocated, 2 hours expected.

Systems/Services Affected

All Grid services provided at eRSA.

Staff Responsible

Garth Denley

Detailed Instructions

  • Shut down the existing ng2, ngdata, and nggums machines.
  • Using the pre-built and tested ng2new, ngdatanew, and nggums new VMs, change the IP address and hostnames for each to that of ng2, ngdata, and nggums.
  • Search through each machine for references to ng2, ngdata, and nggums using standard tools (eg. "find" and "grep"). Check each reference is correct.
  • Reboot each machine to ensure that it comes up cleanly with the new configuration.
  • Test the new machines, performing changes as problems are discovered.

Testing Procedures

  • Using local variants of the tests on http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite , test that job submission to all three clusters (hydra, aquila, corvus) works successfully, as can be best determined by current availability and reliability of those clusters.
  • Manually test gridftp transfers on ngdata.
  • Using the web interface on nggums, test various VO / cert combinations to confirm they are mapped to correct accounts.
  • Confirm that MDS information is being published correctly.
  • Confirm that job logs from all three clusters successfully reaches ng2new/ng2.

Back-out Procedures

  • Change the IP and hostname details for the new machines back to their previous state.
  • Bring the original ng2, ngdata, and nggums VMs back up.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-010 | 18 Jul 2009 - 20:58 | ---+!! Migration of unicast bridge in Perth to new hardware

Description

The unicast bridge agbridge-perth.arcs.org.au will be moved from aging hardware to a virtual machine hosted on new hardware.

Estimated Duration

Two hours.

Systems/Services Affected

AccessGrid unicast. However, there are two other unicast bridges within ARCS that can still be used: agbridge-canberra.arcs.org.au and agbridge-melb.arcs.org.au.

Staff Responsible

DarranCarey

Detailed Instructions

The existing system will be shutdown. The new system will be renamed and brought into service.

Testing Procedures

The system will be minimally tested after installation on Sunday July 19th. Detailed testing will be performed during the AG Test Session Monday July 20th.

Back-out Procedures

If any problem are encountered with the new system, it will be shutdown for later triage, and the existing system will be restarted.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-011 | 28 Jul 2009 - 14:34 | ---+!! TITLE

Enabling Postgres WAL Archiving On Datafabric

Description

We will use archived Write Ahead Log(WAL) files, to prepare backup to Postgres database. We will archive WAL files to another machine(arcs-df.vpac.org) so we can have offsite backup. To archive files to arcs-df.vpac.org, we need to have password less access between two machine[arcs-df.ivec.org & arcs-df.vpac.org] as its already been setup between these machines I am not adding that part in this changenote.

Estimated Duration

30mins

Systems/Services Affected

arcs-df.ivec.org

Staff Responsible

Sridhar Reddapani

Detailed Instructions

Backup of the existing database

  • The master database will be backed up using pgdump.

Stop Davis, iRODS and Postgres in sequence

  • service httpd stop
  • service davis stop
  • service irods stop
  • service postgres stop

Modify postgresql.conf file, add below line from line 183

  • archive_mode = on
  • archive_command = 'ssh arcs-df.vpac.org test ! -f /data/DataFabric_Backups/Current_Wal_Archives/%f && rsync -az %p arcs-df.vpac.org:/data/DataFabric_Backups/Current_Wal_Archives/%f'
  • checkpoint_timeout = 1h
  • archive_timeout = 12h

* Start Postgres, iRODS, davis, httpd in sequence*

  • service postgres start
  • service irods start
  • service davis start
  • service httpd start

Testing Procedures

Test whether files WAL files are being copied to arcs-df.vpac.org:/data/DataFabric_Backups/Current_Wal_Archives/ when they are fill or every 12 Hours.

Back-out Procedures

  • Stop all services in sequence as above
  • comment out above lines in postgresql.conf file
  • Start all services in sequence as above

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-012 | 11 Aug 2009 - 11:29 | ---+!! Upgrade of grid virtual machines at JCU

Description

Replace the existing ng2 and nggums virtual machines with newer versions based on CentOS 5.3 and VDT 1.10.1

Estimated Duration

4 hours allocated, 2 hours expected

Systems/Services Affected

All Grid services provided at JCU

Staff Responsible

WillHsu

Detailed Instructions

Shutdown the existing ng2 and nggums virtual machines. Rename the new ng2 and nggums installations and test the grid workflow. Restart INCA to make sure gridftp tests can be run from ng2.

Testing Procedures

Using local variants of the tests on http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite , test that job submission to cluster is successful.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-013 | 25 Jul 2009 - 20:51 | ---+!! Upgrade of grid virtual machines at NCI NF

Description

Migrate the existing NCI NF ng2 and nggums virtual machines to CentOS? 5.3 and VDT 1.10.1.

Estimated Duration

4 hours allocated, 2 hours expected.

Systems/Services Affected

NCI NF Grid Services

Staff Responsible

PaulWarren

Detailed Instructions

Shutdown the existing ng2 and nggums virtual machines. Rename the new ng2 and nggums installations and test the grid workflow.

Testing Procedures

Tests 1 through 4 of the Globus 4 TestSuite? located here http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite will be performed. If they all succeed, this installation will be put into production.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200907-014 | 01 Aug 2009 - 19:49 | ---+!! Grisu backend fixes

Description

Two changes to the Grisu backend web service are to be implemented to improve client performance.

Estimated Duration

30 minutes.

Systems/Services Affected

Grisu.

Staff Responsible

MarkusBinsteiner, SeanFleming

Detailed Instructions

An updated version of the webservice backend has been built. The tomcat server on grisu.vpac.org needs to be shutdown, the old war file needs to be replaced with the new one and tomcat needs to be restarted. That's all.

This adds better gridftp file listing speed and a force-mpi tag processing capability to the default ARCS Grisu backend.

Testing Procedures

Once the changes are in place and the web service has been restarted, a series of jobs will be submitted to various sites.

Back-out Procedures

If any issues arise during the testing procedures, the changes will be backed out by restoring the web service to it previous configuration and restarting the service. The testing phase will be rerun to ensure the service is operational.

Review

Schedule/Notification

Notification to be sent to the Community, Developers and Techstaff mailing lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-001 | 07 Aug 2009 - 15:52 | ---+!! TITLE

Description

Moving Datafabric ICAT from arcs-df.ivec.org to arcs-df.vpac.org

Estimated Duration

1 Hour

Systems/Services Affected

All Datafabric Servers
  • 1. arcs-df.vpac.org
  • 2. arcs-df.ivec.org
  • 3. arcs-df.ac3.edu.au
  • 4. arcs-df.sf.utas.edu.au
  • 5. arcs-df.hpcu.uq.edu.au
  • 6. arcs-df.eresearchsa.edu.au

Staff Responsible

Sridhar Reddapani

Detailed Instructions

Stopping required services

  • On arcs-df.ivec.org
    • service httpd stop
    • service davis stop
    • service irods stop

  • On arcs-df.vpac.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.hpcu.uq.edu.au and arcs-df.eresearchsa.edu.au
    • service irods stop

Take ICAT backup on arcs-df.ivec.org with pg_dump

  • mkdir /tmp/Backups
  • pg_dump ICAT > /tmp/Backups/ICAT_DUMP_IVEC

Take Full ICAT backup on arcs-df.ivec.org

  • service postgres stop
  • su - rods
  • cd ~/DB/pgsql
  • tar -cf /tmp/Backups/data.tar data/

Burn both ICAT_DUMP_IVEC and data.tar files to DVD and compare md5 sum of original files with copy on DVD, as additional backup

Copy ICAT_DUMP_IVEC from arcs-df.ivec.org to arcs-df.vpac.org

  • scp /tmp/Backups/ICAT_DUMP_IVEC arcs-df.vpac.org:/tmp/

Rebuild the iRODS on arcs-df.vpac.org with ICAT enabled

Tuning performence of postgres

  • Edit postgresql.conf as
    • listen_addresses = '*'
    • max_connections = 250
    • shared_buffers = 2048MB
    • effective_cache_size = 4194MB
    • autovacuum = on

Allow access to postgres from all other DF servers

  • Edit pg_hba.conf as
    • host ICAT rods 127.0.0.1/32 trust
    • # arcs-df.ivec.org
    • host ICAT rods 192.65.130.247/32 trust
    • # arcs-df.vpac.org
    • host ICAT rods 202.158.218.58/32 trust
    • # arcs-df.ac3.edu.au
    • host ICAT rods 203.202.30.74/32 trust
    • # arcs-df.sf.utas.edu.au
    • host ICAT rods 202.6.77.50/32 trust
    • # arcs-df.hpcu.uq.edu.au
    • host ICAT rods 130.102.163.138/32 trust
    • # arcs-df.eresearchsa.edu.au
    • host ICAT rods 129.127.96.93/32 trust

Configure Postgres WAL archiving on arcs-df.vpac.org

  • Enable password less ssh between arcs-df.vpac.org and arcs-df.ac3.edu.au
  • Edit /opt/iRODS-2.0v/Postgres/pgsql/data/postgresql.conf and add
    • archive_mode = on
    • archive_command = 'ssh arcs-df.ac3.edu.au test ! -f /data/DataFabric_Backups/Current_Wal_Archives/%f && rsync -az %p arcs-df.ac3.edu.au:/data/DataFabric_Backups/Current_Wal_Archives/%f'
    • checkpoint_timeout = 1h
    • archive_timeout = 12h

Start postgres on arcs-df.vpac.org and restore ICAT with backup file

  • cp -r /opt/iRODS-2.0v/Postgres/pgsql/data /tmp/data_bkp
  • service postgres start
  • dropdb ICAT
  • createdb ICAT
  • psql ICAT < /tmp/ICAT_DUMP_IVEC

Configure iRODS on all servers talk to arcs-df.vpac.org

  • edit ../server/config/server.config as
    • icatHost arcs-df.vpac.org

Start all required services

  • On arcs-df.vpac.org,arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.hpcu.uq.edu.au and arcs-df.eresearchsa.edu.au*
    • service irods start

Continue with Changenote http://wiki.arcs.org.au/bin/view/Main/ChangeNote200907-003 to move davis from arcs-df.ivec.org to arcs-df.vpac.org

Testing Procedures

  • Try accessing DF with icommands
  • Test auto user creation is working on arcs-df.vpac.org
  • Test all rules are functional on arcs-df.vpac.org

Back-out Procedures

Stopping required services
  • On arcs-df.vpac.org
    • service irods stop
    • service postgres stop
  • On arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.hpcu.uq.edu.au and arcs-df.eresearchsa.edu.au
    • service irods stop

Configure all irod servers talk to arcs-df.ivec.org for ICAT

  • edit ../server/config/server.config as
    • icatHost arcs-df.ivec.org

Start all required services

  • On arcs-df.ivec.org
    • service postgres start
    • service irods start
  • On arcs-df.vpac.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.hpcu.uq.edu.au and arcs-df.eresearchsa.edu.au*
    • service irods start

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-002 | 06 Aug 2009 - 18:27 | ---+!! Upgrade of grid virtual machines at UQ/UQ ESSCC

Description

Replace the existing ng2, nggums, and ngdata virtual machines with newer versions based on CentOS 5.3 and VDT 1.10.1

Estimated Duration

8 hours allocated, 4 hours expected, 2 hours per each site.

Systems/Services Affected

All Grid services provided at UQ and UQ ESSCC

Staff Responsible

WillHsu

Detailed Instructions

Shutdown the existing ng2, nggums, and ngdata virtual machines. Rename the new installations and test the grid workflow.

Testing Procedures

Using local variants of the tests on http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite , test that job submission to cluster is successful.

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then these installations will be turned off, and the previous production installations will be brought back online.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-003 | 04 Sep 2009 - 10:32 | ---+!! Update arcs.org.au Plone site to update products/features

Description

The ARCS Plone website requires a configuration update in order to introduce several new features to the site (forum improvements [email notification, improved post/thread management], Google Apps integration) and to, at the same time, update a number of existing 3rd party add-on products for stability and security. This outage affects the ARCS site located at http://www.arcs.org.au/

Estimated Duration

Two hours, in which the configuration will be updated and products reinstalled on the site.

Systems/Services Affected

ARCS Plone Site: http://www.arcs.org.au/

Staff Responsible

DavidBreitkreutz

Detailed Instructions

  1. Pull updated ARCS buildout configuration onto the server
  2. Re-run the buildout process to pull in the updated products/changes
  3. Pull down the latest version of the ARCS customisations product
  4. Stop the ARCS Plone server process
  5. Start the ARCS Plone server process in the foreground to make sure it is working after the update
  6. Stop the ARCS Plone server process
  7. Start the ARCS Plone server process normally
  8. On the website front end, reinstall each relevant product that required updating

Testing Procedures

  1. Verify site process is operational again
  2. Verify ARCS site is accessible via the web interface

Back-out Procedures

  1. Restore previous configuration
  2. Re-run buildout process to restore previous products
  3. Undo any changes that were carried out on the site.

Review

Schedule/Notification

6:30am 11th September 2009

Tracking completion

RT Ticket

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-004 | 14 Sep 2009 - 11:10 | ---+!! Upgrading iRODS from 2.0.1 to 2.1 on DataFabric

Upgrading iRODS from 2.0.1 to 2.1 on DataFabric

Description

Upgrading iRODS from 2.0.1 to 2.1 on DataFabric

Estimated Duration

3 hours

Systems/Services Affected

All DF Servers
  • arcs-df.vpac.org
  • arcs-df.ivec.org
  • arcs-df.ac3.edu.au
  • arcs-df.sf.utas.edu.au
  • arcs-df.hpcu.uq.edu.au
  • arcs-df.eresearchsa.edu.au
  • arcs-df.hpsc.csiro.au

Staff Responsible

Sridhar Reddapani, Graham Jenkins, Pauline Mak

Detailed Instructions

Note: This documentation is on assumption, that you have installed iRODS2.0.1v at /opt/iRODS-2.0v/iRODS and postgre at /opt/iRODS-2.0v/Postgres

On arcs-df.ivec.org irods is installed at /opt/rods/iRODS and postgres is at /opt/rods/DB

On arcs-df.eresearchsa.edu.au irods is installed at /data/irods/iRODS and postgres is at /data/irods/Postgres

Preliminary Filesystem Reorganisation on arcs-df.vpac.org

As user root run
  • chkconfig --add httpd
  • init 2
  • fuser -c /opt # Ensure that no processes are shown
  • mv /opt /opt-20090912
  • ln -s /data /opt
  • cd /opt-20090912
  • find . -print | cpio -pdm /opt
  • init 3 # Check iRODS/Davis functionality

VDT Upgrade [all servers]

Ref: http://vdt.cs.wisc.edu/releases/1.10.1/release-p24.html

As user root run

  • . /opt/vdt/setup.sh
  • cd $VDT_LOCATION
  • vdt-control --off
  • pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:VDT-Updater
  • pacman -update VDT-Updater
  • mkdir /opt/vdt-20080912
  • cp -pr $VDT_LOCATION /opt/vdt-20080912
  • vdt/update/vdt-updater
  • .. Absolute path to backup: /opt/vdt-20080912/vdt
  • vdt-control --on
  • service irods restart # Check iRODS/Davis functionality

Upgrading ICAT enabled servers[arcs-df.vpac.org]

As user root run
  • service davis stop
  • service httpd stop
  • service irods stop
  • mkdir /opt/iRODS-2.1v
  • chown -R rods:rods /opt/iRODS-2.1v
  • Install openssl-devel [if not already installed]
    • yum install openssl-devel
  • su - rods

As user rods run

  • Download irods2.1.tgz to /tmp
  • cd /opt/iRODS-2.1v
  • tar -xvf /tmp/irods2.1.tgz
  • mkdir /tmp/patch
  • cd /tmp/patch
  • wget ftp://ftp.sdsc.edu/pub/outgoing/mwan/irods/patchFor2.1.tar
  • tar -xvf patchFor2.1.tar
  • cp -r patchFor2.1/* /opt/iRODS-2.1v/iRODS
  • unset IRODS_HOME
  • export PATH=`echo $PATH | sed -e 's/iRODS/xRODS/g'`
  • cd /opt/iRODS-2.1v/iRODS/
  • cp /opt/iRODS-2.0v/iRODS/config/irods.config config/
  • /opt/iRODS-2.0v/Postgres/pgsql/bin/psql ICAT < server/icat/src/psg-patch-v2.0.1tov2.1.sql
  • ./irodssetup --upgrade
  • cd /opt/iRODS-2.1v/iRODS
  • mkdir -p server/bin/local
  • cd server/bin/local
  • wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/Rules/updateRules.sh
  • chmod a+xr updateRules.sh
  • edit crontab [crontab -e] as below to point to new iRODS_HOME=/opt/iRODS/iRODS
    • 0 6 * * * /usr/local/bin/irodsLogPrune.sh /opt/iRODS/iRODS >/dev/null 2>&1 || :
    • 0 * * * * /opt/iRODS/iRODS/server/bin/local/updateRules.sh YOUR_DEFAULT_RESOURCE >/dev/null 2>&1 || :
  • Run "/opt/iRODS/iRODS/server/bin/local/updateRules.sh YOUR_DEFAULT_RESOURCE" to download rule files.

As user root run

Upgrading ICAT less servers

As user root run

  • service irods stop
  • mkdir /opt/iRODS-2.1v
  • chown -R rods:rods /opt/iRODS-2.1v
  • Install openssl-devel [if not already installed]
    • yum install openssl-devel
  • su - rods

As user rods run

  • Download irods2.1.tgz to /tmp
  • cd /opt/iRODS-2.1v
  • tar -xvf /tmp/irods2.1.tgz
  • mkdir /tmp/patch
  • cd /tmp/patch
  • wget ftp://ftp.sdsc.edu/pub/outgoing/mwan/irods/patchFor2.1.tar
  • tar -xvf patchFor2.1.tar
  • cp -r patchFor2.1/* /opt/iRODS-2.1v/iRODS
  • unset IRODS_HOME
  • export PATH=`echo $PATH | sed -e 's/iRODS/xRODS/g'`
  • cd /opt/iRODS-2.1v/iRODS/
  • cp /opt/iRODS-2.0v/iRODS/config/irods.config config/
  • edit irods.config as
    • $IRODS_ICAT_HOST = 'arcs-df.vpac.org';
  • ./irodssetup --upgrade
  • cd /opt/iRODS-2.1v/iRODS
  • mkdir -p server/bin/local
  • cd server/bin/local
  • wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/Rules/updateRules.sh
  • chmod a+xr updateRules.sh
  • edit crontab [crontab -e] as below to point to new iRODS_HOME=/opt/iRODS/iRODS
    • 0 6 * * * /usr/local/bin/irodsLogPrune.sh /opt/iRODS/iRODS >/dev/null 2>&1 || :
    • 0 * * * * /opt/iRODS/iRODS/server/bin/local/updateRules.sh YOUR_DEFAULT_RESOURCE >/dev/null 2>&1 || :
  • Run "/opt/iRODS/iRODS/server/bin/local/updateRules.sh YOUR_DEFAULT_RESOURCE" to download rule files.

As user root run

Enable Trash

  • Add the following line in arcs.irb and check into subversion. 

acTrashPolicy||nop|nop

This will be propagated to all data fabric node within the hour. arcs.irb should also be modified on arcs-df.vpac.org

Testing Procedures

Back-out Procedures

As root
  • service irods stop
  • rm -f /opt/iRODS
  • ln -s /opt/iRODS-2.0v/ /opt/iRODS
  • service irods start

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-005 | 25 Aug 2009 - 11:38 | ---+!! Inca Migration to Arcturus

Description

The Inca VM will be migrated as-is from grid-gateway to Arcturus as part of a multi-step migration.

Proposed Date

The migration will be performed on 19th August from 3pm AEST through 4pm AEST

Estimated Duration

1 hour

Systems/Services Affected

The Inca testing suite will be inactive during the migration. No real user impact.

Staff Responsible

  • Garth Denley: System migration
  • Will Hsu: Inca startup and testing

Detailed Instructions

The existing Inca VM will be shut down. The image file for the root filesystem will be copied (and compressed) from grid-gateway to Arcturus and placed into an preprepared VM shell. The filesystem will then be mounted on the host, expanded to the new size, fstab modified to reflect the new layout, console changed, and then unmounted. The VM will be brought up and tested.

The old Inca VM will be retained as a back-out option for this and future migration tasks.

All of these steps have been previously undertaken in a test environment.

Testing Procedures

The machine will be tested to ensure it is still working correctly.

The Inca suite will be initialised and tested to ensure that it is still behaving properly.

Back-out Procedures

The new VM will be shut down and disabled. The old VM will be brought back up again.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-006 | 20 Aug 2009 - 16:57 | ---+!! Adding rules for EM-XRay collections

Description

Files under the EM-XRay collection must be stored at UQ.

Estimated Duration

1 hour

Systems/Services Affected

Data Fabric on all production boxes:

arcs-df.hpcu.uq.edu.au, arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.sf.utas.edu.au, arcs-df.eresearchsa.edu.au, arcs-df.vpac.org, arcs-df.hpsc.csiro.au

Staff Responsible

Pauline Mak

Detailed Instructions

Add the EMXray rules into the server config $IRODS_HOME/server/config/server.config 

reRuleSet   emxray, imos,arcs,core

The updateRules.sh script should update itself automatically to grab the emxray.irb file from subversion.

Testing Procedures

Login as the rods user, and modify the ~/.irods/.irodsEnv file and comment out the default resource setting.

Put a file in the following directory on the data fabric. Create a dummy text file, then 

icd /ARCS/projects/EMXRAY/
iput <some test file>
ils -L

The result should say the file is now on arcs-df.qcif.edu.au

Remove the file using irm and uncomment the default resource setting in ~/.irods/.irodsEnv.

Back-out Procedures

Remove imos from the server configuration file: $IRODS_HOME/server/config/server.config 

reRuleSet   imos, arcs,core

Remove the imos rules file: 

rm $IRODS_HOME/server/config/reConfigs/emxray.irb

Review

Schedule/Notification

Enable EMXray rules after the outage on 20th August.

Tracking completion

VPAC, TPAC, iVEC, UQ - done.

AC3 - done

eRSA - done, but have to add "wget -o SCRATCH http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/Rules/updateRules.sh?q=$$" to stop proxy cache...

CSIRO - done.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-007 | 03 Nov 2009 - 17:00 | ---+!! Cron Job for Moving IMOS-ACORN Staging Data

Description

This note describes the steps necessary to create a cron job for moving IMOS-ACORN staging data to archive and opendap directories.

Estimated Duration

10 minutes

Systems/Services Affected

Only QCIF/UQ DF machine: arcs-df.hpcu.uq.edu.au

Staff Responsible

KaiLu

Detailed Instructions

As user rods:

run:

cd /opt/iRODS/iRODS/server/bin/local

wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/utils/ACORN_Data_Move.sh/?format=raw -O ACORN_Data_Move.sh

chmod +x ACORN_Data_Move.sh

crontab -e and adding the following line:

5,25,45 * * * * /opt/iRODS/iRODS/server/bin/local/ACORN_Data_Move.sh >> /opt/iRODS/iRODS/server/log/ACORN-Moving-Data 2>&1

The cron job will be run once every 20 minutes

Testing Procedures

As user rods:

run:

cd /opt/iRODS/iRODS/server/bin/local

wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/utils/filesChecking.sh/?format=raw -O filesChecking.sh

chmod +x filesChecking.sh

find /data/Vault/ARCS/projects/IMOS/staging/ACORN -daystart -type f \( -mtime 0 -or -mtime 1 \) > fileList

/opt/iRODS/iRODS/server/bin/local/ACORN_Data_Move.sh >> /opt/iRODS/iRODS/server/log/ACORN-Moving-Data 2>&1

./filesChecking.sh -f fileList

If nothing is printed, this means that the cron job script works properly.

rm -f fileList

When you visit the web site: opendap-qcif.arcs.org.au, you will also see those files that have just been moved to directory opendap from directory staging

Back-out Procedures

As user rods:

run:

crontab -e and remove the following line:

5,25,45 * * * * /opt/iRODS/iRODS/server/bin/local/ACORN_Data_Move.sh >> /opt/iRODS/iRODS/server/log/ACORN-Moving-Data 2>&1

rm -f /opt/iRODS/iRODS/server/bin/local/ACORN-Moving-Data.sh

rm -f /opt/iRODS/iRODS/server/log/ACORN-Moving-Data

rm -f /opt/iRODS/iRODS/server/bin/local/filesChecking.sh

Review

Schedule/Notification

Tracking completion

Done. -- KaiLu - 03 Nov 2009

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-008 | 26 Aug 2009 - 13:37 | ---+!! Hollowworld, CISM - New Trac Instances

Description

This outage is needed to implement:
  1. Hollowworld - a Trac site along the scheme used for Fullmoon
  2. CISM - a Trac site for the Community Ice Shelf Model
  3. Urgent kernel and apache security updates

Estimated Duration

5 minutes

Systems/Services Affected

http://projects.arcs.org.au https://projects.arcs.org.au

Staff Responsible

Simon Yin

Detailed Instructions

The URLs for the new sites are:

https://projects.arcs.org.au/trac/hollowworld https://projects.arcs.org.au/svn/hollowworld

https://projects.arcs.org.au/trac/cism https://projects.arcs.org.au/svn/cism

Detailed instruction may be found at: https://projects.arcs.org.au/trac/systems/wiki/HowTo/UpgradeAndMigrateTracSubversion

Testing Procedures

For Hollowworld, the following scenarios are tested:

  1. Windows XP, IE8 user
    1. Connect anonymously to http://203.202.139.98/trac/hollowworld
      1. View tickets
    2. Register as "TestUser"
    3. (https only) Connect as "TestUser" with no extra permissions to https://203.202.139.98/trac/hollowworld
      1. Create a new ticket
    4. (https only) Connect as user with full admin permissions to https://203.202.139.98/trac/hollowworld
      1. Change the password of "TestUser" (and then login again as "TestUser" to check the password)
    5. Connect to http://203.202.139.98/svn/hollowworld
  2. Repeat the tests in 1. for a Mac OSX, Safari user
  3. Repeat the tests in 1. for a Linux, Firefox 3.5.x user
  4. (https only) Linux, Subversion from command line (if necessary, check out from a known sub-directory of the project-under-test to save time)
    1. svn co https://203.202.139.98/svn/hollowworld

Where tests listed above have used the http protocol, repeat them with the https protocol. The new site is first configured at a standby site (IP address 203.202.139.98). After these tests are completed successfully, the configuration is dumped to the production server (projects.arcs.org.au). A restart of httpd is needed on projects.arcs.org.au.

* Important For Hollowworld * Once running on projects.arcs.org.au, make a test ticket and ensure that emails are sent to the hollowworld-dev@arcs.org.au mailing-list

Back-out Procedures

On projects.arcs.org.au, before changing them to enable Hollowworld and CISM, make copies of

/etc/httpd/conf.d/trac.conf as /etc/httpd/conf.d/trac.conf.YYYYMMDD

/etc/httpd/conf.d/subversion.conf as /etc/httpd/conf.d/subversion.conf .YYYYMMDD

Revert to the above config settings if the changes fail.

Also, the postgreSQL database used by Hollowworld and CISM must be dropped.

Finally, delete the directories:

  1. /var/www/trac/hollowworld
  2. /var/www/svn/hollowworld
  3. /var/www/trac/cism
  4. /var/www/svn/cism

Review

Schedule/Notification

Notification by email:

For ARCS internal projects:  
  techstaff@arcs.org.au, developers@arcs.org.au
For Fullmoon  
  simon.cox@jrc.ec.europa.eu, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com
For podd  
  gkam08@gmail.com, liyf@itee.uq.edu.au, james.eddes@adelaide.edu.au, jane@itee.uq.edu.au, ben.joseph@utas.edu.au, xueqin.wang@csiro.au, paul.coddington@arcs.org.au, andrew.treloar@ands.org.au, Adrienne.McKenzie@anu.edu.au, Donald.Hobern@csiro.au, Robert.Furbank@csiro.au
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au, vincent.post@falw.vu.nl
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For common-grid-libs  
  aron.abrook@qut.edu.au
For commons-vfs-grid  
  manish.saroha@its.monash.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk, nguyen_h@chem.usyd.edu.au, gerson.galang@versi.edu.au
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For grisu  
  y.halytskyy@auckland.ac.nz
For grix  
  aron.abrook@qut.edu.au
For gsub  
  sam.moskwa@sapac.edu.au
For jmoleditor  
  Joachim.Mai@ac3.edu.au, vvv900@gmail.com, Martin.Thompson@unsw.edu.au
For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For access  
  martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For pht3d  
  Henning.Prommer@csiro.au, Aaron.McDonough@csiro.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org
For sggc  
  Terry.Rankine@csiro.au
For slcs-client  
  chi.nguyen@melco.mq.edu.au

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200908-009 | 25 Aug 2009 - 18:08 | ---+!! Inca Grid Monitor software upgrade at eRSA

Description

Upgrade current Inca VM to CentOS 5.3 and Inca 2.5

Estimated Duration

6 hours allocated, 3 hours expected.

Systems/Services Affected

Grid Monitoring/Testing.

Staff Responsible

WillHsu

Detailed Instructions

Shutdown Inca processes, yum update, then upgrade Inca software.
Reboot Inca VM, start Inca processes.

Testing Procedures

Starts manual Inca tests via incat java control application, check test results via http://inca.arcs.org.au/

Back-out Procedures

Shutdown Inca processes
mv /home/inca /home/inca.new
mv /home/inca.old /home/inca
Starts previous version of Inca processes

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-001 | 08 Sep 2009 - 13:05 | ---+!! ARCS Sakai Upgrade to Version 2.5.5

Description

The ARCS Sakai server is being upgraded to version 2.5.5 which includes security fixes. In addition to the upgrade, additional bug fixes to WebDAV? (Windows Vista issue) and Search tool are included.

Proposed Date

Friday 11th September 12:00pm AEST

Estimated Duration

One hour

Systems/Services Affected

All sites on the ARCS Sakai server.

Staff Responsible

  • Denny (via ARCS email, Jabber)- contact for Sakai-level issues.
  • Garth Denley (via ARCS email, Jabber)- contact for system-level issues.

Detailed Instructions

  • The tomcat service will be stopped.
  • The database will be dumped.
  • The filestore used by Sakai will be backed up to the same mountpoint in a different directory (plenty of space).
  • The symbolic link /opt/tomcat will be updated to point to the new version of Sakai.
  • The tomcat service will be started.

Testing Procedures

  • The Sakai site will be tested to see if any problems arise.

Back-out Procedures

  • The tomcat service will be stopped.
  • The database will be dropped and restored from backup.
  • The filestore backup will be copied back to the live filestore.
  • The symbolic link /opt/tomcat will be updated to point to the old version of Sakai.
  • The tomcat service will be started.

UPDATE 8/9:

  • A slight change in the back-out procedures: The modified database will not be dropped. Instead, we will restore the database backup to a new table name, and adjust sakai.properties to use that. This gains us the ability to use the modified database as a last resort should database restoration fail for some reason.

Review

Schedule/Notification

Notifications are to be sent to the Community, Developers, and Techstaff mailing lists. Sakai users will also be notified by email.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-002 | 18 Feb 2010 - 10:55 | ---+!! Davis change for iRODS 2.1

Description

Update Davis to a version that works with iRODS 2.1 (due to Jargon changes...)

Estimated Duration

30 Minutes

Systems/Services Affected

arcs-df.vpac.org

Staff Responsible

Pauline Mak

Detailed Instructions

Stop davis and httpd if running: 

service davis stop
service httpd stop

Then move the current deployed version of davis: 

cd /opt/davis
mv davis-0.7.3 davis-0.7.3_OLD
rm davis

As the davis user, download webavis from GoogleCode? : 

wget http://webdavis.googlecode.com/files/davis-0.7.3.tar.gz
tar zxvf davis-0.7.3.tar.gz
mv /tmp/davis-0.7.3 /opt/davis/

Then copy over config files: 

cp davis-0.7.3_OLD/webapps/root/WEB-INF/web.xml davis-0.7.3/webapps/root/WEB-INF/
cp davis-0.7.3_OLD/etc/*.xml davis-0.7.3/etc/

Then start httpd and davis: 

service davis start
service httpd start

Testing Procedures

Test with litmus on irodsdev.vpac.org

Back-out Procedures

Stop davis Remove the new version of davis and rename davis-0.7.3_OLD to davis-0.7.3 Start davis

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-003 | 09 Sep 2009 - 19:42 | ---+!! Migrate LDAP server for IdP?

Description

Migrate the LDAP server off idp.arcs.org.au onto dedicated host ldap.arcs.org.au

Estimated Duration

30 minutes

Systems/Services Affected

Directly: idp.arcs.org.au, openidp.arcs.org.au

Indirectly: Users of the ARCS IdP will be unable to login to shibboleth services during this time. (DataFabric being the main one)

Staff Responsible

Sam Morrison

Detailed Instructions

Dump LDIF from idp.arcs.org.au then load it into ldap.arcs.org.au

Change configuration on: IdP? registration app idp.arcs.org.au login handler idp.arcs.org.au resolver handler openidp.arcs.org.au login handler openidp.arcs.org.au resolver handler

Testing Procedures

Testing has been done using a copy of the data and the unused idp.arcs.org.au

Back-out Procedures

Change configuration changes back.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-004 | 15 Sep 2009 - 13:02 | ---+!! VDT Upgrade

Description

VDT will be upgraded from version 1.10.1y to 1.10.1p24. This is primarily for security updates.

Estimated Duration

15 minutes.

Systems/Services Affected

All ARCS Grid services.

Staff Responsible

All Systems Services staff.

Detailed Instructions

All ARCS Grid virtual machines will be upgraded from VDT 1.10.1y to 1.10.1p24 via the vdt-updater script.

Testing Procedures

Once the upgrade has been performed, tests 1 and 2 from the test suite will be run. If any fail, the upgrade will be rolled back.

Back-out Procedures

The vdt-updater script forces you to make a backup of the existing VDT installation. If backout is required, the backup copy of VDT will be moved back into production.

Review

Schedule/Notification

Notification to be sent to community, developers, and techstaff lists.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-005 | 18 Sep 2009 - 17:51 | ---+!! Fix to ibun Command

Fix to ibun Command

Description

This fix allow us to extract tar files using "ibun -x" command in any collection, if you have write permission to it.

Estimated Duration

1 Hour

Systems/Services Affected

All DF Server

Staff Responsible

Sridhar Reddapani

Detailed Instructions

Testing Procedures

Test using ibun to extarct tar files to any collection other than home collection

Back-out Procedures

  • service httpd stop [if necessary]
  • service davis stop [if necessary]
  • service irods stop
  • service postgres stop [if necessary]
  • cd /opt/iRODS/iRODS/server/api/src/
  • rm rsStructFileExtAndReg.c
  • mv /tmp/rsStructFileExtAndReg.c
  • cd /opt/iRODS/iRODS
  • make
  • service postgres start [if necessary]
  • service irods start
  • service davis start [if necessary]
  • service httpd start [if necessary]

Review

Schedule/Notification

Tracking completion

Patch applied to

iVEC

eRSA

UQ

Intersect

TPAC

VPAC

iVEC

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-006 | 17 Sep 2009 - 17:23 | ---+!! Upgrade Davis to 0.8.0

Description

Upgrade Davis to 0.8.0

Estimated Duration

30mins

Systems/Services Affected

ARCS DF

Staff Responsible

Shunde Zhang

Detailed Instructions

Stop davis.

Make a copy of current davis instance. Change the owner of that directory properly. 

cp -r davis-0.7.3 davis-0.8.0
replace davis.jar in webapps/root/WEB-INF/lib

Modify web.xml 

      <init-param>
         <description>
            organisation name - default is Davis
         </description>
         <param-name>organisation-name</param-name>
         <param-value>Australian Research Collaboration Service</param-value>
      </init-param>
      <init-param>
         <description>
            authentication realm - default is Davis
         </description>
         <param-name>authentication-realm</param-name>
         <param-value>ARCS Data Fabric</param-value>
      </init-param>
      <init-param>
         <description>
            organisation logo
         </description>
         <param-name>organisation-logo</param-name>
         <param-value>/images/logo.jpg</param-value>
      </init-param>
      <init-param>
         <description>
            organisation logo geometry - widthxheight
         </description>
         <param-name>organisation-logo-geometry</param-name>
         <param-value>32x32</param-value>
      </init-param>   
      <init-param>
         <description>
            favicon for browser title bar
         </description>
         <param-name>favicon</param-name>
         <param-value>http://www.arcs.org.au/favicon.ico</param-value>
      </init-param>
      <init-param>
         <description>
            myproxy server where certs are stored
         </description>
         <param-name>myproxy-server</param-name>
         <param-value>myproxy.arcs.org.au</param-value>
      </init-param>

Copy images files from tar ball to /var/www/html

Change the 'davis' soft link to 0.8.0, and restart davis.

Testing Procedures

Litmus, WebDAV? clients, etc.

Back-out Procedures

Change the 'davis' soft link back to 0.7.3, and restart davis.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-007 | 17 Sep 2009 - 17:12 | ---+!! slcs-server - New Trac Instances

Description

This outage is needed to implement:
  1. slcs-server - a Trac site for the SLCS Proxy project
  2. ciomp - (was CISM) renamed a Trac site
  3. Urgent kernel and apache security updates

Estimated Duration

5 minutes

Systems/Services Affected

http://projects.arcs.org.au https://projects.arcs.org.au

Staff Responsible

Simon Yin

Detailed Instructions

The URLs for the new sites are:

https://projects.arcs.org.au/trac/slcs-server https://projects.arcs.org.au/svn/slcs-server

https://projects.arcs.org.au/trac/ciomp https://projects.arcs.org.au/svn/ciomp

Detailed instruction may be found at: https://projects.arcs.org.au/trac/systems/wiki/HowTo/UpgradeAndMigrateTracSubversion

Testing Procedures

Since ciomp is just a renamed site, no further tests are needed. Since slcs-server is for internal ARCS use, no detailed tests are needed

Back-out Procedures

On projects.arcs.org.au, before changing them to enable Hollowworld and CISM, make copies of

/etc/httpd/conf.d/trac.conf as /etc/httpd/conf.d/trac.conf.YYYYMMDD

/etc/httpd/conf.d/subversion.conf as /etc/httpd/conf.d/subversion.conf .YYYYMMDD

Revert to the above config settings if the changes fail.

Also, the postgreSQL database used by ciomp and slcs-server must be dropped.

Finally, delete the directories:

  1. /var/www/trac/ciomp
  2. /var/www/svn/ciomp
  3. /var/www/trac/slcs-server
  4. /var/www/svn/slcs-server

Review

Schedule/Notification

Notification by email:

For ARCS internal projects:  
  techstaff@arcs.org.au, developers@arcs.org.au
For Fullmoon  
  simon.cox@jrc.ec.europa.eu, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com
For podd  
  gkam08@gmail.com, liyf@itee.uq.edu.au, james.eddes@adelaide.edu.au, jane@itee.uq.edu.au, ben.joseph@utas.edu.au, xueqin.wang@csiro.au, paul.coddington@arcs.org.au, andrew.treloar@ands.org.au, Adrienne.McKenzie@anu.edu.au, Donald.Hobern@csiro.au, Robert.Furbank@csiro.au, joanne.evans@unimelb.edu.au
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au, vincent.post@falw.vu.nl
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For common-grid-libs  
  aron.abrook@qut.edu.au
For commons-vfs-grid  
  manish.saroha@its.monash.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk, nguyen_h@chem.usyd.edu.au, gerson.galang@versi.edu.au
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For grisu  
  y.halytskyy@auckland.ac.nz
For grix  
  aron.abrook@qut.edu.au
For gsub  
  sam.moskwa@sapac.edu.au
For jmoleditor  
  Joachim.Mai@ac3.edu.au, vvv900@gmail.com, Martin.Thompson@unsw.edu.au
For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For access  
  martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org
For sggc  
  Terry.Rankine@csiro.au
For slcs-client  
  chi.nguyen@melco.mq.edu.au
For ciomp  
  bkgalton@utas.edu.au
For holllowworld  
  See Fullmoon

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-008 | 29 Sep 2009 - 17:36 | ---+!! Intersect - Short Term Outages (<5 min) Due To Network Upgrade

Description

Intersect is upgrading to a pair of 1Gb/s switches at Global Switch Data Centre    
The reason for this change is to    
* Improve data transfer speed for the ARCS Data Fabric  
* Provide failover for high-availability needs  

Estimated Duration

< 5 min

Systems/Services Affected

All ARCS systems and services hosted by Intersect    
* projects.arcs.org.au  
* arcs-df.ac3.edu.au  
* ng2.ac3.edu.au  
* nggums.ac3.edu.au  
* idp.ac3.edu.au  
* grid.ac3.edu.au  
* auth86.ac3.edu.au  
* auth01.ac3.edu.au - auth16.ac3.edu.au  
* ARCS piwik server  

Staff Responsible

Simon Yin - simon.yin@arcs.org.au

Detailed Instructions

This is an initial population of 2 new 1Gb/s switches. The interruption will occur when the uplink to AARNET is transferred from an existing switch to the new switches. It will be a short duration outage but will affect all public services.

Testing Procedures

Prior to installation at Global Switch, the new switches have been power cycled and preconfigured by AC3.

Back-out Procedures

Move the uplink to AARNET back to the old Cisco switch.

Review

Schedule/Notification

Notification by email:

For ARCS internal projects:  
  techstaff@arcs.org.au, developers@arcs.org.au
For Fullmoon  
  simon.cox@jrc.ec.europa.eu, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com
For podd  
  gkam08@gmail.com, liyf@itee.uq.edu.au, james.eddes@adelaide.edu.au, jane@itee.uq.edu.au, ben.joseph@utas.edu.au, xueqin.wang@csiro.au, paul.coddington@arcs.org.au, andrew.treloar@ands.org.au, Adrienne.McKenzie@anu.edu.au, Donald.Hobern@csiro.au, Robert.Furbank@csiro.au, joanne.evans@unimelb.edu.au
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au, vincent.post@falw.vu.nl
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For common-grid-libs  
  aron.abrook@qut.edu.au (Defunct)
For commons-vfs-grid  
  manish.saroha@its.monash.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk, nguyen_h@chem.usyd.edu.au, gerson.galang@versi.edu.au
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For grisu  
  y.halytskyy@auckland.ac.nz
For grix  
  aron.abrook@qut.edu.au
For gsub  
  sam.moskwa@sapac.edu.au
For jmoleditor  
  Joachim.Mai@ac3.edu.au, vvv900@gmail.com, Martin.Thompson@unsw.edu.au
For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For access  
  martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org
For sggc  
  Terry.Rankine@csiro.au
For slcs-client  
  chi.nguyen@melco.mq.edu.au (Defunct)
For ciomp  
  bkgalton@utas.edu.au
For holllowworld  
  See Fullmoon

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-009 | 01 Oct 2009 - 11:28 | ---+!! Network changes at VPAC

Description

Network changes at VPAC

Estimated Duration

All day

Systems/Services Affected

All systems hosted at VPAC

Staff Responsible

Andy Botting, Sam Morrison

Detailed Instructions

Network changes at VPAC will cause ARCS services to be unavailable.

  • arcs.org.au
  • arcs-df.vpac.org
  • status.arcs.org.au
  • plonedev1.arcs.org.au
  • voms.arcs.org.au
  • wikitest.arcs.org.au
  • codedev.arcs.org.au
  • shib-mp.arcs.org.au
  • code.arcs.org.au
  • static.arcs.org.au
  • services.arcs.org.au
  • rrdev.arcs.org.au
  • plone.arcs.org.au
  • fedmgr0.arcs.org.au
  • arcs-vh01.vpac.org
  • arcs-vh02.vpac.org
  • arcs-vh03.vpac.org
  • idp.arcs.org.au
  • chat.arcs.org.au
  • www.arcs.org.au
  • plonedev.arcs.org.au
  • rt.arcs.org.au
  • arcs-db.vpac.org
  • crm.arcs.org.au
  • support.arcs.org.au
  • myproxy2.arcs.org.au
  • slcs1.arcs.org.au
  • slcstest.arcs.org.au
  • slcsca.arcs.org.au
  • evo.arcs.org.au
  • test.arcs.org.au
  • dreamwidth.arcs.org.au
  • agbridge-melb.arcs.org.au
  • odl.arcs.org.au
  • openidp.arcs.org.au
  • nagios.arcs.org.au
  • hosting.arcs.org.au
  • ds.arcs.org.au
  • ds-test.arcs.org.au
  • ldap.arcs.org.au
  • vomrs.arcs.org.au

Testing Procedures

All the hostnames listed will be tested once the migration is complete.

Back-out Procedures

The old hardware will still be available if the migration is not successful. Reverting to the old infrastructure will be possible if necessary.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-010 | 29 Sep 2009 - 10:17 | ---+!! Fix to ibun to extract files to any resource on Data Fabric

Fix to ibun to extract files to any resource on DataFabric?

Description

This fix allow us to extract tar files using "ibun -x" command in any resource, if you have write permission to it.

Estimated Duration

1 Hour

Systems/Services Affected

All DF Server

Staff Responsible

Sridhar Reddapani

Detailed Instructions

  • service httpd stop [if necessary]
  • service davis stop [if necessary]
  • service irods stop
  • service postgres stop [if necessary]
  • su - rods
  • cd /opt/iRODS/
  • tar cvf pre_ibun_second_patch_iRODS.tar iRODS/
  • cd /opt/iRODS/iRODS/server/api/src/
  • mv rsPhyPathReg.c /tmp/
  • wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/BugFix/rsPhyPathReg.c
  • cd /opt/iRODS/iRODS
  • make
  • exit
  • service postgres start [if necessary]
  • service irods start
  • service davis start [if necessary]
  • service httpd start [if necessary]

Testing Procedures

Test using ibun to extarct tar files to different resources.

Back-out Procedures

  • service httpd stop [if necessary]
  • service davis stop [if necessary]
  • service irods stop
  • service postgres stop [if necessary]
  • su - rods
  • cd /opt/iRODS
  • mv iRODS /tmp/
  • tar -xvf pre_ibun_second_patch_iRODS.tar
  • exit
  • service postgres start [if necessary]
  • service irods start
  • service davis start [if necessary]
  • service httpd start [if necessary]

Review

Schedule/Notification

Tracking completion

iVEC ---Done

eRSA ---

UQ ---Done

Intersect ---Done

TPAC ---

VPAC ---Done

CISRO --Done

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200909-011 | 28 Sep 2009 - 12:04 | ---+!! Davis and SP integration

Description

Configure Apache to get Davis shibbolized

Estimated Duration

10 mins

Systems/Services Affected

ARCS DF (no downtime is needed)

Staff Responsible

Shunde Zhang

Detailed Instructions

Prerequisites: SP should be installed and working properly.

Make sure the Davis configuration file, web.xml, has: 

    <init-param>
        <param-name>insecureConnection</param-name>
        <param-value>shib</param-value>
    </init-param>       
    <init-param>   
        <param-name>shared-token-header-name</param-name>
        <param-value>auEduPersonSharedToken</param-value>
    </init-param>
    <init-param>
        <param-name>cn-header-name</param-name>
        <param-value>cn</param-value>
    </init-param>     
    <init-param>    
        <param-name>admin-cert-file</param-name>
        <param-value>/etc/grid-security/daviscert.pem</param-value>
    </init-param>
    <init-param> 
        <param-name>admin-key-file</param-name>
        <param-value>/etc/grid-security/daviskey.pem</param-value>
    </init-param>
And rods has DN of the above cert mapped. 
rods@arcs-df ~ $ iadmin lu rods
user_id: 10007
user_name: rods
user_type_name: rodsadmin
zone_name: ARCS
user_distin_name: /C=AU/O=APACGrid/OU=VPAC/CN=arcs-df.vpac.org
user_info: 
r_comment: 
create_ts: 01234939508 : 2009-02-18.17:45:08
modify_ts: 01253773433 : 2009-09-24.16:23:53
Then add a conf file to httpd conf.d. 
vi /etc/httpd/conf.d/arcs.conf

<VirtualHost *:80>
  ServerName df.arcs.org.au
  DocumentRoot "/var/www/html"

  ProxyPass /ARCS/ ajp://localhost:8009/ARCS/

  <Location /ARCS>
  AuthType shibboleth
  ShibRequireSession On
  ShibUseHeaders On
  require valid-user
  </Location>

  <Location /secure>
  AuthType shibboleth
  ShibRequireSession On
  ShibUseHeaders On
  require valid-user
  </Location>

</VirtualHost>
Restart shibd and RELOAD httpd. 
/etc/init.d/shibd restart
/etc/init.d/httpd reload

Testing Procedures

Go to http://df.arcs.org.au/ARCS/home

Back-out Procedures

delete arcs.conf from /etc/httpd/conf.d

reload httpd

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-001 | 06 Oct 2009 - 13:48 | ---+!! TITLE

Upgrading PGPOOL on arcs-01.ivec.org

Description

Upgrading PGPOOL to latest version Pgpool-2.2.5 on arcs-01.ivec.org

Estimated Duration

30 Mins

Systems/Services Affected

arcs-01.ivec.org

Staff Responsible

Sridhar Reddapani

Detailed Instructions

pgpool stop

As Root

cd /data/pgsql/

mkdir pgpool-2.2.5

chown -R postgres:postgres pgpool-2.2.5/

su -postgres

cd /data/pgsql/pgpool-2.2.5/

mkdir pgpool

mkdir logs

Download latest pgpool to /tmp

cd /tmp/

tar -xvf pgpool-II-2.2.5.tar.gz

cd /tmp/pgpool-II-2.2.5

./configure --prefix=/data/pgsql/pgpool-2.2.5/pgpool/

make

make install

cd /data/pgsql/pgpool-2.2.5/pgpool/etc

cp /data/pgsql/pgpool/pgpool/etc/pcp.conf ./

cp /data/pgsql/pgpool/pgpool/etc/pgpool.conf ./

cp /data/pgsql/pgpool/pgpool/etc/pool_hba.conf ./

exit

cd /data/pgsql/

mv pgpool/ pgpool-2.2.1/

ln -s pgpool-2.2.5/ pgpool

su - postgres

pgpool -version

pgpool -d -n 2>&1 | /usr/local/sbin/cronolog --hardlink=/data/pgsql/pgpool/logs/pgpool.log '/data/pgsql/pgpool/logs/%Y-%m-%d-pgpool.log' &

tail -f /data/pgsql/pgpool/logs/pgpool.log

Testing Procedures

Test failover is working by simulating failure node

Back-out Procedures

pgpool stop

As Root

cd /data/pgsql/

rm pgpool

ln -s pgpoo-2.2.1/ pgpool

su - postgres

pgpool -version

pgpool -d -n 2>&1 | /usr/local/sbin/cronolog --hardlink=/data/pgsql/pgpool/logs/pgpool.log '/data/pgsql/pgpool/logs/%Y-%m-%d-pgpool.log' &

tail -f /data/pgsql/pgpool/logs/pgpool.log

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-002 | 20 Oct 2009 - 10:19 | ---+!! Intersect - Move Data Fabric to 1Gb/s link, add Redundant PSU and NIC to Projects machine, Connect ng2/nggums host to 1Gb/s link

Description

Intersect now has 1Gb/s switches at Global Switch Data Centre    
This change is to    
* Improve data transfer speed for the ARCS Data Fabric by moving uplinks to the 1Gb/s switch  
* Install extra PSU and NIC to allow Highly-Available service from the Projects Xen Host  
* Move Grid ng2/nggums services to 1Gb/s switch  

Estimated Duration

1 Hour

Systems/Services Affected

The following ARCS systems and services hosted by Intersect will be intermittent during this period    
* projects.arcs.org.au - Trac and Subversion sites  
* arcs-df.ac3.edu.au  
* ng2.ac3.edu.au  
* nggums.ac3.edu.au  
* idp.ac3.edu.au  

Staff Responsible

Simon Yin - simon.yin@arcs.org.au

Detailed Instructions

projects.arcs.org.au (RU21) - Trac and Subversion sites    
* Shutdown the projects.arcs.org.au VM  
* Activate the standby projects.arcs.org.au VM  
* Power down the host located at RU21  
* Disconnect the power-cord to RU21  
* Connect the new NIC card to PCIe Slot #2  
* Connect the second PSU to the empty power bay  
* Restart the host and disable the standby projects.arcs.org.au  
* Restart the production projects.arcs.org.au VM  

ng2/nggums/idp (RU22)    
Public VLAN    
* Identify the existing Ethernet connection to Public (HPCC_FRONT) VLAN  
* Connect the above to RU27 (ARCS SW1 IA069B? ) Port 5  
Private VLAN    
* Identify an unused Ethernet port  
* Connect the above to RU27 (ARCS SW1 IA069B? ) Port 16  

arcs-df.ac3.edu.au - Data Fabric (RU20)    
Public VLAN    
* Identify the existing Ethernet connection to Public (HPCC_FRONT) VLAN  
* Connect the above to RU27 (ARCS SW1 IA069B? ) Port 4  
Private VLAN    
* Identify the existing Ethernet connection to Private (NODE) VLAN  
* Connect the above to RU27 (ARCS SW1 IA069B? ) Port 15  
mclarenfs VLAN    
* Identify the existing Ethernet connection to Mclarenfs (DATA) VLAN  
* Connect the above to RU27 (ARCS SW1 IA069B? ) Port 22  

Testing Procedures

  • Complete the installation of the second PDU and NIC on the Projects host machine first.
  • After disabling the standby and restarting the production VM, point a web-browser at https://projects.arcs.org.au/trac/systems and confirm the production version has restarted and is accessible
  • Next, complete the connection of the ng2/nggums/idp to the 1Gb/s switch. Confirm the functionality with a grid job submission
  • Once Step 3 is successful, perform the connection upgrade of the arcs-df machine

Back-out Procedures

If a machine does not restart correctly after the installation of new hardware, remove that hardware and return to prior configuration. Contact IBM for warranty replacement of the component.

If a network connection does not return upon moving to the 1Gb/s switch, first try a different port (on the same VLAN). If this is unsuccessful then return to the old Cisco switch connection.

Review

Schedule/Notification

Notification by email:

For ARCS internal projects:  
  techstaff@arcs.org.au, developers@arcs.org.au
For Fullmoon  
  simon.cox@jrc.ec.europa.eu, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com
For podd  
  gkam08@gmail.com, liyf@itee.uq.edu.au, james.eddes@adelaide.edu.au, jane@itee.uq.edu.au, ben.joseph@utas.edu.au, f.davies@uq.edu.au, kxkevin@gmail.com, xueqin.wang@csiro.au, paul.coddington@arcs.org.au, andrew.treloar@ands.org.au, Adrienne.McKenzie@anu.edu.au, Donald.Hobern@csiro.au, Robert.Furbank@csiro.au, joanne.evans@unimelb.edu.au
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au, vincent.post@falw.vu.nl
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For common-grid-libs  
  aron.abrook@qut.edu.au (Defunct)
For commons-vfs-grid  
  manish.saroha@its.monash.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk, nguyen_h@chem.usyd.edu.au, gerson.galang@versi.edu.au
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For grisu  
  y.halytskyy@auckland.ac.nz
For grix  
  aron.abrook@qut.edu.au (Defunct)
For gsub  
  sam.moskwa@sapac.edu.au
For jmoleditor  
  Joachim.Mai@ac3.edu.au, vvv900@gmail.com, Martin.Thompson@unsw.edu.au
For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For access  
  martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au, ronald@ivec.org
For sggc  
  Terry.Rankine@csiro.au
For slcs-client  
  chi.nguyen@melco.mq.edu.au (Defunct)
For ciomp  
  bkgalton@utas.edu.au, msd@ccpo.odu.edu
For hollowworld  
  See Fullmoon

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-003 | 02 Nov 2009 - 11:23 | ---+!! MDS VM upgrade at UQ

Description

Upgrade current MDS VM to CentOS 5.3 and latest vdt release.

Estimated Duration

4 hours allocated, 2 hours expected

Systems/Services Affected

all ng2 publishing MDS info to UQ MDS server.

Staff Responsible

WillHsu

Detailed Instructions

Shut down MDS VM. Boot CentOS 5 MDS VM, install vdt and MDS rpms according to the following doco:

http://projects.arcs.org.au/trac/systems/wiki/InfoSystems/CentralIndexServiceInstall

Testing Procedures

Check log output /opt/vdt/globus/var/container-real.log

Manual check with wsrf-query:

wsrf-query -s https://mds1.arcs.org.au:8443/wsrf/services/DefaultIndexService "//*[local-name()='Site']"|less

Back-out Procedures

If any of the tests fail, and cannot be remedied within the allocated time, then the new VM will be turned off, and the previous production VM will be brought back online.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-004 | 23 Oct 2009 - 09:09 | ---+!! SLCS moving to AAF Federation

Description

The host slcs1.arcs.org.au will be moving from the MAMS Level-2 federation to the AAF Pilot Production Federation

Estimated Duration

Allow 5 mins to include testing etc. Actual down time will be less than 3 seconds (the time it takes to restart shibd)

Systems/Services Affected

All users that request short lived certificates

The two AuScope portals that use the SLCS Delgation Service

Grix and Grisu - They use the SLCS Client

DataFabric? ?? - Do they still use the slcs client?

Staff Responsible

SamMorrison

Detailed Instructions

Modify the shibboleth configuration file on slcs1 to point to the AAF Discovery Service instead of the MAMS Where Are You From Service.

Restart shibd

Testing Procedures

Use slick-init command line tool to request a certificate.

Test Grix and Grisu

Test Datafabric

Back-out Procedures

Revert to old shibboleth configuration

Review

Schedule/Notification

Plan to do this on Thursday the 29th Oct at 9:00am

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-005 | 02 Nov 2009 - 11:20 | ---+!! EVO Server Migration

Description

EVO server will be migrated from it's current physical host to a virtual machine.

Estimated Duration

Less than 2 hours

Systems/Services Affected

EVO Web-start and registrations for http://evo.arcs.org.au

Staff Responsible

Andy Botting, Sam Morrison

Detailed Instructions

The EVO server filesystem will be rsync'd across to a newly created filesystem on an ARCS virtual machine host. Once it has been rsync'd, the EVO services will be shut down, and a final rsync will occur to ensure that the new virtual machine has all the latest configuration.

The EVO server will then be shut down, and the virtual machine will be brought up.

Testing Procedures

Test that EVO can be started by the web-start link and new user registrations are successful.

Back-out Procedures

The old physical server can be booted if the migration is unsuccessful.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-006 | 02 Nov 2009 - 11:37 | ---+!! Migrating ARCS Wiki to another host

Description

The ARCS Wiki will be moved from its current virtual machine (hosted on VPAC infrastructure) to a new ARCS virtual machine.

Estimated Duration

Less than 2 hours

Systems/Services Affected

ARCS TWiki, and all included webs.

Staff Responsible

Andy Botting, Lev Lafayette

Detailed Instructions

The TWiki will be rsync'd from it's current host to a new host. When the change window begins, the current TWiki will be made unavailable and a final rsync will occur.

The TWiki will then be started again on the new host.

Cron jobs for TWiki management will also be moved.

Testing Procedures

Testing each of the webs, and doing some page write tests.

Back-out Procedures

Enable the IP address of the TWiki on the old host. Re-enable the TWiki virtual host on the old virtual machine.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-007 | 22 Oct 2009 - 12:18 | ---+!! TITLE

Attaching New Storage to arcs-df.ac3.edu.au

Description

Attaching 4TB storage to Intesct DataFabric? node[arcs-df.ac3.edu.au]

Estimated Duration

1 Hour

Systems/Services Affected

arcs-df.ac3.edu.au

Staff Responsible

Sridhar Reddapani

Detailed Instructions

As Root

service irods stop

cp -r /DataFabric /DataFabric1

umount /DataFabric

unmount /DataFabric1

mount /dev/DfVolGroup/Datafabric /DataFabric/

Add below line to /etc/fstab

/dev/DfVolGroup/Datafabric /DataFabric1/ ext3 defaults,acl 0 0

service irods start

Testing Procedures

check all files are there with icommands

Back-out Procedures

As Root

service irods stop

umount /DataFabric

mount /dev/etherd/e0.0 /DataFabric

service irods start

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-008 | 28 Oct 2009 - 09:24 | ---+!! TITLE Adding New Index to ICAT Database

Description

Adding New Index [idx_coll_main3] to ICAT database on arcs-df.vpac.org

Estimated Duration

10 Mins

Systems/Services Affected

arcs-df.vpac.org

Staff Responsible

Sridhar Reddapani

Detailed Instructions

As user rods

psql ICAT

create unique index idx_coll_main3 on R_COLL_MAIN (coll_name);

\q

Testing Procedures

psql ICAT

\di

see there are 19 indexes and idx_coll_main3 is listed there

Back-out Procedures

As user rods

psql ICAT

drop index idx_coll_main3;

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200910-009 | 28 Oct 2009 - 10:36 | ---+!! Adding TDS to arcs-df.vpac.org

Description

Installing a THREDDS Data Server on the VPAC data box to serve data through OPeNDAP? .

Estimated Duration

1 hour

Systems/Services Affected

arcs-df.vpac.org and df.arcs.org.au (need a httpd reload/restart)

Staff Responsible

Pauline Mak

Detailed Instructions

Detailed installation notes here: https://projects.arcs.org.au/trac/systems/wiki/DataServices/OPeNDAP-TDS-Install-Apache

Testing Procedures

Back-out Procedures

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-001 | 06 Nov 2009 - 17:03 | ---+!! New Davis update (0.8.1)

Description

Updating davis, which contains a bug fix for trash, etc. See release notes for more detials:

http://projects.arcs.org.au/trac/davis/wiki/ReleaseNotes/0.8.1

Estimated Duration

30 minutes

Systems/Services Affected

ARCS Data Fabric, arcs-df.arcs.org.au

Staff Responsible

Pauline Mak

Detailed Instructions

As root, stop davis and remove symlink:

service davis stop

rm /opt/davis/davis

Download new davis, unpack and recreate the symlink: 

cd /opt/davis 
wget http://webdavis.googlecode.com/files/davis-0.8.1.tar.gz 
tar -zxvf davis-0.8.1.tar.gz 
rm davis-0.8.1.tar.gz  
rm davis 

ln -s davis-0.8.1 davis 
chown -R davis:davis  davis-0.8.1

Make a new directory for persistent config files and copy config over 

mkdir /opt/davis/etc 
cp /opt/davis/davis/webapps/root/WEB-INF/host-dfvpac.properties /opt/davis/etc/host-local.properties

And run the new configuration script: 

sh davis-configure.sh

Then modify settings in /opt/davis/davis/webapps/root/WEB-INF/davis-host.properties 

server-name=localhost
zone-name=ARCS
default-domain=arcs-df.vpac.org
default-resource=
anonymousCollections=/ARCS/worldview,/ARCS/projects/IMOS/public

Copy davis images across to root directory: 

cp /opt/davis/davis/webapps/images/* /var/www/html/images

Restart davis (service davis start)

Testing Procedures

Test with litmus, then check with web interface

Make sure that permissions and metadata can be applied to gz files.

Also test other web interface function

Back-out Procedures

Link back to older version of davis

As root: 

service davis stop 
cd /opt/davis rm davis
ln -s davis-0.8.0 davis   
service davis start

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-002 | 19 Nov 2009 - 15:31 | ---+!! Migrate Plone Storage to blobs

Description

Migrate ZODB to Blob based storage. This will improve the speed of plone and allow larger files to be stored more easily.

Estimated Duration

3 Hours

Systems/Services Affected

www.arcs.org.au

Staff Responsible

Russell Sim

David Breitkreutz

Detailed Instructions

extend buildout to include 

[instance]



...



eggs +=



    plone.app.blob



    Products.contentmigration



zcml +=



    plone.app.blob



    Products.contentmigration

Visit: http://<site>/@@blob-image-migration and http://<site>/@@blob-file-migration to migrate content over to blobs.

Testing Procedures

Test downloading a file and viewing an image.

Back-out Procedures

Backup before begining remove migration section from buildout and revert to backup of database.

Review

Schedule/Notification

6am 1/12/09.

Notification should go to the TechStaff? mailing list

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-003 | 28 Nov 2009 - 10:57 | ---+!! Installing new SLCS client library, Davis 0.8.2

Description

Installing the latest SLCS library to cope with the new AAF Pilot WAYF page as well as updating Davis. Also allowing core dumps to be written to track iRODS crashes.

Estimated Duration

1 hour

Systems/Services Affected

df.arcs.org.au

Staff Responsible

Pauline Mak, Graham Jenkins

Pre-Change

Update /etc/init.d/irods by replacing the first runuser line with: 

    runuser -s /bin/bash - $IRODS_USER -c \
      "ulimit -c 8192; $Env $PORTS $IRODS_HOME/irodsctl $_P"

Detailed Instructions

Stop service, download and install Davis 0.8.2, as root: 

service davis stop 
cd /opt/davis/ 
wget http://webdavis.googlecode.com/files/davis-0.8.2.tar.gz 
tar -zxvf davis-0.8.2 
cd /opt/davis

rm davis 
ln -s /opt/davis/davis-0.8.2/ davis 
chown -R davis:davis /opt/davis/davis-0.8.2 
cd /opt/davis/davis/bin 
sh davis-configure.sh

Modify slcs-client properites to point to the slcs1 server: 

cd /opt/davis/davis/webapps/root/WEB-INF/classes/ 

slcs.server=https://slcs1.arcs.org.au/SLCS/login   
ssl.blindtrust=false

Then modify the davis config like so (/opt/davis/etc/host-local.properties) 

server-name=localhost
zone-name=ARCS
default-domain=arcs-df.vpac.org
default-resource=
anonymousCollections=/ARCS/worldview,/ARCS/projects/IMOS/public
insecureConnection=shib
admin-cert-file=/etc/grid-security/daviscert.pem
admin-key-file=/etc/grid-security/daviskey.pem
shared-token-header-name=auEduPersonSharedToken
cn-header-name=cn
methods=LOCK PROPFIND
organisation-name=Australian Research Collaboration Service
authentication-realm=ARCS Data Fabric
organisation-logo=/images/logo.jpg
organisation-logo-geometry=32x32
favicon=http://www.arcs.org.au/favicon.ico
myproxy-server=myproxy.arcs.org.au
server-type=irods
server-port=1247
default-idp=arcs idp

Save the file and restart Davis

Testing Procedures

Litmus test

Also test using IdP? login with the new AAF Pilot enabled SLCS server.

Back-out Procedures

As the ARCS SLCS server will be using the Pilot AAF, there will be no back out procedure, unless SLCS is also moved back into MAMS.

For some reason, setting blindtrust to false disabled webdav and web based access to the DF. Will investigate issue as it was working on test system.

Review

Schedule/Notification

Tracking completion

Change note applied 28th Nov at 9:15am. Checked with litmus and have passed tests as expected, and also checked on web interface with the TPAC IdP? and Mac Finder mount.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-004 | 15 Dec 2009 - 15:34 | ---+!! Authorisation Service VM domain and IP address change

Description

Intersect will begin using it's own allocation of public IP addresses - 203.161.182.0/24

The Authorisation Team has 16 IP addresses in a borrowed (from AC3) range - migration of these to the Intersect range is a priority because AC3 will soon revoke the borrowed range.

Estimated Duration

Phase 1: Initial test on 1 VM - 1 hour

Phase 2: TBD

Systems/Services Affected

Phase 1: auth13.ac3.edu.au

Staff Responsible

Simon Yin

Detailed Instructions

Step 1: Create an alias IP address on auth13.ac3.edu.au

This involves the following: 

ssh root@auth13.ac3.edu.au

cat - <<EOF >/etc/sysconfig/network-scripts/ifcfg-eth0:1
DEVICE=eth0:1
BOOTPROTO=static
IPADDR=203.161.182.113
NETMASK=255.255.255.0
EOF
Step 2: On auth13.ac3.edu.au, edit /etc/resolv.conf and change the nameserver (existing should be 203.202.139.100) to 203.161.182.100

Step 3: On auth13.ac3.edu.au, edit /etc/sysconfig/network and add GATEWAY=203.161.182.253

Step 4: ifup eth0:1

Perform the testing procedures below before moving to the next step.

Step 5: Contact ac3 to change DNS (a-name) to

auth13.ac3.edu.au 203.161.182.113

Repeat testing procedures below but now use auth13.ac3.edu.au instead of the IP address.

Testing Procedures

After Step 4: * While on a command shell in auth13.ac3.edu.au, test the DNS by trying: nslookup www.google.com and confirm that a response is given.

* From another ARCS site, telnet 203.161.182.113 443 and confirm that a response is given.

* From another ARCS site, telnet 203.161.182.113 8443 and confirm that a response is given.

* From another ARCS site, telnet 203.161.182.113 80 and confirm that a response is given.

Back-out Procedures

If the services do not function as expected once the DNS is changed, then it will be necessary to change the hostname from auth13.ac3.edu.au to auth13.intersect.org.au. DNS changes at ac3 (from Step 5) must be undone. auth13.intersect.org.au should now be registered using Sirca's server DNS records in the intersect.org.au domain (both forward and reverse lookups).

Review

Schedule/Notification

cuong.hoang@arcs.org.au

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-005 | 26 Nov 2009 - 16:39 | ---+!! Migrate from MAMS to AAF Pilot

Description

Moving all ARCS federated services from the MAMS federation to the AAF Pilot Federation. This is due to MAMS federation being shutdown on the 30th November

Estimated Duration

5 Mins

Systems/Services Affected

Directly: slcs1.arcs.org.au and services.arcs.org.au hosts

Indirectly: AAF Shibboleth access to the following ARCS Services

* Data Fabric

* ARCS Compute Grid

Staff Responsible

Sam Morrison <sam.morrison at arcs.org.au>

Detailed Instructions

Telling puppet to move all shibboleth SP's in MAMS to AAF. Puppet will handle all configuration changes.

Testing Procedures

Test the following:

* Shibboleth Access to Data Fabric

* Grix Shibboleth Auth

* Grisu Shibboleth Auth

* services.arcs.org.au registration portal

* Use slick command line client

Back-out Procedures

Tell puppet to move it back to MAMS. NOTE: MAMS is being shut down so this is not an option.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-006 | 26 Nov 2009 - 22:10 | ---+!! Minor Grisu & Grix update

Description

Because of the switch to AAF Grix & Grisu need to be updated. In addition to providing new binaries there will also be a small update on the Grisu backend on grisu.vpac.org which adds support for pooled accounts.

Estimated Duration

10 seconds

Systems/Services Affected

Grisu, Grix

Staff Responsible

Markus Binsteiner

Detailed Instructions

Uploading of new Grix & Grisu binaries, exchanging of old grisu war with new one.

Testing Procedures

Manually testing of Grix & Grisu client applications.

Back-out Procedures

Re-instating old binaries

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-007 | 28 Nov 2009 - 10:58 | ---+!! Modifying memory usage on arcs-df.vpac.org and other minor updates

Description

Modifying the amount of memory used by various services on arcs-df.vpac.org

Estimated Duration

20 minutes

Systems/Services Affected

ARCS Data Fabric (arcs-df.vpac.org)

Staff Responsible

Pauline Mak

Detailed Instructions

Modify Postgres to use 1G of memory:

As root, stop iRODS services: 

service davis stop
service irods stop 

service postgres stop 

vim /opt/iRODS/Postgres/pgsql/data/postgresql.conf
Change the shared_buffer value size to 1G:
shared_buffers = 1024MB

Save the file and restart postgres

Modify the Davis setting in: 

vim /etc/default/davis 
-server -Xms768m -Xmx1024m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC

Then start iRODS and Davis 

service irods start
service davis start

Modify the TDS settings: 

vim /etc/defaults/tds
JAVA_OPTIONS="-server -Xmx256m -Xms256m -Djava.awt.headless=true"

Testing Procedures

Test that iRODS is functioning, and that davis is working by running litmus test.

Back-out Procedures

Revert davis to previous settings:

-server -Xms1024m -Xmx1562m -XX:+AggressiveHeap -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC

And postgres back to

shared_buffer=2048MB

Review

Schedule/Notification

Part of the outage on 28th November betwee 9-11 AEDT.

Note that TDS change has already applied 27th Nov at 2:30 AEDT.

Also modifying auto replication rule to use less retries:

acPostProcForPut|"$objPath" not like "/ARCS/trash/*"|delayExec(<PLUSET>1s</PLUSET><EF>30s DOUBLE UNTIL 5 TIMES</EF>,msiSysReplDataObj(ARCS-FABRIC,null),nop)|nop
acPostProcForCopy|"$objPath" not like "/ARCS/trash/*"|delayExec(<PLUSET>1s</PLUSET><EF>30s DOUBLE UNTIL 5 TIMES</EF>,msiSysReplDataObj(ARCS-FABRIC,null),nop)|nop

Tracking completion

Completed 28th November 2009, 10:58.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-008 | 22 Dec 2009 - 08:45 | ---+!! Upgrade iRODS Servers to Centos 5.4

Description

This Change is intended to improve the resilience of iRODS servers under heavy load conditions, and to reduce the likelihood of a kernel panic when writing to an XFS filesystem (particularly on a 32-bit system) under such conditions.

A standard set of 'yum' operations is used to download and install the necessary operating system patches and updated kernel. Two additional lines are then added to the file /etc/rc.local to ensure that the console doesn't clear during reboots; this is intended an an aid in crash diagnosis.

A reboot is then performed.

Estimated Duration

Sixty minutes.

Systems/Services Affected

Phase 1: srbdev.vpac.org, irodsdev.vpac.org and other development machines as appropriate .. no Production service outage.

Phase 2: arcs-df.vpac.org .. Production service outage required!

Phase 3: arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.qcif.edu.au, arcs-df.tpac.org.au, emii.resource.tpac.org.au, arcs-df.eresearchsa.edu.au, arcs-df.hpsc.csiro.au, datastore.hpsc.csiro.au .. Production service outage required!

Staff Responsible

VPAC: Graham Jenkins
TPAC: Pauline Mak

Pre-Change Tests

As the user rods issue the following commands: 

iput /etc/group
ils -l group
irm -f /etc/group

Also execute (as yourself) the iput/ils/irm tests from your workstation.

Create a myproxy identity using slix.jar and use it to upload, list and delete some files using a browser and a mounted filesystem.

Pre-Change Certificate Update (Phase 2 only)

As the root user, perform the following operations: 

cp /etc/httpd/conf.d/ssl.conf /var/tmp
vi /etc/httpd/conf.d/ssl.conf

Comment the 3 lines starting at line 109 

SSLCertificateFile /etc/grid-security/df-hostcert.pem  .. etc.

Uncomment the 3 lines starting at line 113 

#SSLCertificateFile /etc/httpd/ssl/hostcert.pem
Then:
service httpd restart

And finally .. start a new browser session and test Davis operation as above.

Detailed Instructions

As the root user, add the following lines to the end of the file: /etc/rc.local 

# Set the screen blank timeout to 0 mins
echo -e "\033[9;0]" >/dev/console

# Unblank the screen
echo -e "\033[13]" >/dev/console

Then perform the following steps: 

init 2  # Should shut down irods, postgres, etc.)
yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum --disablerepo=rpmforge update # As required
init 6

If the last 'yum update' directive fails with a dependency issue, you will need to add to the file /etc/yum.conf lines like the following so as to circumvent attempted updates of RPMs not sourced from standard repositories. 

# Local exclusions
exclude shibboleth\*
exclude log4shib\*

Then re-execute: 

yum update
init 6

Testing Procedures

Perform the following test (as the rods user) 

lsb_release -a #  Should show:       Release:  5.4

Then repeat the Pre-Change Tests as shown in an earlier paragraph.

Back-out Procedures

None available

Review

Schedule/Notification

Phase 2: 0800 - 0900 Tues. Dec. 22

Tracking completion

Phase 2: Completed!

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200911-009 | 30 Nov 2009 - 14:00 | ---+!! ARCS IdP? - Software upgrade

Description

Update the software the ARCS IdP? uses.

* Upgrade shibboleth from 2.1.2 -> 2.1.5

* Do general OS update

Estimated Duration

30 minutes

Systems/Services Affected

All users that use the ARCS IdP?

Staff Responsible

SamMorrison

Detailed Instructions

https://spaces.internet2.edu/display/SHIB2/IdP2Upgrade

Testing Procedures

Log into a shibboleth protected website with the ARCS IdP?

Back-out Procedures

Save old code base and revert to old WAR if needed

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-001 | 04 Dec 2009 - 09:06 | ---+!! Enabling permission cronjob on VPAC

Description

Installing a cron job that will change file permissions to allow the THREDDS Data Server to read files.

Estimated Duration

30 minutes

Systems/Services Affected

arcs-df.vpac.org. No outage required

Staff Responsible

Pauline Mak

Detailed Instructions

Download script twice from SVN: 
cd $IRODS_HOME/server/bin/local 
wget http://projects.arcs.org.au/trac/systems/export/1250/trunk/dataFabricScripts/iRODS/utils/opendap_acl.sh

Modify one script for TCMIP (opendap_acl_tcmip.sh) and another for the PUBLUMW (opendap_acl_publumw.sh) group. 

vaultDir=/data/Vault 
user=jetty

Modify the tree variable like so: 

tree="ARCS projects TCMIP"
tree="ARCS projects PUBLUMW"

The add the scripts to the rod user's crontab: 

#OPeNDAP Rules
0 * * * * /opt/iRODS/iRODS/server/bin/local/opendap_acl_tcmip.sh >/dev/null 2>&1 || :
0 * * * * /opt/iRODS/iRODS/server/bin/local/opendap_acl_publumw.sh >/dev/null 2>&1 || :

Testing Procedures

Run the cron job and check the permissions by using getfacl

Back-out Procedures

Remove scripts from the iRODS home directory and also remove the scripts from the rods user's crontab.

Review

Schedule/Notification

Tracking completion

Completed 4th December 2009.

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-002 | 15 Dec 2009 - 16:31 | ---+!! Updating TDS to version 4.1 on iVEC

Description

Updating TDS to the next version because of caching issue and also WMS does not correctly handle fill values for colour scale range (which are fixed in 4.1). Note that this is still an alpha release.

Estimated Duration

30 minutes

Systems/Services Affected

opendap-ivec.arcs.org.au

Staff Responsible

Pauline Mak

Detailed Instructions

Download the latest version from Unidata: 

cd /tmp
wget ftp://ftp.unidata.ucar.edu/pub/thredds/4.1/thredds.war

Install new version:

service tds stop

unzip thredds.war tds 

mv /opt/tds/webapps/thredds ~/thredds_4.0

mv tds /opt/tds/webapps/thredds

Change all of the configured catalogs to not use cache, i.e. add the folllowing attribute to each datasetScan element: 

 
cache="false"

Restart the server

Testing Procedures

Load up thredds in the browser to make sure data can be retrieved. Check the version number in the footer to make sure it is now 4.1+

Back-out Procedures

Stop the server

Remove new thredds webapp

Move ~/thredds_4.0 back into /opt/tds/webapps

Start the server

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-003 | 26 Feb 2010 - 15:59 | ---+!! NCI NGGUMS Upgrade

Description

NGGums upgrade for NCI NF. Involves update to VDT 1.10.1 and Centos 5.4

Estimated Duration

2 hours

Systems/Services Affected

Grid Gateway for terrawulf.anu.edu.au

Staff Responsible

Paul Warren

Detailed Instructions

Set up a new VM image as detailed at http://projects.arcs.org.au/trac/systems/wiki/HowTo/InstallNgGums

test, switch over ip addresses, turn off old machine.

Testing Procedures

Use a selection of Grid Certificates and VOs to test that mappings work correctly

Back-out Procedures

Reinstate old VM image

Review

Schedule/Notification

2009-12-14

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-004 | 04 Jan 2010 - 15:55 | ---+!! Installing TDS 4.1 stable and awstats for opendap-*.arcs.org.au

Description

Updating all TDS instances to the 4.1 stable and install awstat to keep usage statstics

Estimated Duration

2 hours

Systems/Services Affected

opendap-vpac.arcs.org.au, opendap-ivec.arcs.org.au, opendap-tpac.arcs.org.au, opendap-ersa.edu.au, opendap-intersect.edu.au, opendap-qcif.edu.au

The ARCS Data Fabric - as awstat requires a restart of the httpd server.

Staff Responsible

Pauline Mak

Detailed Instructions

Modify the Apache log in /etc/logrotate.d/httpd with the following text: 

/var/log/httpd/*log {
    compress
    missingok
    notifempty
    rotate 2
    sharedscripts
    size=1M
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Then as root, install awstats: 

yum install awstats

This will install awstat to /usr/share/awstats 

cd /usr/share/awstats/
tools/awstats_configure.pl
Then answer the quesiton as follows, substituion irodsdev.arcs.org.au with the host name of the server 

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/httpd/conf/httpd.conf

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> irodsdev.arcs.org.au

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>       

-----> Create config file '/etc/awstats/awstats.irodsdev.arcs.org.au.conf'
 Config file /etc/awstats/awstats.irodsdev.arcs.org.au.conf created.

Then add the following to root's crontab:

#update all statstics on a web server, which may have more than one virtual host

* 0 * * * /usr/bin/perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl/awstats_updateall.pl now

Download the new TDS 4.1

Testing Procedures

Back-out Procedures

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-005 | 18 Dec 2009 - 11:36 | ---+!! Enabling PUBLIC folders for all users

Description

Creating new public folders that will allow users of the ARCS Data Fabric to share files with anyone.

Estimated Duration

1 hour

Systems/Services Affected

df.arcs.org.au, arcs-df.vpac.org

Staff Responsible

Pauline Mak, Graham Jenkins

Detailed Instructions

Stop davis

Install new createUser script

cd /opt/iRODS/iRODS/server/bin/cmd

wget http://projects.arcs.org.au/trac/systems/export/1274/trunk/dataFabricScripts/iRODS/utils/createUser.pl-v3.08

Install createInbox script under /usr/local/bin

wget http://projects.arcs.org.au/trac/systems/export/1274/trunk/dataFabricScripts/iRODS/utils/createInbox.sh

Run createInbox.sh -a to create a INBOX and PUBLIC box for every existing user

Download inbox rules script from subversion

Update arcs.irb to include new inbox rules

Modify Davis to make __PUBLIC a public directory

Restart davis

Testing Procedures

Tested on ngdata-dev.

Login through davis and upload a file in your public directory. Then start a new browser session and try and download the file without logging into the data fabric.

Test putting in a file in another user's inbox, and check that once uploaded that you can no longer see it and that the other user can read and modify the file.

Back-out Procedures

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote200912-006 | 22 Dec 2009 - 13:16 | ---+!! UPDATE CERTIFICATES ON ERSA.EDU.AU HOSTED SERVICES

Description

Update certificates on the systems listed below to ensure continuity of service.

Estimated Duration

1 hour

Systems/Services Affected

arcs-df.ersa.edu.au sakai.arcs.org.au idp.ersa.edu.au authsvc.ersa.edu.au

Staff Responsible

David Logan

Detailed Instructions

On arcs-df.ersa.edu.au

Backup existing certificates and keys

/etc/grid-security/hostcert.pem

/etc/grid-security/hostkey.pem

/etc/grid-security/IPS-IPSCABUNDLE.CRT

Copy the following files to the following locations

/root/startssl_root_certs/arcs-df.ersa.edu.au_req.pem /etc/grid-security/hostcert.pem

/root/startssl_root_certs/hostkey.pem /etc/grid-security/hostkey.pem

/root/startssl_root_certs/sub.class1.server.ca.pem /etc/grid-security/sub.class1.server.ca.pem

/root/startssl_root_certs/ca.pem /etc/grid-security/ca.pem

Modify the following parameters in the /etc/httpd/conf.d/ssl.conf file

SSLCertificateChainFile? /etc/grid-security/sub.class1.server.ca.pem

SSLCACertificateFile? /etc/grid-security/ca.pem

# service httpd restart

***************************************************************************

on server authsvc.ersa.edu.au

Backup existing certificates and keys

/usr/local/ssl/private/authsvc.eresearchsa.edu.au.crt

/usr/local/ssl/certs/authsvc.eresearch.edu.au.key

/usr/local/ssl/private/IPS-IPSCABUNDLE.CRT

Copy the following files to the following locations

/root/startssl_root_certs/authsvc.ersa.edu.au_req.pem /usr/local/ssl/private/authsvc.eresearchsa.edu.au.crt

/root/startssl_root_certs/hostkey.pem /usr/local/ssl/certs/authsvc.eresearchsa.edu.au.key

/root/startssl_root_certs/sub.class1.server.ca.pem /usr/local/ssl/private/sub.class1.server.ca.pem

/root/startssl_root_certs/ca.pem /usr/local/ssl/private//ca.pem

Modify the following parameters in the /etc/httpd/conf.d/ssl.conf file

SSLCertificateChainFile? /usr/local/ssl/private/sub.class1.server.ca.pem

SSLCACertificateFile? /usr/local/ssl/private/ca.pem

# service httpd restart

***************************************************************************

on server idp.ersa.edu.au

Backup existing certificates and keys

/etc/pki/tls/certs/localhost.crt

/etc/pki/tls/private/localhost.key

/etc/pki/tls/certs/IPS-IPSCABUNDLE.crt

Copy the following files to the following locations

/root/startssl_root_certs/idp.ersa.edu.au_req.pem /etc/pki/tls/certs/localhost.crt

/root/startssl_root_certs/hostkey.pem /etc/pki/tls/private/localhost.key

/root/startssl_root_certs/sub.class1.server.ca.pem /etc/pki/tls/certs/sub.class1.server.ca.pem

/root/startssl_root_certs/ca.pem /etc/pki/tls/certs/ca.pem

Modify the following parameters in the /etc/httpd/conf.d/ssl.conf file

SSLCertificateChainFile? /etc/pki/tls/certs/sub.class1.server.ca.pem

SSLCACertificateFile? /etc/pki/tls/certs/ca.pem

Modify the following parameters in the /etc/httpd/conf.d/ssl-federation.conf file

Remove line SSLCACertificatePath?

Replace with SSLCertificateChainFile? /etc/pki/tls/certs/sub.class1.server.ca.pem

Add SSLCACertificateFile? /etc/pki/tls/certs/ca.pem

# service httpd restart

***************************************************************************

on server sakai.arcs.org.au

Backup existing certificates and keys

/etc/httpd/ssl/sakai.arcs.org.au_20090325_100958.crt

/etc/httpd/ssl/sakai.arcs.org.au_20090325_100958.key

/etc/httpd/ssl/IPS-IPSCABUNDLE.CRT

Copy the following files to the following locations

/root/startssl_root_certs/sakai.ersa.edu.au_req.pem /etc/httpd/ssl/sakai.arcs.org.au.crt

/root/startssl_root_certs/hostkey.pem /etc/httpd/ssl/sakai.arcs.org.au.key

/root/startssl_root_certs/sub.class1.server.ca.pem /etc/httpd/ssl/sub.class1.server.ca.pem

/root/startssl_root_certs/ca.pem /etc/httpd/sslca.pem

Modify the following parameters in the /etc/httpd/conf.d/ssl.conf file

SSLCertificateFile? /etc/httpd/ssl/sakai.arcs.org.au.crt

SSLCertificateKeyFile? /etc/httpd/ssl/sakai.arcs.org.au.key

SSLCertificateChainFile? /etc/httpd/ssl/sub.class1.server.ca.pem

SSLCACertificateFile? /etc/httpd/ssl/ca.pem

# service httpd restart

Testing Procedures

Ensure web services can be accessed without issue

Back-out Procedures

Copy back keys and certificates backed up in above procedure

Review

Log to each service and ensure it is working as planned.

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201001-001 | 20 Jan 2010 - 17:23 | ---+!! Migrate ESSCC VMs to Dell Xen host

Description

Migrating ESSCC production VMs to Dell Xen host that is under maintenance contract, and eventually free up the rack space taken up by IBM x346 server.

Estimated Duration

Less than 2 hours to migrate VMs and reboot Dell Xen host. Allocate 4 hours to allow for any unexpected scenarios.

Systems/Services Affected

All UQ and UQ ESSCC VMs:

ng2.hpcu.uq.edu.au

nggums.hpcu.uq.edu.au

ngdata.hpcu.uq.edu.au

ngmds.hpcu.uq.edu.au

myproxy1.arcs.org.au

ng2.esscc.uq.edu.au

nggums.esscc.uq.edu.au

cg-gw.nravs.uq.edu.au will not be able to accept jobs due to nggums.hpcu.uq.edu.au downtime.

Staff Responsible

WillHsu

Detailed Instructions

Shutdown all ESSCC VMs, copy/migrate VM files to Dell Xen host.

Relocate ESSCC network connection from IBM Xen host to Dell Xen host.

Configure Dell Xen host for extra network connection, shutdown all UQ VMs and reboot.

Starts UQ and ESSCC VMs and test with job submissions.

Testing Procedures

submit some test jobs from globus client to ng2

http://wiki.arcs.org.au/bin/view/APACgrid/TestSuite

Back-out Procedures

Relocate ESSCC network connection from Dell Xen host back to IBM Xen host.

Start ESSCC VMs from IBM Xen host.

Revert changes to Dell Xen host and reboot.

Review

Schedule/Notification

Notification sent on 12/Jan/2010 to community, developers, techstaff

Tracking completion

100% complete

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201001-002 | 13 Jan 2010 - 12:53 | ---+!! Migrate BCEES VM to an LVM Based VM

Description

Migrating the BCEES VM from a file base to an LVM base. Also increasing the disk space to 16GB due to growing SQL database

Estimated Duration

1 Hour

Systems/Services Affected

bcees.hpsc.csiro.au, https://bcees.org.au

Staff Responsible

Joel Ludbey-Bruhwel

Detailed Instructions

Shutdown bcees VM

Create same size logical volume to migrate to

sudo lvcreate -L 8G -n bcees vg0

DD filesystem to logical volume

sudo dd if=/xen_guests/bcees.img of=/dev/vg0/bcees

Make relevant change to xen config file /etc/xen/bcees.cfg

Start up VM and test

Shutdown VM again and increase disk space by 8GB

sudo lvextend -L +8G /dev/vg0/bcees

Start up VM, test and archive old disk images if everything works

Testing Procedures

SSH to Host

Check https://bcees.org.au

Upload some test files to msql database via https://bcees.org.au

Back-out Procedures

Restore backed up /etc/xen/bcees.cfg and reboot file based VM

Review

Schedule/Notification

Have already notified Troy Sadkowsky of this and he is notifying the relevant users of bcees.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201001-003 | 05 Feb 2010 - 11:39 | ---+ Deploying Access Service Integration to the ARCS Data Fabric

Description

Estimated Duration

3 hour

Systems/Services Affected

df.arcs.org.au

Staff Responsible

Pauline Mak

Detailed Instructions

__It is assumed that the Access Service has been installed at this point.

Backup the Database

(Based on previous change note: http://wiki.arcs.org.au/bin/view/Main/ChangeNote200908-001)

On df.arcs.org.au, shut down services: 

service httpd stop  
service davis stop 
service irods stop

On all ARCS DF hosts, shut down iRODS:

df.arcs.org.au, arcs-df.sf.utas.edu.au, arcs-df.ivec.org, arcs-df.ac3.edu.au, arcs-df.eresearchsa.edu.au, arcs-df.hpcu.uq.edu.au, arcs-df.hpsc.csiro.au 

service irods stop

Backup ICAT with pgdump on df.arcs.org.au 

mkdir /tmp/Backups  pg_dump ICAT > /tmp/Backups/ICAT_DUMP_BEFORE_AS_DF

Take full ICAT backup on df.arcs.org.au 

service postgres stop su - rods cd ~/DB/pgsql tar -cf /tmp/Backups/data.tar data/

Start iRODSon all hosts 

service irods start

Ingest Users into the Access Service

Export existing users details:

for U in `iadmin lu` ; do iadmin lu $U; echo; done > df_users.txt

Get the list of ARCS IdP? users, extract data, then run match script.

Double check that the number of users in df_users.txt matches the number of users in the CSV (minus users like rods)

Remove Anh and Pauline from export file

Go to AS admin interface and upload the CSV file.

Update createUser script

Update the arcs Rules file in subversion to no longer call the createUser script, by removing line 4. Commit the change.

Add sync user script

As the root user, Check that perl-XML-XPath amd perl-Crypt-SSLeay are installed, if not: 

yum install perl-Crypt-SSLeay perl-XML-XPath

Then as the rods user, download the script: 

cd /usr/local/bin 
wget -N http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/utils/syncUsers.pl

Update the crontab to run this once every 5 minutes: 

*/5 * * * * ( . /etc/profile.d/irods.sh; /usr/local/bin/syncUsers.pl arcs-data@lists.arcs.org.au ) >/dev/null 2>&1 || :

DO NOT RESTART IRODS SERVERS YET.

Update certificate on iVEC

The host certificate (and iRODS host certificate) will need to be updated on iVEC.

Restart iRODS servers

Update Davis to 0.8.3 

service davis stop cd /tmp    
wget http://webdavis.googlecode.com/files/davis-0.8.3.tar.gz 
tar -zxvf davis-0.8.3.tar.gz  
mv davis-0.8.3 /opt/davis  
chown -R davis:davis /opt/davis/davis-0.8.3   
cd /opt/davis 
rm davis  
ln -s /opt/davis/davis-0.8.3 davis  
cd /opt/davis/davis/bin sh davis-configure.sh
Then modify the davis configuration file under /opt/davis/etc/host-local.configuration: 
default-idp=arcs 
arcs-myproxy-server=mp-ca.arcs.org.au
authClass=au.org.arcs.davis.ARCSAuthorizationProcessor

Update Davis to use the slcs1 server. Modify the file: 

vi /opt/davis/davis/webapps/root/WEB-INF/classes/slcs-client.properties

The change: 

slcs.server=https://slcs1.arcs.org.au/SLCS/login

Make sure davis-organisation does not contain settings for the dynamic objects Add in MyProxy? CA bundle to arcs-df.vpac.org. The certificate and signing policies should go into /etc/grid-security/certificates

Get the certificates from ??? Update vdt update script to include this new CA: 

vim /opt/vdt/vdt/etc/vdt-update-certs.conf

And add in the following lines: 

include=/etc/grid-security/157c0362.0 
include /etc/grid-security/157c0362.signing_policy

Then restart Davis 

service davis start 
service httpd start

Testing Procedures

*This is very important* Test that existing login (using SLCS) continues to work. Run through items 2-4 from the Data Fabric/Access Service Integration plan: https://projects.arcs.org.au/trac/systems/wiki/AuthServices/DataFabricIntegration

Back-out Procedures

Restore Data Fabric database: 

cp -r /opt/iRODS-2.0v/Postgres/pgsql/data /tmp/data_bkp  
service postgres  start dropdb ICAT  createdb ICAT  psql ICAT < /tmp/ICAT_DUMP_BEFORE_AS_DF

Revert Davis to version 0.8.2 

rm /opt/davis/davis cd /opt/davis/ ln -s /opt/davis/davis-0.8.2 davis

Use old createUser script 

cd /opt/iRODS-2.1v/iRODS/server/bin/cmd 
mv createUser-beforeASDF createUser

Remove syncUsers.pl from rod user's cron job Insert the createUser rule in arcs.irb: 

acGetUserByDN(*arg,*OUT)||msiExecCmd(createUser,'"*arg"',null,null,null,*OUT)|nop 
 Commit changes back to subversion

Review

Schedule/Notification

Scheduled for Friday evening at 5pm Eastern time.

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201001-004 | 20 Jan 2010 - 14:30 | ---+!! TITLE

Enhancement of Activity Report Program on ARCS IDP Server

Description

User activity on idp.arcs.org.au is recorded in a database on db-2.arcs.org.au for later analysis by ARCS managment staff. This information is used in securing on-going funding.

Information is gathered through periodic (hourly) execution of the 'ShibReport.sh' program located in '/usr/local/bin'. This program parses the most recent log files, and generates usage records in MySQL tables which contain username and source information.

It was observed that log files can roll at indeterminate times, and to cover this case, earlier incarnations of the program were modified so as to parse all lines in the three most recent log files. The usage level has now grown to such an extent that a single invocation of the program can run for several minutes.

This change involves replacement of that program by one which parses only the lines in log files which changed in the previous 125 minutes.

Estimated Duration

This Change involves replacement and testing of a single program. It is of 60 minutes duration.

Systems/Services Affected

idp.arc.org.au

Staff Responsible

Graham Jenkins

Detailed Instructions

On idp.arc.org.au as the 'root' user, do: 

cd /usr/local/bin 
cp ShibReport.sh ShibReport.sh.20100120

Replace it with attached version.

Testing Procedures

Execute the script using the command shown in 'root' crontab.

On a workstation with appropriate query access. do:

select * from idp_stats_ip where id like "20100120%";

select * from idp_stats_user where id like "20100120%"

Ensure that recent activity is shown by both queries.

Back-out Procedures

Restore and test the orginal version of the program.

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201002-001 | 22 Feb 2010 - 13:06 | ---+!! Intersect - 2 Hour Network Outages Starting 9:00am on Sunday 21 February 2010 and Sunday 28 February 2010

Description

This change is to allow Intersect's hosting company AC3 to perform Fiber Link upgrades.

Estimated Duration

2 hours (21 Feb) + 2 hours (28 Feb)

Systems/Services Affected

All services provided by Intersect will be unavailable:

projects.arcs.org.au - Trac and Subversion sites. The standby will also be unavailable  
arcs-df.ac3.edu.au All Authorisation services (auth01-auth16).ac3.edu.au
ng2.ac3.edu.au hpc-ra.intersect.org.au
idp.ac3.edu.au idp.intersect.org.au
nggums.ac3.edu.au  
   

Staff Responsible

Simon Yin - simon.yin@arcs.org.au

Detailed Instructions

No physical work on the machines - all activity is conducted by 3rd parties through AC3.

Testing Procedures

A sanity check of the affected sites will be performed following the completion of the work.

Back-out Procedures

None

Review

Schedule/Notification

Notification by email:

For ARCS Internal Projects  
  techstaff@arcs.org.au, developers@arcs.org.au
For cawcr  
  A.Moise@bom.gov.au, Arnold.Sullivan@csiro.au, b.hu@bom.gov.au, Craig.Heady@csiro.au, Damien.Irving@csiro.au, david.kent@csiro.au, Didier.Monselesan@csiro.au, i.macadam@unsw.edu.au, J.Arblaster@bom.gov.au, j.sisson@bom.gov.au, Janice.Bathols@csiro.au, John.Clarke@csiro.au, Julian.O'grady@csiro.au, l.hanson@bom.gov.au, L.Rikus@bom.gov.au, Lauren.Stevens@csiro.au, Les.Muir@csiro.au, Martin.Dix@csiro.au, Peter.Vanrensch@csiro.au, Sarah.Perkins@csiro.au, Skye.Platten@csiro.au, Stacey.Osbrough@csiro.au, tim.erwin@csiro.au, Tony.Rafter@csiro.au, Will.Thurston@csiro.au
For geosciml  
  a.tellez-arenas@brgm.fr, ben.caradoc-davies@csiro.au, eric.boisvert@rncan-nrcan.gc.ca, ryan.fraser@csiro.au, trd@bgs.ac.uk
For ciomp  
  bkgalton@utas.edu.au, frank.colberg@utas.edu.au, john.hunter@utas.edu.au, kate@arsc.edu, msd@ccpo.odu.edu, tore.hattermann@npolar.no
For hollowworld  
  See Fullmoon
For sggc  
  Terry.Rankine@csiro.au
For portal  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, aron.abrook@qut.edu.au
For mgplot  
  frank.colberg@utas.edu.au
For mem  
  chris.jackett@csiro.au, peter.turner@csiro.au, jenny.lovell@csiro.au, r.williams@utas.edu.au
For access  
 

martin.dix@csiro.au, l.logan@bom.gov.au, mrezny@sgi.com

For macddap  
  jason.lohrey@arcitecta.com, R.Williams@utas.edu.au, paolap@utas.edu.au, pblain@postoffice.utas.edu.au
For jmoleditor  
  joachim.mai@intersect.org.au, Martin.Thompson@unsw.edu.au
For gsub  
  sam.moskwa@sapac.edu.au (defunct)
For grisu  
  y.halytskyy@auckland.ac.nz
For Generic Grid Client  
  ryan.fraser@csiro.au, Terry.Rankine@csiro.au, ronald@ivec.org, vladimir.mencl@canterbury.ac.nz
For DataMINX?  
  p.turner@chem.usyd.edu.au, alexa@intersect.org.au, andrey@intersect.org.au, carlos@intersect.org.au, Doug.duBoulay@gmail.com, xiao.wang@stfc.ac.uk, nguyen_h@chem.usyd.edu.au, gerson.galang@versi.edu.au
For CAPSnav  
  Edward.King@csiro.au, Leon.Majewski@bom.gov.au, Peter.Turner@csiro.au
For aodaac  
  ken.suber@csiro.au, Edward.King@csiro.au, matt.paget@csiro.au
For pht3d  
  Janek.Greskowiak@csiro.au, aaron.mcdonough@csiro.au, vincent.post@falw.vu.nl
For podd  
  gkam08@gmail.com, liyf@itee.uq.edu.au, james.eddes@adelaide.edu.au, jane@itee.uq.edu.au, ben.joseph@utas.edu.au, f.davies@uq.edu.au, kai.xu@anu.edu.au, xueqin.wang@csiro.au, paul.coddington@arcs.org.au, andrew.treloar@ands.org.au, Adrienne.McKenzie@anu.edu.au, Donald.Hobern@csiro.au, Robert.Furbank@csiro.au, joanne.evans@unimelb.edu.au, xavier.sirault@csiro.au
For Fullmoon  
  simon.cox@jrc.ec.europa.eu, ryan.fraser@csiro.au, Alexandre.Robin@spotimage.fr, bryan.lawrence@stfc.ac.uk, eboisver@nrcan.gc.ca, Francois.Letourneau@RNCan-NRCan.gc.ca, Jacek.Radajewski@csiro.au, johannes.echterhoff@igsi.eu, Jonathan.Yu@csiro.au, mggr@pml.ac.uk, pavel.golodoniuc@csiro.au, Peter.Taylor@csiro.au, pcd@ecosystem.com, Rick.Meng@csiro.au, rob.atkinson@csiro.au, robert.woodcock@csiro.au, rocky@cc.gatech.edu, stefan.hansen@lisasoft.com, oliver.raymond@ga.gov.au, Bruce.Simons@dpi.vic.gov.au
   
   
   
   
   
   

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201002-002 | 11 Feb 2010 - 15:30 | ---+!! Cron Job for Moving IMOS-ANFOG Staging Data

Description

This note describes the steps necessary to create a cron job for moving IMOS-ANFOG staging data to archive, opendap and public directories.

Estimated Duration

10 minutes

Systems/Services Affected

Only iVEC DF machine: arcs-df.ivec.org

Staff Responsible

KaiLu

Detailed Instructions

As user rods:

run:

cd /opt/iRODS/iRODS/server/bin/local

wget http://projects.arcs.org.au/svn/systems/trunk/dataFabricScripts/iRODS/utils/ANFOG_Data_Moving.sh/?format=raw -O ANFOG_Data_Moving.sh

chmod +x ANFOG_Data_Moving.sh

crontab -e and adding the following line:

0 23 * * * /opt/iRODS/iRODS/server/bin/local/ANFOG_Data_Moving.sh >> /opt/iRODS/iRODS/server/log/ANFOG-Moving-Data 2>&1

The cron job will be run once per day

Testing Procedures

As user rods:

run:

/opt/iRODS/iRODS/server/bin/local/ANFOG_Data_Moving.sh >> /opt/iRODS/iRODS/server/log/ANFOG-Moving-Data 2>&1

and check if:

the data files has been moved from directory staging into the directoties: archive, opendap and public

Back-out Procedures

As user rods:

run:

crontab -e and remove the following line:

0 23 * * * /opt/iRODS/iRODS/server/bin/local/ANFOG_Data_Moving.sh >> /opt/iRODS/iRODS/server/log/ANFOG-Moving-Data 2>&1

rm -f /opt/iRODS/iRODS/server/bin/local/ANFOG_Data_Moving.sh

rm -f /opt/iRODS/iRODS/server/log/ANFOG-Moving-Data

Review

Schedule/Notification

Tracking completion

Done. -- KaiLu - 10 Feb 2010

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201002-003 | 19 Feb 2010 - 15:05 | ---+!! VPAC Database service downtime

Description

VPAC DB server is being hardened

Estimated Duration

1 hour although actual time should be approx. 10 mins

Systems/Services Affected

RT web interface (note all rt emails will be queued on mail server so no data lost)

ARCS IdP? Registration

services.arcs.org.au website

status.arcs.org.au website

code.arcs.org.au - Some services will be unavailable (hudson and gitorious will be unaffected)

VTiger

Staff Responsible

SamMorrison - VPAC

Detailed Instructions

This email is to notify you of a scheduled VPAC system change:

System affected: VPAC Database Server; VPAC website; RT (helpdesk software); Jabber (chat software); Mailing Lists; Placard; Portals: AutoCRC? , BushfireCRC? , Co2CRC? , Ewater, RMIT, Project Management Portal; Timesheet.

Reason: Upgrading Database Server for increased stability and redundancy
Date of change: 26 February 2010
Time of change: 17:00 AEDT
Duration of outage: 1 hour

Impact: No functionality changes, no-data will be lost, transactions will be locked for the duration of the change over.
Notes: See Laszlo Kun or Brett Pemberton for details.


For further information email help@vpac.org or telephone (03) 9925 4410

Regards,
VPAC Systems Department

Testing Procedures

Visit websites

Back-out Procedures

Can't as rely on VPAC DB

Review

Schedule/Notification

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201003-001 | 10 Mar 2010 - 12:36 | ---+!! Reboot ARCS Sakai Server

Description

Reboot the ARCS Sakai VM on arcturus.ersa.edu.au to allow an lvextend operation and resize2fs of the root partition as it has now reached 83% capacity and is generating warnings on an ongoing basis.

Estimated Duration

30 minutes

Systems/Services Affected

ARCS Sakai server arcssakai.ersa.edu.au

Staff Responsible

David Logan

Detailed Instructions

Take a copy of the arcssakai-root lv prior to anything changing

On arcssakai

# /etc/shutdown

On Arcturus

# lvcreate -L 6G -n arcssakai-root-backup VolGroup00?

# dd if=/dev/VolGroup00/arcssakai-root of=/dev/VolGroup00/arcssakai-root-backup

On Arcturus (this extends the lv)

# lvextend +2G /dev/VolGroup00/arcssakai-root

# xm create arcssakai (This will recreate the filesystem at the new size and allow the resize2fs to take place)

On arcssakai (this extends the actual ext3 filesystem to the maximum size of the )

# resize2fs /dev/sda1 lvcreate -L 6G -n arcssakai-root-backup VolGroup00?

Testing Procedures

Use df to ensure new / partition is correct size

Back-out Procedures

Recreate the old boot logical volume by using the dd command

On Arcturus

# xm shutdown arcssakai 1010

# lvremove VolGroup00? /arcssakai-root

# lvcreate -L 6G -n arcssakai-root VolGroup00?

# dd if=/dev/VolGroup00/arcssakai-root-backup of=/dev/VolGroup00/arcssakai-root

# xm create arcssakai

Review

Schedule/Notification

All users of sakai need to be notified . Scheduled outage to be for 30 minutes starting at 5:00pm ACDT 12 March 201. This will be done just before the University of Adelaide's scheduled power outage (refer to ChangeNote? 201003-002).

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNote201003-002 | 10 Mar 2010 - 12:41 | ---+!! Power outage at eRSA

Power Outage at eRSA - 12th - 15th of March 2010

Description

The power will be disconnected to the equipment room in the Physics Building at the University of Adelaide over the period described above for maintenance to the power systems.

Estimated Duration

63.5 hours

Systems/Services Affected

idp
mds
imos0
imos1
imos2
arcturus
arcssakai
ng2
nggums
ngdata
inca
imos
datadev
arcs-df

Staff Responsible

David Logan

Detailed Instructions

Shut down all VM's and then hosts to ensure system integrity

Testing Procedures

Back-out Procedures

N/A

Review

Schedule/Notification

From 5:00pm 12th of March to 8:30am 15th of March 2010, all times Australian Central Daylight Time

Tracking completion

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

| | ChangeNotes | 30 Jun 2008 - 16:59 | ---+ Change Notes Dynamic list of change notes for ChangeControl policy.

ALERT! This does not show current state yet. Testing in ChangeNotesTest without much luck. Will probably split the search into an APPROVED/not-completed search and a non-APPROVED search.

Topic Updated Description Impact State
ChangeNote200806-002 30 Jun 2008 - 17:10 arcs.org.au downtime for 1 hour. 2C - High -
ChangeNote200806-003 07 Jul 2008 - 17:06 SRB User Federation Cronjob 1B - Low -
ChangeNote200806-004 12 Sep 2008 - 11:40 Gridpulse/MDS SIP 1B - Low -
ChangeNote200806-005 28 Jul 2008 - 15:14 Move MyProxy 1B - Low -
ChangeNote200807-001 14 Jul 2008 - 20:52 output timestamp by Szonesync 1B - Low -
ChangeNote200807-002 25 Jul 2008 - 15:16 Add Will's GTest cert to user 'inca' 1B - Low -
ChangeNote200807-003 27 Nov 2008 - 14:38 Cron Job for the SRB Log Rotation 1B - Low -
ChangeNote200808-002 07 Aug 2008 - 10:11 ARCS RT user account changes Low -
ChangeNote200808-003 10 Sep 2008 - 11:48 Myproxy name change   -
ChangeNote200808-005 12 Sep 2008 - 14:09 Host OS upgrade for apacgrdigw.hpsc.csiro.au 3A - Medium -
ChangeNote200808-006 10 Sep 2008 - 12:21 create new ng2 for ESSCC 3A - Medium -
ChangeNote200809-002 27 Nov 2008 - 15:09 Improve MCAT performance Medium -
ChangeNote200809-003 19 Aug 2009 - 13:22 SRB auto account creation 1B - Low -
ChangeNote200811-001 27 Nov 2008 - 15:10 slcs1 certs for auto account 1B - Low -
ChangeNote200811-002 02 Dec 2008 - 10:32 Data Fabric usage scripts 1B - Low -
ChangeNote200811-003 02 Dec 2008 - 12:16 Davis deployment 1B - Low -
ChangeNote200811-004 02 Dec 2008 - 10:41 Update dnToUserDomain.py 1B - Low -
ChangeNote200812-002 22 Dec 2008 - 14:27 TWiki upgrade Medium (2B) -
ChangeNote200903-001 01 Apr 2009 - 03:28 Update configuration files on IdPs 2B -
ChangeNote200903-002 03 Apr 2009 - 09:47 Staff mail relocating to Google Apps High -
ChangeNote200904-001 11 Jun 2009 - 14:13 Plone service Migration 2D - Severe -
ChangeNote200905-001 07 May 2009 - 12:58 Migration of critical grid infrastructure virtual machines at VPAC Severe -
ChangeNote200905-002 14 May 2009 - 20:08 database server rebuild Low -
ChangeNote200905-003 18 May 2009 - 09:43 Change slcs1.arcs.org.au entityID   -
ChangeNote200905-004 29 May 2009 - 15:12 Update User Records None -
ChangeNote200905-005 26 May 2009 - 13:22 ARCS Mail server migration Severe -
ChangeNote200905-006 26 May 2009 - 13:20 wiki.arcs.org.au upgrade to TWiki 4.3.1 2B - Medium -
ChangeNote200905-007 28 May 2009 - 11:19 SLCS DSD 2B - Medium -
ChangeNote200905-008 09 Jun 2009 - 13:15 Running Davis as unprivileged davis user   -
ChangeNote200905-009 28 May 2009 - 11:29 GSI patch for iRODS   -
ChangeNote200905-010 05 Jun 2009 - 18:02     -
ChangeNote200905-011 28 May 2009 - 12:34 Upgrade Davis to 0.7.0   -
ChangeNote200905-012 08 Aug 2009 - 10:07 GSI Patch to Datafabric   -
ChangeNote200905-013 28 May 2009 - 15:34 VM Upgrade 2A - Low -
ChangeNote200906-001 02 Jun 2009 - 10:50     -
ChangeNote200906-002 03 Jun 2009 - 08:21 Update and migration of ARCS Sakai 3D- Severe -
ChangeNote200906-003 15 Jun 2009 - 11:15 Migration of Trac and Subversion to a new host + virtual machine 2D -
ChangeNote200906-004 17 Jun 2009 - 16:50 Upgrade Davis   -
ChangeNote200906-005 19 Jun 2009 - 09:15 iRODS 'gethostbyname' Problem Resolution None -
ChangeNote200906-006 26 Jun 2009 - 10:54 Upgrade Davis 0.7.2a   -
ChangeNote200906-007 23 Jun 2009 - 23:11 VM upgrade   -
ChangeNote200906-008 08 Aug 2009 - 10:10 Adding rules for IMOS collection   -
ChangeNote200906-009 01 Jul 2009 - 16:46 VM Upgrade   -
ChangeNote200906-010 18 Sep 2009 - 10:17     -
ChangeNote200906-011 08 Aug 2009 - 10:09 Rules update script for iRODS   -
ChangeNote200907-001 09 Jul 2009 - 12:59 Moving Master ICAT of DF from IVEC to VPAC and Implimenting Pgpool   -
ChangeNote200907-002 09 Jul 2009 - 09:20 VM Upgrade   -
ChangeNote200907-003 08 Aug 2009 - 10:08 Moving Davis from iVEC to VPAC   -
ChangeNote200907-004 02 Jul 2009 - 16:19 VM Upgrade   -
ChangeNote200907-005 21 Jul 2009 - 15:40 Migrate ANDS Plone site to VPAC 2A Low -
ChangeNote200907-006 08 Aug 2009 - 10:08 Update to createUser script   -
ChangeNote200907-007 19 Aug 2009 - 12:12 Installing AWSTATS   -
ChangeNote200907-008 03 Sep 2009 - 10:52 Installing MAMS level-1 SP   -
ChangeNote200907-009 18 Jul 2009 - 21:15 Grid VM upgrade at eRSA 3A - Medium -
ChangeNote200907-010 18 Jul 2009 - 20:58 Migration of unicast bridge   -
ChangeNote200907-011 28 Jul 2009 - 14:34 Enabling Postgres WAL Archving on DF low -
ChangeNote200907-012 11 Aug 2009 - 11:29 Grid VM upgrade at JCU 3A - Medium -
ChangeNote200907-013 25 Jul 2009 - 20:51 NG VM Upgrades 3A - Medium -
ChangeNote200907-014 01 Aug 2009 - 19:49 Changes to Grisu backend   -
ChangeNote200908-001 07 Aug 2009 - 15:52 Moving Datafabric ICAT from arcs-df.ivec.org to arcs-df.vpac.org Medium -
ChangeNote200908-002 06 Aug 2009 - 18:27 Grid VM upgrade at UQ/UQ ESSCC 3A - Medium -
ChangeNote200908-003 04 Sep 2009 - 10:32 Update arcs.org.au Plone site to update products/features 2A - Low -
ChangeNote200908-004 14 Sep 2009 - 11:10 Upgrading iRODS from 2.0.1 to 2.1 on DataFabric Medium -
ChangeNote200908-005 25 Aug 2009 - 11:38 Inca migration to Arcturus 2A - Low -
ChangeNote200908-006 20 Aug 2009 - 16:57     -
ChangeNote200908-007 03 Nov 2009 - 17:00 cron job for moving IMOS/ACORN staging data Low -
ChangeNote200908-008 26 Aug 2009 - 13:37 Hollowworld (follows Fullmoon) Low - 1A -
ChangeNote200908-009 25 Aug 2009 - 18:08 Inca 2.5 upgrade 3A - Medium -
ChangeNote200909-001 08 Sep 2009 - 13:05 ARCS Sakai Upgrade to Version 2.5.5 2D - Severe -
ChangeNote200909-002 18 Feb 2010 - 10:55 Davis change for iRODS 2.1   -
ChangeNote200909-003 09 Sep 2009 - 19:42 ARCS IdP? LDAP migration Large -
ChangeNote200909-004 15 Sep 2009 - 13:02 VM Upgrade   -
ChangeNote200909-005 18 Sep 2009 - 17:51 Fixing bug in ibun command Low -
ChangeNote200909-006 17 Sep 2009 - 17:23 Upgrade Davis to 0.8.0   -
ChangeNote200909-007 17 Sep 2009 - 17:12 slcs-server Low - 1A -
ChangeNote200909-008 29 Sep 2009 - 17:36 Upgrade of Intersect <-> AARNET switches Low -
ChangeNote200909-009 01 Oct 2009 - 11:28 Network changes at VPAC 3C - Severe -
ChangeNote200909-010 29 Sep 2009 - 10:17 Bug fix to ibun, extrat to any resource Medium -
ChangeNote200909-011 28 Sep 2009 - 12:04 Davis and SP integration   -
ChangeNote200910-001 06 Oct 2009 - 13:48   Low -
ChangeNote200910-002 20 Oct 2009 - 10:19 Intersect Data Fabric Network Upgrade To 1Gb/s +   -
ChangeNote200910-003 02 Nov 2009 - 11:23 MDS VM upgrade at UQ 3A - Medium -
ChangeNote200910-004 23 Oct 2009 - 09:09 SLCS Federation Move 2D -
ChangeNote200910-005 02 Nov 2009 - 11:20 EVO Server migration 2A - Low -
ChangeNote200910-006 02 Nov 2009 - 11:37 ARCS Wiki migration 2A - Low -
ChangeNote200910-007 22 Oct 2009 - 12:18 Attaching New Storage to arcs-df.ac3.edu.au Low -
ChangeNote200910-008 28 Oct 2009 - 09:24 Adding New Index to ICAT Database Low -
ChangeNote200910-009 28 Oct 2009 - 10:36     -
ChangeNote200911-001 06 Nov 2009 - 17:03     -
ChangeNote200911-002 19 Nov 2009 - 15:31 Migrate Plone Storage to blobs Medium -
ChangeNote200911-003 28 Nov 2009 - 10:57     -
ChangeNote200911-004 15 Dec 2009 - 15:34 16 IP addresses to be moved to Intersect's IP range 3A - Medium -
ChangeNote200911-005 26 Nov 2009 - 16:39 MAMS migration to AAF Large -
ChangeNote200911-006 26 Nov 2009 - 22:10 Grix & Grisu update Minor -
ChangeNote200911-007 28 Nov 2009 - 10:58     -
ChangeNote200911-008 22 Dec 2009 - 08:45     -
ChangeNote200911-009 30 Nov 2009 - 14:00 ARCS IdP? Shib upgrade Minimal -
ChangeNote200912-001 04 Dec 2009 - 09:06     -
ChangeNote200912-002 15 Dec 2009 - 16:31     -
ChangeNote200912-003 26 Feb 2010 - 15:59 NCI GUMS upgrade LOW -
ChangeNote200912-004 04 Jan 2010 - 15:55     -
ChangeNote200912-005 18 Dec 2009 - 11:36     -
ChangeNote200912-006 22 Dec 2009 - 13:16 Update certificates on ersa hosted ARCS services Low -
ChangeNote201001-001 20 Jan 2010 - 17:23 Migrate ESSCC VMs 3D - Severe -
ChangeNote201001-002 13 Jan 2010 - 12:53 Migrate bcees to VLM based VM and increase disk space Low -
ChangeNote201001-003 05 Feb 2010 - 11:39     -
ChangeNote201001-004 20 Jan 2010 - 14:30     -
ChangeNote201002-001 22 Feb 2010 - 13:06 Network Outage - All Services - Intersect High - 4 hours -
ChangeNote201002-002 11 Feb 2010 - 15:30 cron job for moving IMOS/ANFOG staging data Low -
ChangeNote201002-003 19 Feb 2010 - 15:05 VPAC DB Downtime Large but short -
ChangeNote201003-001 10 Mar 2010 - 12:36 Reboot ARCS Sakai server to increase root partition size Short outage -
ChangeNote201003-002 10 Mar 2010 - 12:41 Power Outage Severe -
ChangeNotes 30 Jun 2008 - 16:59     -
Number of topics: 113

See: VarSEARCH, FormattedSearch, TablePlugin

  • Hint: use ?raw=debug in URL to see raw topic text including metadata.

-- DanielCox - 23 Jun 2008 |

Number of topics: 113

| ChangeNote200806-001 | 01 Jul 2008 - 11:30 | Testing for ChangeNotesTest - trying to work out why WorkflowPlugin variables are not being expanded correctly in search results.

WF: ABC: %ABC%

Change Control

This document used for ARCS ChangeControl Policy. The contents of this section comes from ChangeNoteInclude.

State: -

History:

|

Number of topics: 1

Hint: use ?raw=debug in URL to see raw topic text including metadata.

Try form label field?

-- DanielCox - 30 Jun 2008

Edit | Attach | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 30 Jun 2008 - 16:52:49 - DanielCox
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback