Basic Xen Install

Responsible Person

GrahamJenkins, VPAC

Technology Summary and Contents

• CentOS 4.4
• Xen 3.0.3

Basic Installation, Creation of Domain 0

• If you have an IBM xSeries-346 server, you should boot from a recent (Jan 2006 or later) Update-Express CD (#1) and select all available updates. It will probably be necessary to reboot with the CD still loaded so that the update process completes satisfactorily. This is especially important for Xen 3.x.x installations.
• If you have an IBM ServeRAID card and 2 or more disks, install the root disks in positions 0 and 1, then boot from the (green) ServeRAID support disk, and use the wizard to mirror those disks in their entirety.
• If you wish to use kickstart, some configuration files are available at: http://gramps.ac3.com.au/ng/ .. these may not be up-to-date.
• Install Centos 4.4 (i386 version, not x86_64)
• Select "personal desktop", manually partition disk, sda1 / ext3 (4G), sda2 swap (512M), sda3 LVM (balance).
• Assign static IP, no SELinux, no firewall, customise, minimal install.
• Edit /etc/yum.repos.d/CentOS-Base.repo and enable the centosplus repository.
• rpm --import http://mirror.centos.org/centos/4/os/i386/RPM-GPG-KEY-centos4 - Note that rpm pays no attention to the $http_proxy environment variable, so if you're behind a proxy you'll need to fetch the key separately first (using wget).
• yum install gcc curl curl-devel zlib-devel
• yum install bridge-utils python-devel ncurses-devel ntp
• chkconfig --add ntpd; chkconfig ntpd on
• Edit /etc/ntp.conf so as to use your preferred timeserver(s).
• ntpdate {timeserver}; init 6 (if time was incorrect)
• service ntpd start
• mv /lib/tls /lib/tls.disabled
• mkdir /tmp/x && cd /tmp/x && wget http://bits.xensource.com/oss-xen/release/3.0.3-0/bin.tgz/xen-3.0.3_0-install-x86_32p.tgz
• cd dist
• sh ./install.sh
• depmod 2.6.16.29-xen
• mkinitrd /boot/initrd.img-2.6.16.29-xen 2.6.16.29-xen
• mkdir -p /var/run/xenstored
• vi /boot/grub/grub.conf
• Add this boot option as default, customizing your root partition, and the prefix for the files if necessary as follows:

   title Xen 3.0 / XenLinux 2.6
     kernel /boot/xen-3.gz dom0_mem=262144
     module /boot/vmlinuz-2.6.16.29-xen ro root=/dev/sda1
     module /boot/initrd.img-2.6.16.29-xen

• init 6

NIC Allocation

• By default, Domain0 uses eth0. In the interests of uniformity on IBM xSeries-346 machines, it is desirable that eth0 corresponds to NIC 1 on the motherboard. To bring this about, you may need to do something like this (to switch eth0 and eth2):
• ifdown eth0
• cd /etc/sysconfig/network-scripts
• cp ifcfg-eth[02] /tmp; cp ifcfg-eth2 ifcfg-eth0; cp /tmp/ifcfg-eth0 ifcfg-eth2
• vi ifcfg-eth[02] [switch names in "DEVICE" line]
• ifup eth0

Domain 0 Monitoring

• Copy the appropriate version of 'ipssend' from the (green) ServeRAID support disk to /usr/bin and ensure that is executable by root.
• Do: cd /etc/yum.repos.d && wget http://www.grid.apac.edu.au/repository/dist/APAC-Grid.repo
• Then do: yum install Gpulse Gbuild
• Ensure that your machine is able to send mail to the outside world. The easiest way is probably to send all mail through an existing SMTP gateway.
• yum install mailx postfix .. then in /etc/postfix/main.cf, un-comment 'myorigin = $myhostname'. If you are using an SMTP gateway then add relayhost = [your.smtp.gateway] in the appropriate section. Some sites may wish to use an alternative to postfix.
• For postfix, do: chkconfig postfix on; service postfix start .
• When executed in non-interactive mode, gridpulse.sh actually delays mailing the machine's status for a psuedo-random interval (up to 3 minutes) so as to reduce the impact of multiple simultaneous arrivals at the mail-server.

Domain Shutdown Actions

• By default, when Xen host shutdown occurs, the state of each guest domain will be saved as a large file in directory: /var/lib/xen/save
• This is probably not a good idea; you can change this behaviour by changing the XENDOMAINS_SAVE value in file: /etc/sysconfig/xendomains
• It is suggested that you change the value to "" so that guest domains are shut down rather than saved; this will ensure that guest domains come up with a correct time after Xen host startup

Creation of Other Domains (e.g. NG2)

• pvcreate /dev/sda3
• vgcreate VolumeGroup00 /dev/sda3
• lvcreate -L 16384M -n NG2Root? VolumeGroup00
• lvcreate -L 512M -n NG2Swap? VolumeGroup00
• mkswap /dev/VolumeGroup00/NG2Swap
• mkfs -t ext3 /dev/VolumeGroup00/NG2Root

Bootstrapping CentOS 4.4

• If you are using SELinux, disable it via boot options before you start.
• mount -t ext3 /dev/VolumeGroup00/NG2Root /srv/vm1
• /usr/local/sbin/BuildGuest.sh /srv/vm1 This will build a bootable machine which obtains its IP address via DHCP.
• Supply a root password when prompted. The new guest's root filesystem will then be unmounted.
• create a new xen config for the new VM NG2 (Insert your own value, similar to that shown, for the MAC address!):
• vi /etc/xen/NG2

kernel = "/boot/vmlinuz-2.6.16-xen"
ramdisk = "/boot/initrd.img-2.6.16-xen"
memory = 512  # Use 512 for NG2 machines, 256 for others
name = "NG2"
cpus = ""     # Leave to Xen to pick
disk = [ 'phy:VolumeGroup00/NG2Root,sda1,w',
         'phy:VolumeGroup00/NG2Swap,sda2,w']
root= "/dev/sda1"
extra = "ro selinux=0 3"
vif = [ 'mac=00:16:3e:00:00:22, bridge=xenbr0' ]   

• If this is your first guest, do: chkconfig --add xend; service xend start
• Start it up: xm create -c NG2
• If you receive a message like: "device 2049 (vbd) could not be connected", mount the filesystem again, reissue the above command and watch it fail, then unmount the filesystem.
• Login as root.
• Do: rpm --import http://mirror.centos.org/centos/4/os/i386/RPM-GPG-KEY-centos4 - Note that rpm pays no attention to the $http_proxy environment variable, so if you're behind a proxy you'll need to fetch the key separately first (using wget).
• Then: yum install postfix (or other mail program as used at your site)
• Edit /etc/yum.repos.d/CentOS-Base.repo and enable the centosplus repository.
• Edit /etc/ntp.conf and /etc/postfix/main.cf appropriately for your site.
• Also change network settings as required; files you might need to edit are /etc/sysconfig/network, /etc/sysconfig/network-scripts/ifcfg-eth0, /etc/hosts, /etc/resolv.conf and /etc/nsswitch.conf
• chkconfig --add ntpd; chkconfig ntpd on; service ntpd start; service postfix start If your site uses sendmail, perform whatever equivalent operations are appropriate.
• Link a timezone file to /etc/localtime e.g. rm /etc/localtime ; ln -s /usr/share/zoneinfo/Australia/Melbourne /etc/localtime
• cd /etc/yum.repos.d
• wget http://www.grid.apac.edu.au/repository/dist/APAC-Grid.repo
• Most gateway VMs will need a host certificate: HostCertRequestAPAC

Installing Complete Domain Images

• If you have a complete domain image, you can copy it to a mounted empty filesystem thus:
• cd /srv/vm1; tar xjpf /tmp/image.tbz
• Then create an appropriate Xen configuration file as above, create the essential devices, unmount the filesystem, and start the domain.

Starting/Stopping Domains Automatically

• To start/stop domains automatically during xen host boot/shutdown ..
• chkconfig --add xendomains; service xendomains start
• ln -s /etc/xen/NG2 /etc/xen/auto (for 'NG2')

Using Additional NICs

• To make the additional interfaces accessible, change the 'vif' line in your Xen guest configuration script(s) so that it looks something like:

     vif = [ 'mac=00:16:3e:00:00:30, bridge=xenbr0',
             'mac=00:16:3e:00:00:31, bridge=xenbr1' ]
     

• Also in directory /etc/xen/scripts do:

  mv network-bridge network-bridge.dist
  cat >network-bridge <<"EOF"
  #!/bin/sh
  /etc/xen/scripts/network-bridge.dist $1 netdev=eth0 bridge=xenbr0 vifnum=0
  /etc/xen/scripts/network-bridge.dist $1 netdev=eth1 bridge=xenbr1 vifnum=1
  EOF
  chmod a+xr network-bridge
     

• Then reboot.

Upgrading from an Older Version of Xen

• If you are upgrading from Xen 3.0.2, start by shutting down all guests, then do: chkconfig --del xendomains
• Upgrade whatever Xen host operating system you currently have in accordance with the steps shown on the CentOS website. After doing this, do: mv /lib/tls.disabled /lib/tls.disabled.OLD
• Save copies of the entire /boot and /etc/xen directories, then remove from /boot those symbolic links which point to xen kernels and images
• Proceed as shown in the "Basic Installation" paragraph above, starting at the wget step. Before rebooting, you should also perform the "Domain Shutdown Actions" and (if appropriate) "Using Additional NICs" steps
• After rebooting, copy the new kernel modules to the guest domains (e.g. as shown at XenMemory under the heading "Fixup Guest Kernel Modules")
• Start and check one guest, then shut it down
• Do: chkconfig --add xendomains, then: service xendomains start

Improving Network Performance

• The default CentOS system configuration can be changed as shown hereunder so as to enhance network performance. It has been found that there are some kernel-related page-allocation issues when this is done, and you may need to increase guest-machine memory (e.g. to 512M) to alleviate these.
• On the guest machine, add the following lines to: /etc/sysctl.conf

# Ref: http://www-didc.lbl.gov/TCP-tuning/linux.html
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 2500
net.ipv4.tcp_congestion_control=bic

• Then do: sysctl -p

How to shutdown and Remove a VM

• From xen root login: xm shutdown -w your_VM
  • If it doesn't want to work the nice way: xm destroy your_VM
• Remove VM from autostart list: rm /etc/xen/auto/your_VM
• Remove Root and Swap Logical Volumes. Check in /etc/xen/your_VM to see what the LV names are, then:
  lvremove VolumeGroup00/your_VMRoot && lvremove VolumeGroup00/your_VMSwap
• Remove configuration file: rm /etc/xen/your_VM

See Also

• XenSourceRPMs
• XenMemory
• NativePosixThreadLibrary

-- GrahamJenkins - 23 Apr 2007