Xen Memory

IMPORTANT: This procedure is not currently approved for any GatewayMachines.

Our GatewayMachines have 4GB RAM but this is not all available with the default Xen 3.0.2 install.

xm info shows 3328MB (short by 768MB).

xm dmesg shows a warning Truncating the memory map...

(XEN) WARNING: Only the first 4 GB of the physical memory map can be accessed
(XEN)          by Xen in 32-bit mode. Truncating the memory map...
(XEN) Physical RAM map:
...
(XEN) System RAM: 3327MB (3407264kB)
(XEN) Xen heap: 10MB (10532kB)
(XEN) Using scheduler: Simple EDF Scheduler (sedf)
(XEN) PAE disabled.

VM Allocation

We currently allocate 256M to each VM and most sites have at least: ng1, ng2, ngdata and ngportal. A total of only 1.25GB (including 256M for Dom0).

Java memory requirements are still being looked into, but to reduce swap - each VM that uses Java (not normally ngdata) should really have 512M. Total 2GB.

Running development/testing versions of each VM is not possible without the extra 768MB.

Note: With dynamic memory ballooning in Xen 3, it is not necessary to limit Dom0 memory usage in grub.conf. This appears to work but has not been tested in a long running situation (DC @SAPAC).

Solution

This can be fixed in x86-32 using Xen PAE support and requires a replacement hypervisor and linux kernel (for both Dom0 and all DomU). 64-bit Xen/Linux is not required.

Definition: PAE at Wikipedia

Shutdown or Save State

When Dom0 is shutdown it will either :-

• shutdown each VM properly
• OR (the default) save the state to disk and restore on next boot

There is unlikely to be enough disk space for the state of all VMs, especially if they are given more memory - and shutdown is NOT called for the VM when save fails!

Also, the saved state may not be suitable for running with a different version of the Xen hypervisor and will certainly not be using the new HIGHMEM64G? kernel.

Set XENDOMAINS_SAVE="" in /etc/sysconfig/xendomains to disable saving. If not always, then at least when rebooting after changing Xen/Linux kernel.

Backup First

Make a full copy of /boot to help if something goes wrong. Eg. cp -av /boot /boot.060904

With a complete copy (instead of a .tar), Grub will still load and it will be easy to use E to edit the kernel/initrd path.

Installation

Prebuilt RPMs (the result of the process below) are available from SAPAC here.

xen,xen-kernel are required and can be installed on a system that has been setup following XenInstall instructions.

• may need to uninstall lksctp-tools
• backup /boot
• edit /boot/grub/grub.conf to use xen-3.gz,vmlinuz-2.6-xen,initrd-2.6-xen.img
• see XenSourceRPMs for more detail

It is worth testing that the system works before VMs are started, so disable guest auto startup: chkconfig xendomains off

Edit guest config files (kernel/initrd) OR link old names to the new versions.

• vmlinuz-2.6.16-xen -> vmlinuz-2.6.16-xen3_86.1_rhel4.1
• initrd.img-2.6.16-xen - initrd-2.6.16-xen3_86.1_rhel4.1.img

Fixup Guest Kernel Modules

This copies the currently running kernel modules into each root LV.

yum install rsync
service xendomains stop
find /dev/VolumeGroup00/ -iname "*root" -exec sh -c "echo {}; \
  mount {} /srv/vm1 && rsync -auv /lib/modules/`uname -r` \
  /srv/vm1/lib/modules; umount /srv/vm1" \;

Restart Guest VMs: chkconfig xendomains on; service xendomains start

Testing Results

SAPAC have been using these PAE enabled RPMs without problems on an IBM machine identical to the GatewayMachines since 1Aug06. We are keen to modify our grid-gateway once this procedure is approved.

xm info shows 4096MB and there are no warnings in xm dmesg output.

Building the RPMs

There are many dependencies which are not normally needed so I setup a logical volume based on another machine and used chroot.

yum install make rpm-build
yum install zlib-devel curl-devel gcc-c++ python-devel ncurses-devel e2fsprogs-devel \
 tetex-latex tetex-dvips ghostscript transfig

Download: xen-3.0-x86_32-rhel4.1.src.tar 57MB (from XenSourceRPMs)

tar xvf xen-3.0-x86_32-rhel4.1.src.tar
rpm -ivh xen-3.0-x86_32-rhel4.1.src/xen-3.0.2.2-86.1_rhel4.1.src.rpm
cd /usr/src/redhat/SPECS
rpmbuild -bp xen.spec

1. Edit the %prep section of xen.spec to prevent removal/overwriting of extracted files in BUILD/ using -D -T options. ie. %setup -q -D -T -n ...
2. Edit ../BUILD/xen-3.0.2.2-86/Config.mk and set XEN_TARGET_X86_PAE=y
   • It should be possible to set this on the command line with rpmbuild but it fails to build i386 target
3. Edit ../BUILD/xen-3.0.2.2-86/buildconfigs/linux-defconfig_xen_x86_32. Uncomment CONFIG_HIGHMEM64G and enable.
4. Build xen (../RPMS/i386): rpmbuild -bb --target i386 xen.spec
5. Build the Linux kernel (../RPMS/i686): rpmbuild -bb --target i686 xen.spec

Other Prebuilt Binaries

Will @UQ pointed out that there are different versions of the binary installer which should already have high memory support.

xen-3.0.2-install-x86_32p.tgz is now documented for use in XenInstall. It works well on grid-gateway.sapac.edu.au since 6Oct06.

-- DanielCox - 04 Sep 2006