Native POSIX Thread Library

Definition: NPTL at Wikipedia

From the instructions at XenInstall, NPTL has been disabled on Dom0 and all DomU guests by renaming /lib/tls to /lib/tls.disabled.

But enabling NPTL can greatly increase the performance of multi-threaded applications, especially MySQL and Java in our case. It also helps in monitoring the system as only 1 process line appears in the ps/top output. But there is an issue when running in Xen, solution below ...

TODO Daniel: include some Java benchmarks here

Checking NPTL Support

Run: /lib/libc.so.6

GNU C Library stable release version 2.3.4, by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.4.5 20051201 (Red Hat 3.4.5-2).
Compiled on a Linux 2.4.20 system on 2006-03-07.
Available extensions:
   GNU libio by Per Bothner
   crypt add-on version 2.1 by Michael Glad and others
   linuxthreads-0.10 by Xavier Leroy
   The C stubs add-on version 2.1.2.
   BIND-8.2.3-T5B
   NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
   Glibc-2.0 compatibility add-on by Cristian Gafton 
   GNU Libidn by Simon Josefsson
   libthread_db work sponsored by Alpha Processor Inc
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

This shows that TLS (for NPTL) is supported.

Run: getconf GNU_LIBPTHREAD_VERSION

linuxthreads-0.10

Shows that NPTL is not in use.

Re-enable (emulated) NPTL

On the guest (DomU) only.

mv /lib/tls.disabled /lib/tls
ldconfig
ls

  ***************************************************************
  ***************************************************************
  ** WARNING: Currently emulating unsupported memory accesses  **
  **          in /lib/tls glibc libraries. The emulation is    **
  **          slow. To ensure full performance you should      **
  **          install a 'xen-friendly' (nosegneg) version of   **
  **          the library, or disable tls support by executing **
  **          the following as root:                           **
  **          mv /lib/tls /lib/tls.disabled                    **
  ** Offending process: ls (pid=2717)                          **
  ***************************************************************
  ***************************************************************

This message will appear on the main console (ie. xm console), if connected by ssh the system will appear to hang for a few seconds. Check /var/log/messages or dmesg for the warning.

Now run: getconf GNU_LIBPTHREAD_VERSION

NPTL 2.3.4

Shows NPTL is in use, this warning is the reason we currently rename tls, as described in Xen FAQ 4.1

To see the TLS libraries loaded/requested by a program: LD_DEBUG=libs /opt/java/bin/java -version 2>&1|grep tls

• Setting LD_DEBUG_OUTPUT=/tmp/debug.out will save debug output to a file with PID extension.

The problem is described in more detail at XenSpecificGlibc

On 32-bit x86 platforms, Xen uses segmentation to provide protection of the memory used for the hypervisor. This results in some performance issues since wrap-around segments as used by glibc need expensive extra handling.

It is possible to rebuild glibc so that it only uses segments such that there is no performance penalty. To do this, you need to apply the patch below to the glibc sources and then rebuild glibc with the -mno-tls-direct-seg-refs option.

Solution

Manually replace glibc. Unfortunately these RPMs can't be provided in a repository for yum as they are not actually newer versions.

Nothing needs to be changed on Dom0! You can leave /lib/tls.disabled.

RPMs are available from http://xenbits.xensource.com/glibcs/rhel44/RPMS/. This script will automatically download and install:

yum install glibc-kernheaders

URL=http://xenbits.xensource.com/glibcs/rhel44/RPMS
VER=2.3.4-2.25.xs3.2.0.58
mkdir /tmp/glibc-rpm; cd /tmp/glibc-rpm
for i in glibc-common glibc-devel glibc-headers nscd; \
 do wget $URL/i386/$i-$VER.i386.rpm; done
wget $URL/i686/glibc-2.3.4-2.25.xs3.2.0.58.i686.rpm
rpm -Uvh --force *
cd
rm -rf /tmp/glibc-rpm

• this script slightly more complicated than the old one below, rpm was fussy with these
• rpm supports proxy with: --httpproxy www-proxy --httpport 8080
• this installs a new /lib/tls, you can delete /lib/tls.disabled (if you didn't rename it earlier)
• torque-client depends on glibc-devel/headers (these are also included in the set from Xen source)

3/8/07 DC Tested on CentOS 4.5/Xen 3.1 with glibc 2.3.4-2.36, glibc-kernheaders 2.4-9.1.100.EL

Preventing YUM from Clobbering the changed glibc (ie. PIN glibc)

The next yum update will want to undo these changes. Fix by adding to the [main] section in /etc/yum.conf.

exclude=kernel,glibc*,nscd

This prevents having to put --exclude on the command line.

A neater process list

This is the ng2dev.sapac.edu.au process list after enabling NPTL.

[root@ng2dev ~]$ ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:00 init [3]                                 
    2 ?        S      0:00 [migration/0]
    3 ?        SN     0:00 [ksoftirqd/0]
    4 ?        S      0:00 [watchdog/0]
    5 ?        S<     0:00 [events/0]
    6 ?        S<     0:00 [khelper]
    7 ?        S<     0:00 [kthread]
    8 ?        S<     0:00 [xenwatch]
    9 ?        S<     0:00 [xenbus]
   15 ?        S<     0:00 [kblockd/0]
   19 ?        S<     0:00 [khubd]
   60 ?        S      0:00 [pdflush]
   61 ?        S      0:00 [pdflush]
   63 ?        S<     0:00 [aio/0]
  579 ?        S<     0:00 [kseriod]
   62 ?        S      0:00 [kswapd0]
  658 ?        S<     0:00 [kpsmoused]
  687 ?        S      0:00 [kjournald]
 1325 ?        S<s    0:00 udevd
 1709 ?        Ss     0:00 syslogd -m 0
 1713 ?        Ss     0:00 klogd -x
 1722 ?        Ss     0:00 portmap
 1741 ?        Ss     0:00 rpc.statd
 1793 ?        S<     0:00 [rpciod/0]
 1794 ?        S      0:00 [lockd]
 1831 ?        Ss     0:00 /usr/sbin/sshd
 1852 ?        Ss     0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 1864 ?        SLs    0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 1892 ?        S      0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data -i
 1894 ?        S      0:00 postgres: stats buffer process                       
 1895 ?        S      0:00 postgres: stats collector process                    
 1958 ?        Ssl    0:36 /opt/java/bin/java -Dlog4j.configuration=container-lo
 2118 ?        S      0:00 /usr/bin/perl /usr/local/pbs-telltail/pbs-logmaker /u
 2152 ?        S      0:00 /usr/local/globus/libexec/globus-scheduler-event-gene
 2194 ?        S      0:00 /usr/local/globus/libexec/globus-scheduler-event-gene
 2351 ?        Ss     0:00 /usr/libexec/postfix/master
 2365 ?        S      0:00 pickup -l -t fifo -u
 2366 ?        S      0:00 qmgr -l -t fifo -u
 2367 ?        Ss     0:00 crond
 2375 ?        Ss     0:00 login -- root     
 2376 tty2     Ss+    0:00 /sbin/mingetty tty2
 2377 tty3     Ss+    0:00 /sbin/mingetty tty3
 2378 tty4     Ss+    0:00 /sbin/mingetty tty4
 2379 tty5     Ss+    0:00 /sbin/mingetty tty5
 2380 tty6     Ss+    0:00 /sbin/mingetty tty6
 2390 ?        S      0:00 postgres: globus rftDatabase 129.127.96.105 idle     
 2485 tty1     Ss     0:00 -tcsh
 2704 ?        S      0:00 postgres: globus rftDatabase 129.127.96.105 idle     
 2705 ?        S      0:00 postgres: globus rftDatabase 129.127.96.105 idle     
 2802 tty1     R+     0:00 ps ax

Older Versions

2.3.4-2.9xen2 (hosted on a SAPAC web server), using: xen-3.0-x86_32-rhel4.1.bin.tar 48MB (from XenSourceRPMs)

yum install glibc-kernheaders

URL=http://castor.sapac.edu.au/sapac
VER=2.3.4-2.9xen2
for i in glibc-common glibc glibc-devel glibc-headers nscd; \
 do echo $URL/$i-$VER.i386.rpm; done | xargs rpm -Uvh --force

3/8/07 DC Tested (this old version) on CentOS 4.5/Xen 3.1 with glibc 2.3.4-2.36, glibc-kernheaders 2.4-9.1.100.EL

Previously tested on CentOS 4.4/Xen 3.0.2, with current RPM versions :-

• glibc-common-2.3.4-2.19
• glibc-2.3.4-2.19
• glibc-kernheaders-2.4-9.1.98.EL REQUIRED!

Temporarily Disabling NPTL

The LD_ASSUME_KERNEL variable can be used to disable NPTL for a specific program. This is useful when testing or comparing performance.

Example: LD_ASSUME_KERNEL=2.4.1 getconf GNU_LIBPTHREAD_VERSION

linuxthreads-0.10

-- DanielCox - 23 Aug 2006