IBM System x high end servers can now take advantage of the NUMA ((Non Uniform Memory Architecture) awareness feature. The NUMA awareness function was recently enabled with the primary Linux distributions and also within the Oracle database itself.
The Linux 2.6 kernel from Red Hat (specifically RHEL4 Update 5, or RHEL 5) and Novell’s SLES10 are NUMA aware, meaning that the kernel can detect whether it is running on a NUMA system. Libraries and a control interface are available to optimize the execution of an application for NUMA.
Oracle Database 11g and Oracle 10gR2 (with patch set 10.2.0.4) include the NUMA awareness feature as well. When these versions are run with Red Hat 4 Update 5 or higher or with Novell’s SLES10, systems such as IBM’s x3950 or x3950 2 can be configured and optimized for performance with up to 16 sockets.
The IBM x3950 M2 can expand from 4 to 16 sockets in a scale –up multi-chassis configuration based upon NUMA. So far, this machine has not been able to fully use processing power beyond 8 sockets when running Oracle databases. This was due to the inability of the Linux operating system and the Oracle database to utilize system resources housed in chassis 3 and 4 of the x3950 to complete an assigned process. With these new levels of software, this inability is no longer an issue
For customers running Red Hat 4 Update 5 there is also a need to document the following commands to turn on the NUMA bits in RHEL 4. (These commands are automatically performed when RHEL 5 boots).
Here is how to detect if machine is NUMA enabled and what to do next:
The first step is to address whether or not the customer’s IBM x3950 is correctly enabled for
NUMA. When using RHEL 4, update 5, there is a boot parameter required for Linux to correctly
detect the NUMA characteristics of the machine. One can check this by doing the following:
[root@x3950-2 ~]# numactl –hardware
available: 4 nodes (0-3)
node 0 size: 15887 MB
node 0 free: 15048 MB
node 1 size: 15907 MB
node 1 free: 15824 MB
node 2 size: 15907 MB
node 2 free: 15821 MB
node 3 size: 14897 MB
node 3 free: 14838 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
You should see 4 nodes listed like above. If you see an answer such as the following lines:
[root@athena-1a ~]# numactl –hardware
available: 1 nodes (0-0)
node 0 size: 32300 MB
node 0 free: 28042 MB
node distances:
node 0
0: 10
then NUMA is not enabled. You will need to add “numa=acpi” to the kernel command line.
This is best done in the grub config file /boot/grub/menu.lst
In summary, with the right level of the Linux operating system and the right level of Oracle Database, customers should be able to fully exploit the NUMA characteristics of the IBM x3950 or other IBM machines that are NUMA enabled.