Monday, February 9, 2009

Build a 64-bit Computer Cluster -- Part 3: Install & Configure Perceus for Node Provisioning

Building a 64-bit Computer Cluster Using CentOS Linux 5.2, Perceus 1.4, Sun Grid Engine 6.2, Matlab Distributed Computing Server 4.0 (MDCS) and MATLAB 7.7

Installing and Configuring Perceus 1.4 to Handle Node Provisioning

Most of these steps will be the same as in the Perceus 1.4.0 Manual
mkdir /usr/src/perceus
cd /usr/src/perceus
We're going to use source RPM's instead of source for ease of future administration
A default CentOS 5.2 install will be missing a few components, use yum to add them:
yum install rpm-build gcc perl perl-DBI autoconf nasm openssl-devel elfutils-libelf-devel
Now build the dependency RPM's:
rpmbuild --rebuild bash-completion-20060301-1.caos.src.rpm
rpmbuild --rebuild perl-IO-Interface-0.98-1.caos.src.rpm
rpmbuild --rebuild perl-Net-ARP-1.0-2.caos.src.rpm
rpmbuild --rebuild perl-Unix-Syslog-0.99-1.caos.src.rpm
Install the dependency RPM's:
cd /usr/src/redhat/RPMS/noarch/

rpm -Uvh bash-completion-20060301-1.redhat.noarch.rpm
rpm -Uvh perl-IO-Interface-0.98-1.redhat.noarch.rpm
rpm -Uvh perl-Net-ARP-1.0-2.redhat.noarch.rpm
rpm -Uvh perl-Unix-Syslog-0.99-1.redhat.noarch.rpm
Build and Install Perceus:
cd /usr/src/perceus
rpmbuild -ta perceus-1.4.4.tar.gz
This step will take a few minutes. When it finishes you will have a nice shiny RPM to install:
rpm -ivh /usr/src/redhat/RPMS/x86_64/perceus-1.4.4-1966.x86_64.rpm
Do a little initial configuration before starting Perceus:

edit /etc/perceus/perceus.conf

You must define the master network device on line 12 (i.e. eth0 eth1)

master network device = eth0

Start Perceus for the first time
  • When it asks if you want to do a full system initialization say yes
  • Fill out the registration
  • Accept the default node boot address ranges
  • Enter your domain or just accept the default
  • Accept the default console output
Activate the following Perceus modules:
perceus module activate masterauth init/all
perceus module activate hostname init/all
perceus module activate ipaddr init/all
Using the ipaddr module is helpful for assigning a permanent address to each node and seems to be recommended by the Perceus developers. The node will grab an IP address from the DHCP pool during PXE boot, but will then be assigned the address you define when the final kernel boots.

If you have more than one ethernet interface on your worker nodes, then how you write this file will determine whether or not each interface is brought up at boot time. If you don't specify it, it won't be activated. The example pattern below should be sufficient for most small setups, just copy it for as many nodes as you have.
nano -w /etc/perceus/modules/ipaddr
n0000 eth0:
n0001 eth0:
n0002 eth0:
n0003 eth0:
n0004 eth0:

Add the nodes to /etc/hosts on the head node: n0000 n0001 n0002 n0003 n0004

Build your first Perceus VNFS Capsule:
cd /usr/share/perceus/vnfs-scripts
You'll see scripts to build several different distributions and versions.  For our purposes we will be building CentOS 5.2 to go along with our head node. Using the centos-5-genchroot script will build a current CentOS verison.
This step will take a few minutes.  When it finishes you have to run this script to make it stateless:
./ /var/tmp/vnfs/centos-5-1.x86_64
Import the VNFS Capsule:
perceus vnfs import /usr/src/perceus/centos-5-1.stateless.x86_64.vnfs
  • Input the root password for the capsule (do not leave blank)
  • Say yes to hybridization support
  • Accept the default console tty0
  • The IP address of the DNS server should be the IP address of the private ethernet interface on the head node. Confirm and accept the default.
  • The IP address of the syslogd host should be the IP address of the private ethernet interface on the head node. Confirm and accept the default.

Edit /etc/perceus/defaults.conf to set the default VNFS capsule:
nano -w /etc/perceus/defaults.conf

# Define the default VNFS image that should be assigned to new nodes
Vnfs Name = centos-5-1.stateless.x86_64

Mount the VNFS Capsule:
perceus vnfs mount centos-5-1.stateless.x86_64
Install the client side Perceus daemon provisiond:
rpm -ivh --root /mnt/centos-5-1.stateless.x86_64/usr/src/redhat/RPMS/x86_64/perceus-provisiond-1.4.4-1966.x86_64.rpm
Edit /mnt/centos-5-1.stateless.x86_64/etc/ntp.conf , comment out the centos pool time servers and add the line:


If you're going to install Sun Grid Engine, copy /etc/services to /etc/services.bak
cp /etc/services /etc/services.bak
Then, edit /etc/services on the head node and remove the following lines:

opalis-rdv 536/tcp
opalis-rdv 536/udp
nmsp 537/tcp
nmsp 537/udp

Then copy the revised /etc/services to the VNFS capsule:
cp /etc/services /mnt/centos-5-1.stateless.x86_64/etc/services
Edit /mnt/centos-5-1.stateless.x86_64/etc/hosts and add a line for the head node and the other nodes: headnode n0000 n0001 n0002 n0003 n0004

Unmount the capsule:
perceus vnfs umount centos-5-1.stateless.x86_64
You are now ready to boot the worker nodes for the first time. I recommend attaching a monitor and keyboard the first time so you can watch for any problems during bootup, afterwards you can monitor the nodes during reboot using the command:
perceus node status
If you're continuing to install SGE or Matlab, we need to work on our VNFS capsule some more, so don't boot all your worker nodes yet and leave a keyboard and monitor connected to your test worker.

Using yum to add additional packages to a VNFS Capsule:

If you want to use yum to add packages to your capsule, you first need to mount it. Then use the following yum syntax to add a package like nano:
yum --installroot /mnt/centos-5-1.stateless.x86_64 install nano
I actually recommend you run the above command since the default VNFS capsule won't have an editor installed

Make sure to unmount when finished. Perceus can sync the filesystem of a running node with an updated VNFS capsule, and this works pretty well in a lot of situations where you are just making some basic additions or modifications. I do recommend that if you are doing something like adding a daemon that will start at boot that you reboot the node so you can make sure that the node doesn't choke on startup or shutdown.

Note on Swap Space

By default the worker nodes will not have any swap space. Whether or not you need it is up to you, but if you decide to implement swap you need to remember that the fstab in the VNFS capsule will define the swap space across all of the worker nodes. Therefore, if your worker nodes have hard drives, the swap partition should be the same partition on all of them i.e. /dev/sda1. Use fdisk to create the swap partition, and mkswap to format it. You can minimize the amount of swapping that the nodes will do by editing the file /etc/sysctl.conf in the VNFS capsule and adding the following line:


No comments:

Post a Comment