SplitBrain

Now, equipped with the possibility to create a new Debian-DomU every 5 minutes (see previous posts), we use this new gained knowledge to set up a high available frontend server for our services. The plan is to have two xen-vm’s running on different physical servers, which provide failover for our external IP in case the active server fails.

It would be possible to use openais/pacemaker (e.g. with DRBD) on the SLES11 host-system. If you want this, have a look at[1].

For now we decided to take another route, using a simple hearbeat setup inside the vm’s without drbd. Reasons are:

• no real need for DRBD, as the frontend is only proxy, no data-server
• DRBD could help having to configure only one node. But having two nodes is also not bad, as this way new configuration could be tested on the not-active host, and switched if working.
• having heartbeat inside the vm makes us independent from underlying host system
• not adding complexity. lets go the simple route for now

This blogpost is moving along the lines of [2], so you may find similarities .

The Setup is:

shared IP: 192.168.0.60
+-----+  +-----+
| lb1 |  | lb2 |
+-----+  +-----+
IP: .61  IP: .62

Lets get started:
first make sure, both nodes know each other by name – add following to /etc/hosts

127.0.0.1       localhost
192.168.0.61   lb1
192.168.0.62   lb2

install hearbeat on lb1 and lb2;

apt-get install heartbeat

allow binding of shared ip adress by editing /etc/sysctl.conf adding the following line (lb1&lb2)

net.ipv4.ip_nonlocal_bind=1

run

sysctl -p

generate file /etc/ha.d/authkeys on lb1&lb2 with following content:

auth 3
3 md5 somerandomstring

where somerandomstring is the password, which heartbeat-daemons use to authenticate. choose something secure. According to [2] md5 is the most secure auth-mechanism.
Set Permissions (only root should read)

chmod 600 /etc/ha.d/authkeys

edit the file /etc/ha.d/ha.cf on lb1:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.0.62
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
#logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1
node    lb2

ucast needs to be the ip of lb2, as this file tells hearbeat where and how to reach the other cluster-nodes.

Put the same file on lb2, just changing the ucast to match the ip of lb1.

Now we configure a resource heartbeat should take care of, in our case its the IP adress 192.168.1.60. Edit the file /etc/ha.d/haresources an put

lb1 192.168.1.60

This tells heartbeat to care for 192.168.1.60, which means in our case to take the IP over to lb2 in case lb1 is not responding any more. So lb1 is the primary node to have this IP. The first entry lb1 needs to match the output of uname -n on that machine.

Now start heartbeat on both nodes:

/etc/init.d/heartbeat start

Check with ifconfig, on lb1 should be an interface eth0:0 with the IP 192.168.1.60. After stopping lb1 (e.g. in xen with xm shutdown lb1) this interface should show up on lb2. Starting lb1 again it should result in having that IP back there.

Were done!

Links:
[1] http://www.howtoforge.com/installation-and-setup-guide-for-drbd-openais-pacemaker-xen-on-opensuse-11.1
[2] http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-heartbeat-on-debian-lenny-p2