Implementing Failover Redundancy in Ubuntu 11.10 with Heartbeat + Rsync

1. Install Heartbeat on both hosts:

[root@host1 ~]# apt-get install heartbeat
[root@host2 ~]# apt-get install heartbeat

 2. Rsync can be used to synchronize any file/directory for any service,
    however in this example we will use Apache service:

[root@host1 ~]# apt-get install rsync
[root@host2 ~]# apt-get install rsync

 3. Check fully qualified domain names (fqdn) on host 1 and host2:

[root@host1 ~]# uname –n
[root@host2 ~]# uname –n
In my case host names (fqdn): host1.corp.un
Second Machine name (fqdn): host2.corp.un
IP Address of host1: 192.168.1.130
IP Address of host2: 192.168.1.131
Default Gateway of both hosts: 192.168.1.1
DNS Server: 192.168.1.1
DNS Server: 192.168.1.2

 4. Make sure that you have httpd installed:

 service apache2 status

You should see the following:

Apache2 is running (pid xxxx).

Where xxxx is the PID number.

5. Configure network settings for both nodes and check by pinging and resolving hostnames.

5.1. For host1:

[root@host1 ~]# cat /etc/network/interfaces

# The primary network interface
auto eth0
iface eth0 inet static

        address   192.168.1.130
        netmask   255.255.255.0
        network   192.168.1.0
        broadcast 192.168.1.255
        gateway   192.168.1.1

[root@host1 ~]# cat /etc/resolv.conf

domain corp1.un
nameserver 192.168.1.1
nameserver 192.168.1.2

[root@host1 ~]# cat /etc/hosts

127.0.0.1       localhost
192.168.1.130    host1.corp.un host1
192.168.1.131    host2.corp.un host2

5.2. For host2:

[root@host2 ~]# cat /etc/network/interfaces

# The primary network interface
auto eth0
iface eth0 inet static

        address   192.168.1.131
        netmask   255.255.255.0
        network   192.168.1.0
        broadcast 192.168.1.255
        gateway   192.168.1.1

[root@host2 ~]# cat /etc/resolv.conf

domain corp1.un
nameserver 192.168.1.1
nameserver 192.168.1.2

[root@host2 ~]# cat /etc/hosts

127.0.0.1       localhost
192.168.1.130    host1.corp.un host1
192.168.1.131    host2.corp.un host2

6. Configuring Heartbeat on both computers
Heartbeat has three main configuration files:

/etc/ha.d/authkeys
/etc/ha.d/ha.cf
/etc/ha.d/haresources

6.1. Configuring /etc/ha.d/authkeys File

The authkeys file must be owned by root and be chmod 600.
The actual format of the authkeys file is two-lines simple format.
There is an auth directive with an associated method ID number,
and there is a line that has the authentication method and
the key that go with the ID number of the
auth directive. There are three supported authentication methods:

crc, md5 and sha1.

[root@host1 ~]# vim /etc/ha.d/authkeys

Add the following :

auth 1

1 sha1 ThisIsALongPasswordForSecurityReasons

And change the permission of the authkeys file:

[root@host1 ~]# chmod 600 /etc/ha.d/authkeys

6.2. Configuring /etc/ha.d/ha.cf File

Edit the /etc/ha.d/ha.cf file in your favorite text editor:

[root@host1 ~]# vim /etc/ha.d/ha.cf

Add the following lines in the ha.cf file:

Configuration 1 (taken from Linux Journal with some changes)

keepalive 1
deadtime 10
initdead 120
warntime 5
udpport 694
# host1 heartbeat ip address is 192.168.1.130
# host2 heartbeat ip address is 192.168.1.131
# For simple two-server clusters, ucast or bcast can be used
ucast eth0 192.168.1.130
# bcast eth0
# serial /dev/ttyS1
# baud 19200
auto_failback off
stonith_host host1 wti_nps ares.example.com erisIsTheKey
stonith_host host2 wti_nps ares.example.com erisIsTheKey
# The following names must match the output of uname -n
node host1
node host2
use_logd yes
# logfile /var/log/ha-log
# logfacility local0

We can use, however, an example from usr/share/doc/heartbeat/
and adjust it according our needs:

zless /usr/share/doc/heartbeat/ha.cf.gz

6.3. Configuring /etc/ha.d/haresources File

This file contains the information about resources which we want to make highly available.
All services that we want Heartbeat to manage must be removed from the system init
for all init levels.

The command to accomplish this task is:
[root@host1 ~]#  /usr/sbin/update-rc.d -f <service_name> remove
[root@host2 ~]#  /usr/sbin/update-rc.d -f <service_name> remove
[root@host1 ~]#  vim /etc/ha.d/haresources
For minimalistic configuration add the following lines:

host1.corp.un IPaddr::192.168.1.132/24/eth0 apache2
This will assign a virtual ip address to the eth0 which can be verified by
ifconfig command:
eth0:0    Link encap:Ethernet  HWaddr 06:00:27:1e:07:db
          inet addr:192.168.1.132  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        
6.4. Copy the /etc/ha.d/ directory from host1 to host2:

[root@host1 ~] scp -r /etc/ha.d/ root@host2:/etc/

7. Configuring rsycn

For synchronization of the content we will use rsync package which by default is installed in

/usr/bin/rsync:

[root@host1 ~] crontab -e

*/1 * * * * /usr/bin/rsync -avz --perms --delete --links -e ssh /var/www/

host2:/var/www/

[root@host2 ~] crontab -e

*/1 * * * * /usr/bin/rsync -avz --perms --delete --links -e ssh /var/www/

host1:/var/www/

Note:

*/1 Check directories every minute and if there any changes,

perform transfer the changed files/directories.
/usr/bin/rysnc start rsync
-a archive
-v verbosity
-z perform compression
--perms means retain the original permissions
--delete delete extra files from host2 which are not found on node1
--links copy the symbolic links as it is.
-e define rsh command, in our case it is ssh

8. Starting Services and Testing

8.1. Start heartbeat:

[root@host1 ~]# /etc/init.d/heartbeat start
[root@host2 ~]# /etc/init.d/heartbeat start

The following will be observed:

Starting High-Availability services: IPaddr[xxxx]: INFO:  Resource is stopped

Done.

8.2. Open an SSH terminal on each cluster member and type:

[root@host1 ~]# tail -f /var/log/syslog
[root@host1 ~]# tail -f /var/log/ha-debug
[root@host1 ~]# tail -f /var/log/ha-log
[root@host1~]#  cl_status nodestatus host1.corp.un
[root@host2 ~]# tail -f /var/log/syslog
[root@host2 ~]# tail -f /var/log/ha-debug
[root@host2 ~]# tail -f /var/log/ha-log
[root@host2~]#  cl_status nodestatus host2.corp.un

Observe if there are any problems.

8.3. Configuring Apache2

Create the file index.html on both nodes:
On host1:

[root@host1 ~]# echo "host1 apache test" > /var/www/html/index.html
On host2:

[root@host2 ~]# echo "host1 apache test" > /var/www/html/index.html

Open web-browser and type in the URL: http://192.168.1.132
It will show host1 apache test.
Stop the heartbeat daemon on host1:

[root@host1 ~]# service heartbeat stop

In the browser type in the URL http://192.168.1.132 and press Enter.
It will show host2 apache test.
Bring the node back by issuing the command:

[root@host1 ~]# service heartbeat start

Now stop the heartbeat daemon on host2:

[root@host2 ~]# service heartbeat stop

In the browser type in the URL http://192.168.1.132 and press Enter.
It will show host1 apache test.
Bring the node back by issuing the command:

[root@host2 ~]# service heartbeat start

A virtual network interface and the associated IP address (192.168.1.132) is created by Heartbeat,
no manual configuration is required.
Also all services placed in /etc/ha.d/haresources file will start automatically.
Don't use the IP addresses 192.168.1.130 and 192.168.1.131 for services.
These addresses are used by heartbeat for communication between host1 and host2.
If used for virtual addressing, it will cause disturbance and heartbeat and will not work properly.

Useful links:
Configuring A High Availability Cluster (Heartbeat) On CentOS