Rancher High-Availability

This blog post is the last part of a series of three tutorials dedicated to Rancher High-Availability.

We previously installed Rancher single-node using a external databases:

• a replicated Master-Master MySQL database,
• a MariaDB Galera Cluster.

This time we will go full HA by installing Rancher on three servers.

Pre-requisites

You need an existing Rancher server (single-node) that uses an external database. You also need from 3 to 5 servers with at least 1GB RAM (8GB recommended) and a load balancer:

• Rancher Server One available at IP address 1.1.1.1,
• Rancher Server Two available at IP address 2.2.2.2,
• Rancher Server Three available at IP address 3.3.3.3,
• Load Balancer available at IP address 4.4.4.4.

All servers must have Docker installed.

Firewall configuration

You need a global access for each Rancher server on TCP Ports 22, 80, 443 and 18080.

You need access between Rancher servers :

• to UDP Ports 500, 4500
• to TCP Ports: 2181, 2376, 2888, 3888, 6379

Heap Size configuration

For larger deployments you may also need to increase the Rancher Heap Size. That can be done by passing the environment variable -e JAVA_OPTS="-Xms2048m -Xmx2048m" when running the rancher/server container.

Installing the Rancher Load Balancer

We need a load balancer in front of our 3 Rancher servers. The HA documentation gives us configuration samples for Nginx, Apache and HAProxy. Let’s give a try to HAProxy.

Copy the configuration file from Rancher documentation and edit it using you 3 nodes IP addresses:

global
  maxconn 4096
  ssl-server-verify none

defaults
  mode tcp
  balance roundrobin
  option redispatch
  option forwardfor

  timeout connect 5s
  timeout queue 5s
  timeout client 36000s
  timeout server 36000s

frontend http-in
  mode tcp
  bind *:443 ssl crt /etc/haproxy/certificate.pem
  default_backend rancher_servers

  acl is_websocket hdr(Upgrade) -i WebSocket
  acl is_websocket hdr_beg(Host) -i ws
  use_backend rancher_servers if is_websocket

backend rancher_servers
  server websrv1 1.1.1.1:443 weight 1 maxconn 1024 ssl
  server websrv2 2.2.2.2:443 weight 1 maxconn 1024 ssl
  server websrv3 3.3.3.3:443 weight 1 maxconn 1024 ssl

Connect to the load balancer server using SSH and paste the HAProxy configuration in a file named haproxy.cfg.

You may also need a SSL certificate. Generate it using the command:

 sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /root/rancher.key -out /root/rancher.crt

And create the .pem file using cat rancher.crt rancher.key > rancher.pem

Then run a Docker container for HAProxy:

docker run -d --name haproxy -v /root/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -v /root/rancher.pem:/etc/haproxy/certificate.pem:ro haproxy:1.5

You can display the logs using the command docker logs -f haproxy.

Exporting the Rancher High-Availability script

Go to the Rancher High-Availability page:

1. Copy/paste the IP address of your load balancer into the Host Registration URL field.
2. Copy/paste the content of the rancher.key file into the Private Key field and the content of rancher.crt into Certificate.
3. Click on Generate Config Script and on Download Config Script.
4. Stop the Rancher container on Rancher Server One (docker stop <containerID>).

5. Copy the downloaded script on each node, and execute it:

   sudo bash ./rancher-ha.sh rancher/server:v1.1.4
   

You can display the logs on each node using docker logs -f rancher-ha. Also you can access each Rancher UI at http://1.1.1.1:1800/ and view the Audit Logs (Admin > Audit Logs) and the HA Environment stack.

Wait for all the servers to start (this may take a few minutes). Finally the Rancher cluster UI should be available at the IP address of your load balancer https//4.4.4.4/ and should display the High-Availability status:

Conclusion

We managed to have multiple Rancher severs working together but the load balancer is still a Single Point of Failure. If it goes does the whole infrastructure would be unavailable.