eworldproblems - HAProxy “backup” and “option redispatch”

I’m testing out my new infrastructure-in-progress for redundant web hosting, described in my last post. The quick outline of the components again, is a total of four VMs:

Two small & inexpensive instances in a tier 4 datacenter running Apache Traffic Server + HAProxy.
One bigger, rather spendy backend VM in a tier 4 datacenter running Apache, webapp code, databases, etc. Call it hosting1.
One VM roughly equivalent in specs to the backend VM above, but running on a host machine in my home office so its marginal cost to me is $0. Call it hosting2.

The thinking is that hosting1 is going to be available 99%+ of the time, so “free” is a nice price for the backup VM relative to the bandwidth / latency hit I’ll very occasionally take by serving requests over my home Internet connection. Apache Traffic Server will return cached copies of lots of the big static media anyway. But this plan requires getting HAProxy to get with the program - don’t EVER proxy to hosting2 unless hosting1 is down.

HAProxy does have a configuration for that - you simply mark the server(s) you want to use as backups with, sensibly enough, “backup,” and they’ll only see traffic if all other servers are failed. However, when testing this, at least under HAProxy 1.4.24, there’s a little problem: option redispatch doesn’t work. Option redispatch is supposed to smooth things over for requests that happen to come in during the interval after the backend has failed but before the health checks have declared it down, by reissuing the proxied request to another server in the pool. Instead, when you lose the last (and for me, only) backend in the proper load balance pool, requests received during this interval wait until the health checks establish the non-backup backend as down, and then return a 503, “No servers are available to handle your request.”

I did a quick test and reconfigured hosting1 and hosting2 to both be regular load balance backends. With this more typical configuration, option redispatch worked as advertised, but I now ran the risk of traffic being directed at hosting2 when it didn’t need to be.

Through some experimentation, I’ve come up with a configuration that my testing indicates gives the effect of “backup” but puts both servers in the regular load balance pool, so option redispatch works. The secret, I think, is a “stick match” based on destination port – so matching all incoming requests – combined with a very imbalanced weight favoring hosting1.

Here’s a complete config that is working out for me:

global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  log  10.2.3.4 local0
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  stats  socket /var/lib/haproxy/stats
  user  haproxy
 
defaults
  log  global
  maxconn  8000
  option  redispatch
  retries  3
  stats  enable
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s
 
listen puppet00
  bind 127.0.0.1:8100
  mode http
  balance static-rr
  option redispatch
  retries 2
  stick match dst_port
  stick-table type integer size 100 expire 96h
  server hosting1 10.0.1.100:80 check weight 100
  server hosting2 10.0.2.2:80 check weight 1

I tested this by putting different index.html’s on my backend servers and hitting HAProxy with cURL every 0.5 seconds for about an hour, using the watch command to highlight any difference in the data returned:

watch -x --d=permanent -n 0.5 curl -H 'Host: www.mysite.com' http://104.156.201.58  --stderr /dev/null

It didn’t return the index.html from hosting2 a single time, until I shutdown apache on hosting1 after about an hour.

There’s a small amount of magic here that I unfortunately don’t understand, but will resign myself to being happy that it works as desired. Once failed over to hosting2, I would kind of expect the stick table to update and cause everything to stay with hosting2 even after hosting1 comes back. In actuality, it returns to hosting1. So cool, I guess.