Load Balancer

Note: I used this setup for about a month but started to see stale connections to my mail servers. I tracked the problem down to loadd. I don't know, exactly, what was causing the problem. I decided, instead, to deploy a pair of Gentoo boxes running Linux Virtual Server. This setup allowed me to easily setup failover between the LBs. You can find that doc here.

Note: These are for FreeBSD.

My original setup used basic round-robin NAT to spread the connections around the nodes. This works reasonably well if your systems are identical, or close enough, but starts to fail when the machines are different.

You will need to build a custom kernel with the options below. Building a new kernel is beyond the scope of this document. See the Handbook for details.

# IPFW (Always on; default to accept)
options         IPFIREWALL              #firewall
options         IPFIREWALL_VERBOSE      #enable logging to syslogd(8)
options         IPFIREWALL_FORWARD      #enable transparent proxy support
#options        IPFIREWALL_VERBOSE_LIMIT=100    #limit verbosity
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by default
options         IPV6FIREWALL            #firewall for IPv6
options         IPV6FIREWALL_VERBOSE
#options        IPV6FIREWALL_VERBOSE_LIMIT=100
options         IPV6FIREWALL_DEFAULT_TO_ACCEPT
options         IPDIVERT                #divert sockets

Original Setup

/usr/local/etc/ipfw.conf

# natd
add 50 divert natd all from any to any via fxp0

/usr/local/etc/natd.conf

# fxp0 is the WAN interface.
interface fxp0
##### EMAIL
### Defaults
redirect_port tcp mail1:25,mail2:25,mail3:25 209.94.64.30:25
redirect_port tcp mail1:110,mail2:110,mail3:110 209.94.64.30:110
redirect_port tcp mail1:143,mail2:143,mail3:143 209.94.64.30:143

/etc/rc.conf

firewall_enable="YES"
firewall_type="/usr/local/etc/ipfw.conf"
natd_enable="YES"
natd_interface="fxp0"
natd_flags="-config /usr/local/etc/natd.conf"

This worked well enough for me for over a year but adding or changing a node became problematic because slower machines would get hammered while the beefy machines idled. Not a good senario. So, finally got around to setting up a load balancer.

New Setup

The individual nodes in my cluster are not homogeneous. That makes using basic round-robin NAT impractical. From the beginning, I was looking at using loadd/lmd. It looked like a good load balancer but, after looking at it a bit, I found that lmd only supported HTTP out-of-the-box. Fortunately, it uses modules to check service status so it's not too bad to add new service checks. Not having time then, I put loadd on the back burner and went with the natd solution above. I finally had to so here's what I did.

Gateway Server

Install net/loadd from ports.

/usr/local/etc/ipfw.conf

Note: It's important that loadd goes before natd.

# loadd
add 49 divert 8670 tcp from any to any via fxp0
# natd
add 50 divert natd all from any to any via fxp0

/usr/local/etc/loadd.conf

ipaliasing is the public IP address of the cluster.

servers = 10.1.1.2,10.1.1.3,10.1.1.4,10.1.1.5
ipaliasing = 209.94.64.30
ports = 25,110,143
verbosemode = yes
balancingmode = intlloadsharing
daemon = yes
transparent_proxy = yes

/usr/local/etc/natd.conf

# fxp0 is the WAN interface.
interface fxp0

/etc/rc.conf

firewall_enable="YES"
firewall_type="/usr/local/etc/ipfw.conf"
natd_enable="YES"
natd_interface="fxp0"
natd_flags="-config /usr/local/etc/natd.conf"
loadd_enable="YES"

On the Nodes

Install net/lmd from ports.

/usr/local/etc/lmd.conf

Node: loaddservers is a comma delimited list of loadd servers.

loaddservers = 10.1.1.1

/etc/rc.conf

lmd_enable="YES"