Sync bonding?

[Bob_Zimmerman]

Sync is kind of a pain. People want to use a cable directly between the firewalls, but this can delay or even prevent proper automatic failover. When one firewall reboots, the other firewall sees its sync interface go down and has to probe its other interfaces to see whether it is the broken one. If the link probing fails on any interface, the member may refuse to take over.

Using a single switch also doesn't work very well, since when you reboot the switch (say, to apply updates), the members both see their sync interfaces go down, so they both start probing interfaces. This can cause active contention or it can cause both members to go down, depending on the probe results.



Attempting to solve these problems, I arrived at this config:

add bonding group 0
add bonding group 0 interface eth0
add bonding group 0 interface eth1
set bonding group 0 mode round-robin

I connected the interfaces to a pair of dumb switches (but did not connect the switches to each other) and set up my cluster using bond0 for sync. It seemed to work perfectly. I rebooted the switches one at a time, and I never appeared to lose sync.

This seems pretty much ideal. No switch-side need for special multi-chassis link aggregation systems like Cisco's vPC. You can reboot either switch for updates without breaking communications between the firewall members. You can reboot either firewall member without causing loss of link on the other.

Am I missing anything? Does anyone know of a reason we shouldn't use a bond like this for sync?

[cciesec2006]

add bonding group 0
add bonding group 0 interface eth0
add bonding group 0 interface eth1
set bonding group 0 mode round-robin

I would change the mode from "round-robin" to "Active-Standby" because this is SYNC interface.

Am I missing anything? Does anyone know of a reason we shouldn't use a bond like this for sync?

This works very well under the assumption that spanning tree is working properly on the switches. If the switches have spanning tree issues, there goes your SYNC :-(

[Bob_Zimmerman]

I would change the mode from "round-robin" to "Active-Standby" because this is SYNC interface.

Either should work. I picked round-robin because I have had situations where I want more than 1g of sync capacity, but I've never needed more than 2g. Rather than burning a 10g interface for sync, this lets me use 1g interfaces built into basically every server, which would otherwise go unused.

Edited to add: Come to think of it, if I ever want 3g (or more) of sync capacity, I could just add a third disconnected switch, plug a third copper interface into it, and add the third interface to the bond.

This works very well under the assumption that spanning tree is working properly on the switches. If the switches have spanning tree issues, there goes your SYNC :-(

Spanning tree shouldn't matter because the switches shouldn't see each other. Bonded interfaces on the firewalls don't forward frames from one to the other. The switches don't need any kind of link between them (and in fact should not have one, for the reason you mention).

[cciesec2006]

Edited to add: Come to think of it, if I ever want 3g (or more) of sync capacity, I could just add a third disconnected switch, plug a third copper interface into it, and add the third interface to the bond.

I don't think you can use 3 interfaces. 802.3AD supports only 2, 4, or 8 interfaces.

Spanning tree shouldn't matter because the switches shouldn't see each other. Bonded interfaces on the firewalls don't forward frames from one to the other. The switches don't need any kind of link between them (and in fact should not have one, for the reason you mention).

.

if there is no link between the switches, how does the bond work?

[Bob_Zimmerman]

I don't think you can use 3 interfaces. 802.3AD supports only 2, 4, or 8 interfaces.

Not doing 802.3AD. With this topology, the switches don't need to be aware of each other or of the fact that the endpoints are aggregating the link. 802.3AD is good for managing the aggregation state and allows more complicated load distribution than round-robin, but you can set up an aggregate link with crossover cables between two servers.

if there is no link between the switches, how does the bond work?

I made a brief diagram showing the topology so we can use shared terms.



First, let's talk about received traffic. The aggregated interfaces on the firewall side get one IP address and one MAC address. The OS does not care which physical link it receives a frame on. If the frame is to its MAC address, it hands the frame up to the network stack. If the packet inside the frame is to the interface's IP address, the network stack hands it to the process listening on the appropriate interface and port.

Transmitting is a little more complicated. Linux takes a look at all available links in the bond, then picks one which is up to send the frame. In active-passive link aggregation, it always picks the same link until that one goes down, at which point it moves to the next link. In round-robin, it picks the next available link in the sequence from the last one it used.

When the frame hits the wire, the switch gets it. If the switch knows the MAC address of the destination, it sends it only to that port. If it doesn't, it sends it out all ports. Easy enough.

Let's say firewall 1 is active.

It sends the first sync frame out link A to firewall 2's sync MAC. It sends the second sync frame out link B to firewall 2's sync MAC. The switches don't have to know about each other. They can both forward the frames just fine, and they happen to be going to the same place. If firewall B only sends traffic rarely, one switch (or both) may not know its MAC address, so they may end up flooding the traffic out all interfaces frequently. With two members, there only need to be two switchports in the VLAN, so this is not really a problem.

Essentially, the switches are just acting like a cable here, but if one firewall reboots, the switches keep the other firewall's link status up.



The part which really interests me is this seems like it can be extended to an arbitrary number of switches and an arbitrary number of firewalls. Let's say you want a four-member firewall cluster with three sync links. You just plug all four firewalls into all three switches:



As long as all of the ports on a given switch are in the same VLAN, and that VLAN doesn't see any of the other switches, you should be completely fine. Firewall 1 sends a frame to firewall 3. It doesn't matter whether it goes through switch 1, 2, or 3. Any of them can reach the MAC address of firewall 3.

You may have to extend the MAC learning timeout to maintain efficient switching if one member is sending and the other members are only receiving the traffic. If all members are talking over sync (like with a load-sharing cluster), this should not be needed.

[Bob_Zimmerman]

Thinking about it further, if a cable or transceiver fails, there would be a problem.

If FW2LA to SS1 is down, FW1 won't know this, so it will keep sending traffic out FW1LA. SS1 will not be able to forward it anywhere usefully. FW2 would then only see every other frame. LACP with MCEC, vPC, or other proprietary multi-chassis link aggregation technologies would address that, and you would just end up with an unequal-size link. On the plus side, at least if FW2LA fails, the member will see a link failure and FW1 should take over. I will need to test that.

In my experience, updates on either the firewall members or the switches between them are at least two orders of magnitude more common than actual hardware failures in the transceivers (especially copper gigabit interfaces) or cables. In environments less concerned about updates, this may not be an acceptable trade.