[kaldek]

I agree with the previous posts. I have a customer who is running a 4-node cluster spread across two sites, about 3 kilometres apart. The inter-site link is a dark fibre (i.e. they own the entire line) connected between two Cisco Catalyst 6000 series switches. The link carries a large number of VLANs aside from just the sync network. This customer is also operating in multicast load sharing mode, so the potential bandwidth usage there might seem scary. The cross-site link is 1Gb/s Ethernet handling all those VLANs, but the firewall and its sync networks are all 100Mb/s, which explains why it is working.

We have recently discussed with the customer the long term goal of migrating to a Layer-3 redundancy design (i.e. Layer 4-7 load balancing to distribute user connections to the two different sites) and setting up separate clusters at each site, due in part to the fact that we know their network usage will grow and eventually saturate the cross-site link due to the severe number of multicast packets.

Having two separate clusters also increases your redundancy, because a software fault on one cluster won't affect the other one. As it is right now, if the customer stuffed up their static MAC address tables in their switches, or broke the static ARP entries in their routers, the entire cluster would malfunction. Given all that, this customer is one of the largest organisations in the country, and I have not heard a peep out of them regarding firewall problems since we installed the solution. Their website performance is also exemplary, so sometimes things that seem precarious can end up being quite solid.

P.S. In case you're wondering why we even built a cross-site cluster in the first place, you must understand the customer's history. Their existing cluster was cross-site, based on Checkpoint Firewall-1 4.0 on Solaris with StoneBeat providing the HA functionality. Behind these firewalls, the entirety of their server infrastructure is shared-layer 2 cross-site VLANs. Changing that entire design to a routed environment between the sites was just not economically feasible in the short term.