Cluster XL pair configs out of synch

[leahrev]

We have a pair of Linux boxes set up w/ checkpoint FW software (NG w/ AI, running SPLAT in an HA pair scenario) The HA pair was working fine, but then something got out of synch with the config on the two machines because we noticed that when fw1 was up it would not allow our OWA (web version of Microsoft Outlook) to work. So as a short-term fix, we just forced fw2 as the “Active Up” member of the HA pair. This seemed to work well until a new global policy was pushed that messed something up and caused the HA pair to flap. So we stopped fw1 via the Smart Dashboard GUI. This caused fw2 to stay active and things remained stable, but then all of a sudden a bunch of our NATs (used for outbound emails) failed! The NATs only began working again once fw1 was brought back up – eventhough it is still the standby member of the HA pair! So, my questions are:

1. Any thoughts on why/how this happened? Do the standby members of an HA pair still handle any traffic? They apparently are in our case, but is this “normal” or is something else messed up?!

2. Is there any way to force a synch of the members in an HA pair – ie, we know the two FWs have mismatched configs (since OWA App works when one FW is active and consistently fails when the other is active) but the checkpoint GUI just looks at the pair's overall policy as one-in-the-same. So is there a way to push a config from one member of an HA pair to another? Is this best done via the GUI or CLI?

3. I may just end up building a new machine and adding it to the XL cluster – any feedback or recommendations on this would be greatly appreciated since I have limited experience w/ HA in SPLAT

Thanks in advance for any/all feedback!
Regards, Leah

[northlandboy]

When you install policy, it will install policy on both nodes in the cluster - default is revert the installation on one if it fails on the other. Cluster nodes will start complaining if there is a different policy on each of them.

That is one reason why when you restart a member of a cluster, it pulls policy down from the other member, to check for differences.

ClusterXL is not keeping things like routing and proxy ARP in sync though. My guess is that someone's added stuff to the configuration of one node, and not the other.

You mention that OWA was failing when fw1 was up - what troubleshooting did you do? Where was it failing? Did you trace the traffic? If you've got a consistent failure, this should be trivial to trace.

Similarly with the NAT failures - what was actually happening? Was the firewall not natting the packets? Or were you not receiving any replies? What troubleshooting did you do? A random guess here would be that you've got proxy ARP configured for the NAT addresses on fw1, using the cluster MAC, but you haven't configured it on the secondary. This sort of thing is reasonably easy to troubleshoot - where were you seeing the failures?

A standby member should not handle traffic, unless you have got messed up routing somewhere - that's always a possibility, that you've got routes pointing to real addresses. I've seen that too. Was fw1 actually handling the traffic? Again, what did you see when looking at packets?

What would be the point in building a new machine? What problem are you trying to solve there?

[leahrev]

Thanks so much for your post! I haven't typically had a chance to take the time to do many traceroutes etc while these problems occured since many people were scrambling and just wanted the problem resolved ASAP. So I'm thinking about scheduling a maintenance window during which I'll force failover from fw2 to fw1 so that I can work on the OWA problem without people breathing down my neck.

As for the NAT issue - the NATs were in use for our email servers and when it started happening our email admin advised that all outbound mail destined for outside our network was queuing at the firewall (all inbound mail from outside companies were successfully delivered as well as emails sent from within the company to other addresses inside the company). In your post, you mentioned that it's possible that we have proxy ARP configured for the NAT on fw1 but not fw2. Pardon my ignorance, but where on the GUI and/or CLI can I check for this?

I think I'll hold off on building a second machine for now and continue troubleshooting the one we have in hopes that we can get the problem fixed.

Thanks for all your help!
Regards, Leah

[northlandboy]

arp -an

and yes, I think that scheduling a maintenance window would be a good idea, so that you can actually do useful troubleshooting steps like looking at logs, tcpdump, etc., rather than just taking random actions and hoping one of them resolves it.