Timeouts on Nokia cluster failover

[ddurocher]

I have an issue with a redundant firewall configuration. I am using 2 Nokia IP380 devices both running IPSO 4.0 and Checkpoint R60.

The basic design is shown below. Please ignore the dots, it was the only way I could get the spacing to work.

SVR1--SW1--FW1--SW4-->to other networks
....\/.|....|....|
..../\.|....|....|
SVR2--SW2--FW2--SW5-->to other networks

where,
SVR = Server
SW = Switch
FW = Firewall

Servers 1 and 2 are using NIC teaming to connecting to Switches 1 and 2.
There is a trunk between Switches 1 and 2 and Swtiches 4 and 5.
Default MAC timeout on all switches is 300 seconds.
The connected networks are all private and there is no NATing.

The problem is that if FW1, for example, is the master node and it fails (powered off) Server 2 will take up to 5 minutes to recover. My assumption is that this is due to the MAC timeout of 300s since if the MAC tables are cleared on the switches the connections timing out recover immediately. However, Server 1 recovers immediately. I expect this is most likely due to SW2 still trying to pass traffic to SW1 to reach the cluster IP since that is the info in its MAC table. The same is also true if FW2 is the master then Server 1 has the same issues reconnecting in a timely fashion.

From my research I understand that the Nokia box that assumes control of the cluster should be sending out a gratuitous ARP to notify devices on the network of the change. So perhaps the ARP isn't being passed over the trunk ports. I'm also wondering if our physical design is wrong and we should add links like SW1 <-> FW2, SW2 <-> FW1.

Has anyone ever seen an issue like this in a redundant network design?

[Olivn]

VRRP or IPcluster ?
Multicast or Unicast ?

[northlandboy]

Sadly it's been a few years since I did much work with IPSO, but I'm having flashbacks to issues with VRRP and gratuitous ARP.

From memory, the firewall needs to be able to send a gratuitous ARP as the virtual IP. I think we had to ensure that our cluster objects were set up correctly, with all the virtual IPs - we hadn't always had that previously, as cluster configuration was a little different back in the pre-R55 days.

I would try triggering a failover, and use tcpdump to see if a gratuitous ARP is being sent. This SHOULD update all ports, including trunk ports. It is definitely a valid network configuration that you have.

[z3ndrag0n]

Does your switch config include portfast (Note the assumption of Cisco switches here!)?

Cheers,
Barry

[msjouw]

Or have the ports been set with port security, this could also have a time-out of 5 minutes for a not allowed second mac address on the same port, when the clustermember becomes master => 2 mac addresses from the same port...
Just my 2cts

[ddurocher]

Hello all, Thanks for the responses. To answer some questions I'm using IP Cluster in this configuration. Connected switches are not Cisco, unfortunately, but Alcatel devices. There is no port security configured but I'll have to look into whether or not they have something enabled by default. I want to do some packet captures on the network as well but I'm about 5000km away from the equipment so I need to coordinate that one with our site folks. Thanks, Darren

[ddurocher]

Well it has been a while but I did finally get this problem sorted out. Had to go to the UK (where the equipment is installed) to fix it. So I wanted to take a moment to post some notes on what I learned in case this will help someone else down the road.

Note that for my purposes I chose to reconfigure the cluster to use VRRP instead of IP clustering as I don't need the load balancing.

1. I had some issues with routing that I wasn't expecting. Ultimately it was the nodes external to the network were forwarding packets to the VLAN IP address of my external switches instead of the Virtual IP of my firewall.

2. Under the VMAC Mode configuration I used a combination of either Interface or Extended. My internal network switches are Alcaltel 6100 series switches and Interface mode allowed them to handle the failover properly. Externally we are using Alcaltel 6600 series switches and again I used Interface mode except on one VLAN with two directly connected devices which worked only with Extended mode. This was just some trial and error tweaking once I got the other issues resolved.

I initially deviated from the default VRRP option because I noticed that all of my VIPs had been assigned the same virtual MAC address. I thought that might be causing some issues based on the network architecture. I never did test to see if VRRP mode would work once I had the routing issues sorted out so take this with a grain of salt.

3. Also, something I hadn't realized initially was that I needed to have a firewall rule allowing vrrp and igmp between the firewall nodes and the multicast network.

4. The clocks being out of sync caused some issues briefly as well. This wasn't a major issue but keep this one in mind and save some hair!

At the end of the day this ended up being a exercise in understanding not only the Nokia and Check Point Products but how they need to be massaged into the network design as a whole, and how the different devices we are using affect each other.

So after much teeth gnashing I have a VRRP enabled cluster that fails over and back beautifully. Thanks to everyone's posts I read on here for all the little issues I had to sort out.

Darren

[northlandboy]

Originally Posted by ddurocher

I initially deviated from the default VRRP option because I noticed that all of my VIPs had been assigned the same virtual MAC address. I thought that might be causing some issues based on the network architecture. I never did test to see if VRRP mode would work once I had the routing issues sorted out so take this with a grain of salt.

Thanks for letting us know you've got things working.

Regards having the same MAC, it's not a problem since all the VIPs are in different VLANs, right? Switches are fine with having a the same MAC appear in different VLANs.

It only becomes a problem when you've got multiple VRRP clusters in the same VLAN, with the same VRID (which is used to create the virtual MAC).

[ddurocher]

Yes, all of the VIPs are in different VLANs, I was thinking along those lines but to be honest I just didn't want to push my luck after I got the thing working. :)