ClusterXL standby node strange behavior

[Mariusz1]

Hello.

We have two ClusterXL Gaia R77.30 identical nodes with following hotfixes installed:
• Jumbo Hotfix Accumulator General Availability for R77.30 Take 216
• HOTFIX_R77_30
Implicitly Installed Hotfixes (installed as part of another hotfix):
• Check_Point_R77_30_JUMBO_HF_1_Bundle_T205_FULL.tgz

Acceleration CoreXL and SecureXL are enabled on both nodes.

Cluster works in HA mode:
[Expert@gw2:0]# cphaprob stat
Cluster Mode: High Availability (Primary Up) with IGMP Membership
Number Unique Address Assigned Load State
1 172.16.0.2 100% Active
2 (local) 172.16.0.3 0% Standby

Problem:
Few days ago our stanby node (gw2) "automagically" - without any admin intervention - has started to forward some traffic (about 1000 pkt/s) and it's CPU load (as shown by SmartViewMonitor->System Counters->FireWall History) raised up to 30 - 40 %. The active node gw1 is proccessing much, much more traffic - about 20 000 pkt/s by 70% CPU. At the same time icmp probes (ping) time for traffic coming through the cluster raised from few ms to about 60-80 ms.

Gw2 behaves strange as it would work in LS pivot mode (?!), although is in HA for sure. We have installed security policies few times and even reboot both nodes (of course not simultanously), but it is still the same issue. What's wrong with gw2? If we switch gw2 from standby to active, than gw1 behaves the same (1000 pkt/s, 30 - 40 % CPU).

Regards
Mariusz1

[Chili]

Hi Mariusz,

My Check Point partner told me there are several instabilities in Jumbo 216 that are already fixed in Jumbo 225.

...but I have no details.

BR
Sven


Gesendet von iPhone mit Tapatalk

[cciesec2006]

Hi Mariusz,

My Check Point partner told me there are several instabilities in Jumbo 216 that are already fixed in Jumbo 225.

...but I have no details.

BR
Sven


Gesendet von iPhone mit Tapatalk

I looked at the released notes https://supportcenter.checkpoint.com...Ongoing%20Take and I couldn't find anything relate to this issue.

it is a big concern for me as well, because I am also running JHFA 216 :-(

[jflemingeds]

Can you look at the router on the inside of that firewall and check its arp table and route table?

What you're looking for is the following.

Do you have any routes pointing to fw02's interface directly (instead of the VIP ( i know just making sure))

Next check the arp table and make sure all the next hop gateway address have the mac address of the active member.

I think VMAC will show in cphaprob stat output, but can you confirm if your using VMAC or not?


If that doesn't turn up anything i would look at tcpdump -ni $INTERFACE -w output.cap 'host $IP'

replace $INTERFACE with the interface seeing the traffic and replace $IP with an IP your seeing go to the standby. Pull this into wireshark and check the src and dst mac address of the packets to see if they help uncover who is send the traffic to fw02.


Maybe that will help uncover something. The other option would be diabling securexl and doing a firewall montior on both firewalls at the same time to see if you can verify if trafffic is really being pivoted, but you'll need to do that on off peak hours as disabling securexl can jack cpu.

[Mariusz1]

Frist of all it appeared before T216 install. We have installed T216 in hope maybe it will resolve our issue. Yes, I confirm - we are using VMAC, but it works fine.

By the way Gaia Portal (GUI) still shows:
Check Point Upgrade Service Engine (CPUSE) | R77.30 take 204 Hotfixes | Last updated on: Fri May 19 8:08 2017
and on Overview:
Kernel: 2.6.18-92cpx86_64
Edition: 64-bit
Build Number: 204

Why 204 and not 216?

Besides on gw2 in Gaia Portal on Upgrades (CPUSE)->Status and Actions page we have "Check for Updates", "Import Package" and "Add hotfixes from the cloud" buttons on the upper right, but in gw1 we have only the last one (and "Import package" is in More submenu).

Regards
Mariusz1

[jflemingeds]

ah ok, so changing gears. Run the following to try to figure out the take version difference.

installed_jumbo_take
/opt/CPinfo-10/bin/cpinfo -y all

compare the output of the two.

might as well post all the cphaprob commands from both members as well

cphaprob stat
cphaprob -i list
cphaprob -a if

[Mariusz1]

On both gateways we have:
[Expert@gw1:0]# installed_jumbo_take
R77.30 Jumbo Hotfix Accumulator take_216 is installed, see sk106162.
[Expert@gw1:0]# /opt/CPinfo-10/bin/cpinfo -y all

This is Check Point CPinfo Build 914000173 for GAIA
[FW1]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF

FW1 build number:
This is Check Point's software version R77.30 - Build 048
kernel: R77.30 - Build 048

[SecurePlatform]
HOTFIX_R77_30_JUMBO_HF

[CPinfo]
No hotfixes..

[PPACK]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF

[CVPN]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF

[CPUpdates]
BUNDLE_R77_30_JUMBO_HF

[DIAG]
HOTFIX_R77_30

[rtm]
No hotfixes..
##########################
Important:

It seems like gw2 is stepping in when CPU load on gw1 is about/above 80%. Now it's about 65% on gw1 and gw2 is normal - about 4%.
Is it Cluster Under Load (CUL) mechanism?

Regards
Mariusz1