ClusterXL : connection drop when Policy Push

[julzor]

Hi,

I have a problem on my CP 12200 Cluster.

When i push a policy to the cluster, some connections are getting "dropped".

Actually, i see between 200 & 400 WiFi access point (~30% of all the APs) losing their CapWap tunnels.
I'am not sure i'am "losing" anything else, but this is the thing i can see because of the monitoring.

I have no clue why it is happening, but i'am pretty sure i've never experienced that before upgrading to the R80.10 (Mgmt + firewalls).

Does anyone have an idea regarding this issue?

Thx

[PhoneBoy]

There is a setting that determines whether or not the connection table gets flushed on a policy install.
This is set on the Gateway object under Other > Connection Persistence:

It's possible the connections aren't getting Rematched correctly--you should check the logs to see what errors show up on the dropped connections.
In which case you may need to use the "Keep all connections" setting instead.

[julzor]

Hi,

Here the messages shown by the firewall when the policy is pushed :

;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:50813 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51119 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51115 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51239 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;

The messages show (in some case) either Client->Server or Server->Client packets getting dropped.

As you mentionned, it looks like the connections are not well Rematched.

I tried to check the "Keep connection when policy is installed" but only for the services involved in that case. It didn't work..
I do not want to enable to enable this globablly.

Any idea why these connections are not Rematched..?

Thx for your reply!

[PhoneBoy]

UDP Session Timer is generally 40 seconds (meaning if no packets are seen, then the connection would be closed).
You might try increasing the timer for that specific service to a larger value.

[cciesec2006]

UDP Session Timer is generally 40 seconds (meaning if no packets are seen, then the connection would be closed).
You might try increasing the timer for that specific service to a larger value.

I think I know what the issue is and I can help you fix it.

since you've mentioned that you did NOT have this issue before upgrading to R80.10 and I've run into this issue as well, in R77.30, I think. I didn't think much of it until I see your post.

The solution for you is to disable "Dynamic Dispatchers" aka DD. This feature does not exist until R77.x and it is OFF by default. Some geniuses from Checkpoint decide that it is such a good thing that they enable this feature by default in R80.10. It has given me some issues and when I turned off DD, the issue goes away. I have a ticket opened with Checkpoint for a few weeks now and haven't heard back from them :-(

To turn off DD, please do the followings, assuming you have an H/A cluster:

0- confirm the DD is on:
[Expert@Power-1-S:0]# fw ctl multik get_mode
Current mode is On
[Expert@Power-1-S:0]#

if DD is OFF, you will see this:
[Expert@Power-1-S:0]# fw ctl multik get_mode
Current mode is Off
[Expert@Power-1-S:0]#

1- on the Standby gateways, run this command in expert mode: fw ctl multik set_mode 0
2- reboot the gateway,
3- verify that that DD is off: fw ctl multik get_mode,
4- once standby gateways is back online, perform this on the active gateways: clusterXL_admin down and clusterXL_admin up
5- now that standby firewall in step #4 will be active and the active gateway in step #4 will be standby,
6- perform fw ctl multik set_mode 0 on the standby gateway in step #5, and then reboot the gateway,
7- "fw ctl multik get_mode" to confirm that DD is OFF,
8- perform clusterXL_admin down and clusterXL_admin up on the active gateway in step #5
9- Now you're back to the same state you were before you perform step #0 but now DD on both gateways is now OFF


10- At the point, push the policy. I am highly confident that the connection will NOT drop when the policy is pushed.

Let me know the result.

Best of luck !!!!!

[julzor]

Thank you !

I will try this ASAP.

I just verified the DD "mode" on all our firewalls and it's "Off" on R77.30 and "On" on R80.10.

Sadly we have also 24 CP Standalone gateway, and it's gonna be harder to reboot them.

Does this problem also impact Standalone appliances, or only Clusters?

Thx a lot.

EDIT : i forgot to mention, did you notice any drawbacks disabling this feature?

[cciesec2006]

Thank you !

I will try this ASAP.

I just verified the DD "mode" on all our firewalls and it's "Off" on R77.30 and "On" on R80.10.

Sadly we have also 24 CP Standalone gateway, and it's gonna be harder to reboot them.

Does this problem also impact Standalone appliances, or only Clusters?

Thx a lot.

EDIT : i forgot to mention, did you notice any drawbacks disabling this feature?

DD is independent of cluster. It is one of the technologies that has been promoted by Checkpoint SEs without much understanding the potential down side of this. It sound good in theory but sucks in practice.

The drawbacks of disabling this features might be one of your CPUs might be running hotter and other CPUs, if you have CoreXL enabled.

I would stay away from DD until Checkpoint R90, after it has worked out all the "bugs".

Best of luck and please keep me posted whether this fixes your issue.

[julzor]

Hi again,

I tried to disable DD this morning on my cluster.

Everything went fine, but sadly it didn't work : i still lose some Access Points when a Policy is Pushed :-(

I still get these messages :

[...]
;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:22871 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:35682 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:6957 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:6957 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
[...]

I really don't understand why it's being dropped since i configured the CapWap Service Object (port 5246 & 5247) to always keep the established connections.


I will try to put a higher UDP time-out just to see if i notice any differences, and maybe try to enable globally the "Keep established connection" when a new policy is installed. Just to see if it works...

Thank you for your support anyway, at least i learned about this DD things.

EDIT : i tried to enable "Keep all connections" globally, and everything works fine : i didn't lost a single session.
I only had a few messages like these ones, but i guess it's normal :

;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5247 -> y.y.y.y:18876 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;
;[cpu_0];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:7152 -> y.y.y.y:5247 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;
;[cpu_0];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:7152 -> y.y.y.y:5247 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;

I will not let this behavior like this, i'am switching back to "Rematch Connection" for security reason, so i'am still trying to figure out why this happens.

[jflemingeds]

Did you re-enable Dynamic Dispatcher? If you didn't have an issue with it i would leave it on. The only issue someone has reported on here is they couldn't ping or ssh a remote interface on the standby.

[jflemingeds]

I don't know if this could be part of the issue or not. Some of the docs talk about clearing the state table after making this change. Maybe the open udp connection isn't getting the "keep connection open" attribute.

Do you happen to have a lab to try to replicate this in? It seems like this would be painful to test in production. This should be easy to test with VMs. You can use the unix untility netcat to send random data on a given udp port. You could setup a cluster and 2 unix nodes on both sides to throw the udp data. I'd set a ipfilter rule on the destination to drop it. Or just set a static arp on the firewall to throw the destination to never never land (just any static arp to a destination that doesn't exist).

Then you can debug and try whatever without effecting production. I'd also keep the global setting on until you figure this out. I'm assuming you're not writing that many deny rules anyway, which is the only case where you wouldn't want that.

[ShadowPeak.com]

There were quite a few enhancements to SecureXL in R80.10, and SecureXL has to resync its own copies of various state tables to the main ones managed by the Firewall Worker Cores when policy is reinstalled. It is possible that something in this internal resync is going awry; might be interesting to try disabling SecureXL with rematch still set and see if the issue persists.

[nirro]

DD is independent of cluster. It is one of the technologies that has been promoted by Checkpoint SEs without much understanding the potential down side of this. It sound good in theory but sucks in practice.

The drawbacks of disabling this features might be one of your CPUs might be running hotter and other CPUs, if you have CoreXL enabled.

I would stay away from DD until Checkpoint R90, after it has worked out all the "bugs".

Best of luck and please keep me posted whether this fixes your issue.

Dear Checkpoint user,

My name is Nir and I'm the manager of the RnD group responsible for the CoreXL Dynamic Dispatcher functionality.
This functionality had been enabled by numerous R77.30 customers and we have full confidence in it as default configuration in R80.10 Security Gateway.
Nonetheless if there are issues with this feature I'm welcoming you to contact me directly - I will be happy to assist with the closure of such tickets/SRs.

Thanks,
Nir Rotgold
Framework Group manager @ Checkpoint
nirro@checkpoint.com

[julzor]

Hi,

Thank you everybody for your replies.

Yes, i have re-activated DD since it didn't solve the issue; it doens't look related to this feature.

I tried to enable "Keep all connections" for the whole gateway and it solved the problem.
I don't have any weird Drops anymore...

Of course, i can't let this like that, i want my gateway to rematch trafic when a policy changed occur.

So it really looks like the problem is coming from the "Rematch" things, somehow.

It also means that when i checked "Keep all connections" for the services involved (and not the whole gateway), it didn't take it into account.

Am i missing something?

I'am kinda stuck on this.

Will open a case soon.

Thx!

[ShadowPeak.com]

Hi,

Thank you everybody for your replies.

Yes, i have re-activated DD since it didn't solve the issue; it doens't look related to this feature.

I tried to enable "Keep all connections" for the whole gateway and it solved the problem.
I don't have any weird Drops anymore...

Of course, i can't let this like that, i want my gateway to rematch trafic when a policy changed occur.

So it really looks like the problem is coming from the "Rematch" things, somehow.

It also means that when i checked "Keep all connections" for the services involved (and not the whole gateway), it didn't take it into account.

Am i missing something?

I'am kinda stuck on this.

Will open a case soon.

Thx!

As I mentioned earlier in the thread, it could be a SecureXL sync issue, what happens with rematch set and SecureXL disabled?