CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


First, I hope you're all well and staying safe.
Second, I want to give a "heads up" that you should see more activity here shortly, and maybe a few cosmetic changes.
I'll post more details to the "Announcements" forum soon, so be on the lookout. -E

 

Results 1 to 14 of 14

Thread: ClusterXL : connection drop when Policy Push

  1. #1
    Join Date
    2017-05-29
    Posts
    11
    Rep Power
    0

    Default ClusterXL : connection drop when Policy Push

    Hi,

    I have a problem on my CP 12200 Cluster.

    When i push a policy to the cluster, some connections are getting "dropped".

    Actually, i see between 200 & 400 WiFi access point (~30% of all the APs) losing their CapWap tunnels.
    I'am not sure i'am "losing" anything else, but this is the thing i can see because of the monitoring.

    I have no clue why it is happening, but i'am pretty sure i've never experienced that before upgrading to the R80.10 (Mgmt + firewalls).

    Does anyone have an idea regarding this issue?

    Thx

  2. #2
    Join Date
    2005-08-14
    Location
    Gig Harbor, WA, USA
    Posts
    2,499
    Rep Power
    18

    Default Re: ClusterXL : connection drop when Policy Push

    There is a setting that determines whether or not the connection table gets flushed on a policy install.
    This is set on the Gateway object under Other > Connection Persistence:
    Click image for larger version. 

Name:	Screen Shot 2017-06-20 at 12.06.12 PM.png 
Views:	563 
Size:	73.5 KB 
ID:	1289
    It's possible the connections aren't getting Rematched correctly--you should check the logs to see what errors show up on the dropped connections.
    In which case you may need to use the "Keep all connections" setting instead.
    http://phoneboy.org
    Unless otherwise noted, views expressed are my own

  3. #3
    Join Date
    2017-05-29
    Posts
    11
    Rep Power
    0

    Default Re: ClusterXL : connection drop when Policy Push

    Hi,

    Here the messages shown by the firewall when the policy is pushed :

    ;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:50813 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51119 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51115 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5246 -> y.y.y.y:51239 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;

    The messages show (in some case) either Client->Server or Server->Client packets getting dropped.

    As you mentionned, it looks like the connections are not well Rematched.

    I tried to check the "Keep connection when policy is installed" but only for the services involved in that case. It didn't work..
    I do not want to enable to enable this globablly.

    Any idea why these connections are not Rematched..?

    Thx for your reply!

  4. #4
    Join Date
    2005-08-14
    Location
    Gig Harbor, WA, USA
    Posts
    2,499
    Rep Power
    18

    Default Re: ClusterXL : connection drop when Policy Push

    UDP Session Timer is generally 40 seconds (meaning if no packets are seen, then the connection would be closed).
    You might try increasing the timer for that specific service to a larger value.
    http://phoneboy.org
    Unless otherwise noted, views expressed are my own

  5. #5
    Join Date
    2006-09-26
    Posts
    3,199
    Rep Power
    18

    Default Re: ClusterXL : connection drop when Policy Push

    Quote Originally Posted by PhoneBoy View Post
    UDP Session Timer is generally 40 seconds (meaning if no packets are seen, then the connection would be closed).
    You might try increasing the timer for that specific service to a larger value.
    I think I know what the issue is and I can help you fix it.

    since you've mentioned that you did NOT have this issue before upgrading to R80.10 and I've run into this issue as well, in R77.30, I think. I didn't think much of it until I see your post.

    The solution for you is to disable "Dynamic Dispatchers" aka DD. This feature does not exist until R77.x and it is OFF by default. Some geniuses from Checkpoint decide that it is such a good thing that they enable this feature by default in R80.10. It has given me some issues and when I turned off DD, the issue goes away. I have a ticket opened with Checkpoint for a few weeks now and haven't heard back from them :-(

    To turn off DD, please do the followings, assuming you have an H/A cluster:

    0- confirm the DD is on:
    [Expert@Power-1-S:0]# fw ctl multik get_mode
    Current mode is On
    [Expert@Power-1-S:0]#

    if DD is OFF, you will see this:
    [Expert@Power-1-S:0]# fw ctl multik get_mode
    Current mode is Off
    [Expert@Power-1-S:0]#

    1- on the Standby gateways, run this command in expert mode: fw ctl multik set_mode 0
    2- reboot the gateway,
    3- verify that that DD is off: fw ctl multik get_mode,
    4- once standby gateways is back online, perform this on the active gateways: clusterXL_admin down and clusterXL_admin up
    5- now that standby firewall in step #4 will be active and the active gateway in step #4 will be standby,
    6- perform fw ctl multik set_mode 0 on the standby gateway in step #5, and then reboot the gateway,
    7- "fw ctl multik get_mode" to confirm that DD is OFF,
    8- perform clusterXL_admin down and clusterXL_admin up on the active gateway in step #5
    9- Now you're back to the same state you were before you perform step #0 but now DD on both gateways is now OFF


    10- At the point, push the policy. I am highly confident that the connection will NOT drop when the policy is pushed.

    Let me know the result.

    Best of luck !!!!!
    Last edited by cciesec2006; 2017-06-21 at 21:14.

  6. #6
    Join Date
    2017-05-29
    Posts
    11
    Rep Power
    0

    Default Re: ClusterXL : connection drop when Policy Push

    Thank you !

    I will try this ASAP.

    I just verified the DD "mode" on all our firewalls and it's "Off" on R77.30 and "On" on R80.10.

    Sadly we have also 24 CP Standalone gateway, and it's gonna be harder to reboot them.

    Does this problem also impact Standalone appliances, or only Clusters?

    Thx a lot.

    EDIT : i forgot to mention, did you notice any drawbacks disabling this feature?
    Last edited by julzor; 2017-06-22 at 10:15.

  7. #7
    Join Date
    2006-09-26
    Posts
    3,199
    Rep Power
    18

    Default Re: ClusterXL : connection drop when Policy Push

    Quote Originally Posted by julzor View Post
    Thank you !

    I will try this ASAP.

    I just verified the DD "mode" on all our firewalls and it's "Off" on R77.30 and "On" on R80.10.

    Sadly we have also 24 CP Standalone gateway, and it's gonna be harder to reboot them.

    Does this problem also impact Standalone appliances, or only Clusters?

    Thx a lot.

    EDIT : i forgot to mention, did you notice any drawbacks disabling this feature?
    DD is independent of cluster. It is one of the technologies that has been promoted by Checkpoint SEs without much understanding the potential down side of this. It sound good in theory but sucks in practice.

    The drawbacks of disabling this features might be one of your CPUs might be running hotter and other CPUs, if you have CoreXL enabled.

    I would stay away from DD until Checkpoint R90, after it has worked out all the "bugs".

    Best of luck and please keep me posted whether this fixes your issue.

  8. #8
    Join Date
    2017-05-29
    Posts
    11
    Rep Power
    0

    Default Re: ClusterXL : connection drop when Policy Push

    Hi again,

    I tried to disable DD this morning on my cluster.

    Everything went fine, but sadly it didn't work : i still lose some Access Points when a Policy is Pushed :-(

    I still get these messages :

    [...]
    ;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:22871 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:35682 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:6957 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    ;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 x.x.x.x:6957 -> y.y.y.y:5246 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
    [...]

    I really don't understand why it's being dropped since i configured the CapWap Service Object (port 5246 & 5247) to always keep the established connections.


    I will try to put a higher UDP time-out just to see if i notice any differences, and maybe try to enable globally the "Keep established connection" when a new policy is installed. Just to see if it works...

    Thank you for your support anyway, at least i learned about this DD things.

    EDIT : i tried to enable "Keep all connections" globally, and everything works fine : i didn't lost a single session.
    I only had a few messages like these ones, but i guess it's normal :

    ;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:5247 -> y.y.y.y:18876 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;
    ;[cpu_0];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:7152 -> y.y.y.y:5247 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;
    ;[cpu_0];[fw4_0];fw_log_drop_ex: Packet proto=17 x.x.x.x:7152 -> y.y.y.y:5247 dropped by fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized;

    I will not let this behavior like this, i'am switching back to "Rematch Connection" for security reason, so i'am still trying to figure out why this happens.
    Last edited by julzor; 2017-06-23 at 08:40.

  9. #9
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,668
    Rep Power
    11

    Default Re: ClusterXL : connection drop when Policy Push

    Did you re-enable Dynamic Dispatcher? If you didn't have an issue with it i would leave it on. The only issue someone has reported on here is they couldn't ping or ssh a remote interface on the standby.

  10. #10
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,668
    Rep Power
    11

    Default Re: ClusterXL : connection drop when Policy Push

    I don't know if this could be part of the issue or not. Some of the docs talk about clearing the state table after making this change. Maybe the open udp connection isn't getting the "keep connection open" attribute.

    Do you happen to have a lab to try to replicate this in? It seems like this would be painful to test in production. This should be easy to test with VMs. You can use the unix untility netcat to send random data on a given udp port. You could setup a cluster and 2 unix nodes on both sides to throw the udp data. I'd set a ipfilter rule on the destination to drop it. Or just set a static arp on the firewall to throw the destination to never never land (just any static arp to a destination that doesn't exist).

    Then you can debug and try whatever without effecting production. I'd also keep the global setting on until you figure this out. I'm assuming you're not writing that many deny rules anyway, which is the only case where you wouldn't want that.

  11. #11
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    16

    Default Re: ClusterXL : connection drop when Policy Push

    There were quite a few enhancements to SecureXL in R80.10, and SecureXL has to resync its own copies of various state tables to the main ones managed by the Firewall Worker Cores when policy is reinstalled. It is possible that something in this internal resync is going awry; might be interesting to try disabling SecureXL with rematch still set and see if the issue persists.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  12. #12
    Join Date
    2017-06-26
    Posts
    1
    Rep Power
    0

    Default Re: ClusterXL : connection drop when Policy Push

    Quote Originally Posted by cciesec2006 View Post
    DD is independent of cluster. It is one of the technologies that has been promoted by Checkpoint SEs without much understanding the potential down side of this. It sound good in theory but sucks in practice.

    The drawbacks of disabling this features might be one of your CPUs might be running hotter and other CPUs, if you have CoreXL enabled.

    I would stay away from DD until Checkpoint R90, after it has worked out all the "bugs".

    Best of luck and please keep me posted whether this fixes your issue.

    Dear Checkpoint user,

    My name is Nir and I'm the manager of the RnD group responsible for the CoreXL Dynamic Dispatcher functionality.
    This functionality had been enabled by numerous R77.30 customers and we have full confidence in it as default configuration in R80.10 Security Gateway.
    Nonetheless if there are issues with this feature I'm welcoming you to contact me directly - I will be happy to assist with the closure of such tickets/SRs.

    Thanks,
    Nir Rotgold
    Framework Group manager @ Checkpoint
    nirro@checkpoint.com

  13. #13
    Join Date
    2017-05-29
    Posts
    11
    Rep Power
    0

    Default Re: ClusterXL : connection drop when Policy Push

    Hi,

    Thank you everybody for your replies.

    Yes, i have re-activated DD since it didn't solve the issue; it doens't look related to this feature.

    I tried to enable "Keep all connections" for the whole gateway and it solved the problem.
    I don't have any weird Drops anymore...

    Of course, i can't let this like that, i want my gateway to rematch trafic when a policy changed occur.

    So it really looks like the problem is coming from the "Rematch" things, somehow.

    It also means that when i checked "Keep all connections" for the services involved (and not the whole gateway), it didn't take it into account.

    Am i missing something?

    I'am kinda stuck on this.

    Will open a case soon.

    Thx!

  14. #14
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    16

    Default Re: ClusterXL : connection drop when Policy Push

    Quote Originally Posted by julzor View Post
    Hi,

    Thank you everybody for your replies.

    Yes, i have re-activated DD since it didn't solve the issue; it doens't look related to this feature.

    I tried to enable "Keep all connections" for the whole gateway and it solved the problem.
    I don't have any weird Drops anymore...

    Of course, i can't let this like that, i want my gateway to rematch trafic when a policy changed occur.

    So it really looks like the problem is coming from the "Rematch" things, somehow.

    It also means that when i checked "Keep all connections" for the services involved (and not the whole gateway), it didn't take it into account.

    Am i missing something?

    I'am kinda stuck on this.

    Will open a case soon.

    Thx!
    As I mentioned earlier in the thread, it could be a SecureXL sync issue, what happens with rematch set and SecureXL disabled?
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

Similar Threads

  1. DHCP drop on ClusterXL after R80.10 upgrade
    By julzor in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 5
    Last Post: 2017-06-01, 08:15
  2. Exchange Connection Drop during Policy Push
    By DHewes in forum Miscellaneous
    Replies: 3
    Last Post: 2011-09-15, 05:15
  3. Drop in-progress connection
    By ttran in forum Check Point Safe@Office Appliances
    Replies: 0
    Last Post: 2009-11-06, 02:56
  4. Occasional Drop of Connection RDP
    By menz456 in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 5
    Last Post: 2008-11-07, 13:56
  5. Sqlnet2 data connection drop after policy install
    By Hitman in forum Services (TCP, UDP, ICMP, etc.)
    Replies: 6
    Last Post: 2008-06-04, 14:13

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •