CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


First, I hope you're all well and staying safe.
Second, I want to give a "heads up" that you should see more activity here shortly, and maybe a few cosmetic changes.
I'll post more details to the "Announcements" forum soon, so be on the lookout. -E

 

Results 1 to 13 of 13

Thread: When i install policy network goes down for 10-12 seconds

  1. #1
    Join Date
    2014-10-03
    Posts
    30
    Rep Power
    0

    Default When i install policy network goes down for 10-12 seconds

    I have noticed that when ever i add a new rule and install the policy on cluster XL nettwork goes down for about 10-12 seconds. Has anybody any idea about what could be the problem

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    15

    Default Re: When i install policy network goes down for 10-12 seconds

    Sounds like the spanning tree protocol on the switch the firewall is attached to thinks it is detecting a possible bridging loop and blocking traffic in listening/learning mode, 10-12 seconds sounds about right. You'll need to set portfast on the switch ports attached to the firewall.

    Also you may want to check if a failover occurred during the policy push, you can check this from the SmartView Tracker, in the All Records view filter for Control/wrench events (it is the very skinny column to the right of of the Origin column). If a failover is occurring that could explain the outage as well, you'll need to freeze the cluster state during policy install to keep that from happening. See sk32488.

  3. #3
    Join Date
    2014-10-03
    Posts
    30
    Rep Power
    0

    Default Re: When i install policy network goes down for 10-12 seconds

    But no switch over happens during policy install. i have checked the logs for control events. there has never happened any switchover when we install policy after changing some rules.

    But i think as you have mentioned it could be spanning tree problem. i will put the links as port fast and will check if that helps.

    thanks for the reply.

  4. #4
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    15

    Default Re: When i install policy network goes down for 10-12 seconds

    It is unlikely that spanning-tree is causing the outage if there is not a cluster transition but portfast is definitely worth a try first.

    Next culprit could be an overloaded CPU on the active member during the push. When installing policy you should see "Verifying" then "Verifying & Installing" then "Installing". I assume the 10-12 second outage occurs during the "Installing" phase? Try running a top on the active gateway during a policy push and see if your outage corresponds with the CPU getting pegged at 100% in sy and/or si space. If that is the case you can try modifying the Connection Persistence setting on the cluster object from "Rematch Connections" to "Keep all connections" which will significantly reduce CPU load; push policy twice and see if the outage is reduced or abated on the second policy install. Worse yet would be seeing the "wa/wio" utilization in top spiking which could indicate a shortage of RAM memory and causing the need for paging to virtual disk memory during policy installation which will absolutely butcher performance.

    If your outage is still occurring during policy load after all the above the last-ditch thing to try is powering off the standby member and installing policy with the For Gateway clusters install on all the members, if it fails do not install at all UNchecked on the policy install screen. If the outage goes away that would generally indicate some kind of bad interaction between the cluster and your network; could be ARP, could be multicast-related, could be lots of different things at Layer 2 or 3.
    Last edited by ShadowPeak.com; 2014-10-26 at 10:41.

  5. #5
    Join Date
    2014-10-03
    Posts
    30
    Rep Power
    0

    Default Re: When i install policy network goes down for 10-12 seconds

    I have tried by defineing the switch port as portfast and still got the network outage. The amazing thing is that network outage happens when i get the screen for policy install successfull, means when the policy install is complete and finished then network outage happens for 10-12 seconds, it does not happen during the verification or installation of policy.

  6. #6
    Join Date
    2013-09-25
    Location
    Bucharest
    Posts
    649
    Rep Power
    8

    Default Re: When i install policy network goes down for 10-12 seconds

    Quote Originally Posted by Opera View Post
    I have tried by defineing the switch port as portfast and still got the network outage. The amazing thing is that network outage happens when i get the screen for policy install successfull, means when the policy install is complete and finished then network outage happens for 10-12 seconds, it does not happen during the verification or installation of policy.
    Did you try what previous post hints?

  7. #7
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    15

    Default Re: When i install policy network goes down for 10-12 seconds

    Quote Originally Posted by Opera View Post
    I have tried by defineing the switch port as portfast and still got the network outage. The amazing thing is that network outage happens when i get the screen for policy install successfull, means when the policy install is complete and finished then network outage happens for 10-12 seconds, it does not happen during the verification or installation of policy.
    Either the high CPU load is continuing after the policy load is complete (probably due to the rematch) or there is some bad interaction with your network playing out. See my suggestions above and let us know how it goes.

  8. #8
    Join Date
    2012-10-03
    Posts
    72
    Rep Power
    9

    Default Re: When i install policy network goes down for 10-12 seconds

    check the /var/log/messages file for something like CUL_Freeze messages that correspond to the same time. I have this exact problem with an undersized cluster

  9. #9
    Join Date
    2006-09-26
    Posts
    3,199
    Rep Power
    18

    Default Re: When i install policy network goes down for 10-12 seconds

    Quote Originally Posted by DannyW View Post
    check the /var/log/messages file for something like CUL_Freeze messages that correspond to the same time. I have this exact problem with an undersized cluster
    I normally put this parameter in the $FWDIR/boot/modules/fwkern.conf file:

    fwha_freeze_state_machine_timeout=0x1E

    try that and see if it help.

  10. #10
    Join Date
    2014-11-20
    Posts
    2
    Rep Power
    0

    Default Re: When i install policy network goes down for 10-12 seconds

    Hi,

    Do you have a solution for this already?
    I've got the same problem at one of our customer's cluster but only with VPN traffic.
    I am awaiting a special and improved hotfix from Check Point now regarding this issue. Maybe I can help you afterwards if the issue still persists on your site.

    best regards,
    Werner

  11. #11
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    15

    Default Re: When i install policy network goes down for 10-12 seconds

    Quote Originally Posted by Werner View Post
    Hi,

    Do you have a solution for this already?
    I've got the same problem at one of our customer's cluster but only with VPN traffic.
    I am awaiting a special and improved hotfix from Check Point now regarding this issue. Maybe I can help you afterwards if the issue still persists on your site.

    best regards,
    Werner
    If your VPN tunnels are hanging upon policy push, it may be due to all IKE Phase 1 tunnels getting reset by default during the policy load process. Under Policy...Global Properties...SmartDashboard Customization...Advanced Config...Configure...VPN Advanced Propertries...VPN IKE Properties try checking "keep_IKE_SAs" which will override this behavior and see if it helps.

  12. #12
    Join Date
    2014-11-20
    Posts
    2
    Rep Power
    0

    Default Re: When i install policy network goes down for 10-12 seconds

    Quote Originally Posted by ShadowPeak.com View Post
    If your VPN tunnels are hanging upon policy push, it may be due to all IKE Phase 1 tunnels getting reset by default during the policy load process. Under Policy...Global Properties...SmartDashboard Customization...Advanced Config...Configure...VPN Advanced Propertries...VPN IKE Properties try checking "keep_IKE_SAs" which will override this behavior and see if it helps.
    That was also the first idea of the Check Point engineer. This option was actually set (maybe even by default?)
    But after doing a lot of debugs, Check Point has found the root cause why there are still drops. It has something to do with Link Selection when there are a lot of vpn tunnels terminating at the Gateway/Cluster. There is a fix for this mentioned in sk55244. This fix caused some problems with 3rd party vpn tunnels so be carefull when using it.


    I am still interested in the status of the issue mentioned in the first post anyway..

  13. #13
    Join Date
    2014-07-31
    Posts
    11
    Rep Power
    0

    Default Re: When i install policy network goes down for 10-12 seconds

    Does the same drop happen when the standby gateway is made the active? If not, it could just be your primary gateway that has the issue.

    I had this issue with a standalone Gaia gateway. I rebuilt it in the end and it was fine after that...

Similar Threads

  1. Install/Update SSL Network Extender without admin permission ?
    By ark.heidel in forum SNX - SSL Network Extender
    Replies: 0
    Last Post: 2013-02-26, 03:06
  2. Differences between policy install, database install etc
    By vbavbalist in forum SmartDashboard
    Replies: 3
    Last Post: 2010-05-28, 10:43
  3. Policy install fails for security policy with more than 4096 NAT rules
    By cciesec2006 in forum NAT (Network Address Translation)
    Replies: 8
    Last Post: 2009-06-07, 09:41
  4. SPLAT Network Install
    By fw1engineer in forum Check Point SecurePlatform (SPLAT)
    Replies: 0
    Last Post: 2006-05-16, 10:12
  5. Policy Editor Locks Up on Save or Policy Install
    By roadrunner in forum SmartDashboard
    Replies: 0
    Last Post: 2005-08-14, 12:03

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •