CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


CPUG Challenge 2018?? We will be holding another CPUG Challenge for 2018.
The plan is to time it around CPX again (earlier this year), but not necessarily limit it to those in attendance.
I'll provide more details as we get a bit closer, but be ready! -E

 

Page 2 of 3 FirstFirst 123 LastLast
Results 21 to 40 of 43

Thread: Freezes/Lock-Out on our firewall that have CP puzzled.

  1. #21
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    103
    Rep Power
    11

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    The issue is very much firewall related I believe. After many lines of further investigation on our network, DNS etc, I pretty confident now it is the box.

    Aside from ARP, Any other suggestions? I would increase the ARP cache, but I dont really see why the current ARP cache isnt big enough if you understand me?
    Firewall-the-box, sure. Doesnít feel like a firewall-the-software issue to me, though. Feels more like an OS issue.

    Is the system able to respond to traffic during the issue?

    How do you run the cpstop/cpstart? Physical console/LOM, or are you able to connect to the system with SSH?

    Do any connections through the system work during the issue?
    Zimmie

  2. #22
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,088
    Rep Power
    12

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    The issue is very much firewall related I believe. After many lines of further investigation on our network, DNS etc, I pretty confident now it is the box.

    Aside from ARP, Any other suggestions? I would increase the ARP cache, but I dont really see why the current ARP cache isnt big enough if you understand me?
    Try turning off SecureXL during a problem period (fwaccel off) and see if that instantly resolves it, that helps pin down specifically where the problem is. You could preemptively turn it off and see if the situation does not recur, but that isn't as conclusive as turning it off during a problem period and instantly having it fix everything.
    --
    My Book "Max Power: Check Point Firewall Performance Optimization"
    Second Edition Coming Soon

  3. #23
    Join Date
    2006-09-26
    Posts
    3,055
    Rep Power
    15

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by Bob_Zimmerman View Post
    Firewall-the-box, sure. Doesnít feel like a firewall-the-software issue to me, though. Feels more like an OS issue.

    Is the system able to respond to traffic during the issue?

    How do you run the cpstop/cpstart? Physical console/LOM, or are you able to connect to the system with SSH?

    Do any connections through the system work during the issue?
    OS issue? Isn't checkpoint responsible for both the OS and the application itself since it is now GAIA?

    When I had something very similar to this a few months back, I have clusterXL and not VRRP, the traffics to and from the firewalls were fine. Any traffics that had to traverse the firewall just fell into a black hole. cpstop/cpstart didn't help, only a reboot did.

  4. #24
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    103
    Rep Power
    11

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by cciesec2006 View Post
    OS issue? Isn't checkpoint responsible for both the OS and the application itself since it is now GAIA?
    Well yes, obviously, but look at the title of this thread. Check Point support evidently doesn’t know what’s going on.

    This is one of the things I dislike about closed-source software. When an outage threatens my job, I would much rather have a way to fix a problem myself than have “a throat to choke”, to use the common phrase.

    Edited to add: Also, firewall-the-software issues and OS-level issues are troubleshot with wildly different tools. At least half of fixing any IT issue is figuring out where it is so you know which tools to use to find out what it is.

    Quote Originally Posted by cciesec2006 View Post
    When I had something very similar to this a few months back, I have clusterXL and not VRRP, the traffics to and from the firewalls were fine. Any traffics that had to traverse the firewall just fell into a black hole. cpstop/cpstart didn't help, only a reboot did.
    That sounds like the OS forgot it should be forwarding IP. The box’s boot config then reenabled it.
    Last edited by Bob_Zimmerman; 3 Weeks Ago at 21:43.
    Zimmie

  5. #25
    Join Date
    2006-09-26
    Posts
    3,055
    Rep Power
    15

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by Bob_Zimmerman View Post
    That sounds like the OS forgot it should be forwarding IP. The boxís boot config then reenabled it.
    Funny you said that because, at that time when the gateways stopped forwarding traffics and I was able to ssh into the box. I checked the ip forwarding flag and it was set to enable forwarding at the value of 1:
    cat /proc/sys/net/ipv4/ip_forward
    1

    I was under the gun so I had no choice but reboot the box.

  6. #26
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,480
    Rep Power
    8

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    I haven't seen this asked yet (or i missed it). Have you noticed if this happens on both firewalls or is it only happening when of the firewalls is active?

  7. #27
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Morning all -

    We only see it on one box currently but we haven't tried to flip it over to the other to test. As much that could fix it, it doesnt really fix that we would still have a faulty box in backup if you get me? Id rather pinpoint and isolate the issue, and the only way to do this is to let it happen!

  8. #28
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    We 'broke' the firewall in a controlled way last night, through pushing the policy.

    From the debug reports, and after filtering out all the rule drops, we had these CPU messages amongst the policy installation messages?

    Anything worth to note or just normal?

    [cpu_2] [fw4_1] FW-1: [cul_load_freeze][CUL - Cluster] Setting CUL FREEZE_ON, high kernel CPU usage (84%) on local Member 0, threshold = 80%
    [cpu_2] [fw4_1] FW-1: [cul_load_freeze][CUL - Cluster] CUL should be OFF (short timeout of 10 seconds expired) but at least one member reported high CPU usage 7 seconds ago
    [cpu_2] [fw4_1] FW-1: [cul_load_freeze][CUL - Cluster] CUL should be OFF (short timeout of 10 seconds expired) but at least one member reported high CPU usage 8 seconds ago
    [cpu_2] [fw4_1] FW-1: [freeze_on_remote] freeze state on remote member 1 has changed from 0 to 1
    [cpu_2] [fw4_1] fwioctl: Policy has started. Extending dead timeouts
    [cpu_2] [fw4_1] FW-1: [cul_policy_freeze][CUL - Member] fwha_cul_policy_freeze_state_change: set Policy Freeze [ON], FREEZING state machine at ACTIVE (time=191684405, caller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE, freeze_timeout=300, freeze_event_timeout=150)
    [cpu_2] [fw4_1] FW-1: [cul_load_freeze][CUL - Cluster] CUL should be OFF (short timeout of 10 seconds expired) but at least one member reported high CPU usage 9 seconds ago
    [cpu_2] [fw4_1] FW-1: [cul_load_freeze][CUL - Cluster] Changing CUL state to OFF, no Member reported high CPU usage for the past 10 seconds, CUL was ON for 13 seconds
    [cpu_2] [fw4_1] FW-1: [freeze_on_remote] freeze state on remote member 1 has changed from 1 to 0
    [cpu_1] [fw4_1] fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_policy_done_time=191684704)
    [cpu_1] [fw4_1] FW-1: [CUL - Member] Policy Freeze mechanism disabled, Enabling state machine at 4 (time=191684704, caller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE)
    [cpu_0] [fw4_1] eth7

  9. #29
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,480
    Rep Power
    8

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Need to know if both firewalls have the same problem or not.

    Also dmesg from both.

    What is that very last line about eth7? Was something cut off?

  10. #30
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by jflemingeds View Post
    Need to know if both firewalls have the same problem or not.

    Also dmesg from both.

    What is that very last line about eth7? Was something cut off?
    nothing after apart from dropped traffic after the eth7. Nothing else shows.

    dmesg:


    [fw4_1];fwioctl: Policy has started. Extending dead timeouts
    [fw4_1];FW-1: [cul_policy_freeze][CUL - Member] fwha_cul_policy_freeze_state_cha nge: set Policy Freeze [ON], FREEZING state machine at ACTIVE (time=191678415, c aller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE, freeze_timeout=300, freeze_event_ti meout=150)
    [fw4_0];FW-1: monitor filter loaded
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=191678415)
    [fw4_1];FW-1: [CUL - Member] Policy Freeze mechanism disabled, Enabling state ma chine at 4 (time=191678415, caller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE)
    [fw4_1];fwioctl: Policy has started. Extending dead timeouts
    [fw4_1];FW-1: [cul_policy_freeze][CUL - Member] fwha_cul_policy_freeze_state_cha nge: set Policy Freeze [ON], FREEZING state machine at ACTIVE (time=191678415, c aller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE, freeze_timeout=300, freeze_event_ti meout=150)
    [fw4_1];FW-1: monitor filter loaded
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=191678415)
    [fw4_1];FW-1: [CUL - Member] Policy Freeze mechanism disabled, Enabling state ma chine at 4 (time=191678415, caller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE)
    [fw4_1];fwioctl: Policy has started. Extending dead timeouts
    [fw4_1];FW-1: [cul_policy_freeze][CUL - Member] fwha_cul_policy_freeze_state_cha nge: set Policy Freeze [ON], FREEZING state machine at ACTIVE (time=191678416, c aller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE, freeze_timeout=300, freeze_event_ti meout=150)
    [fw4_2];FW-1: monitor filter loaded
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=191678416)
    [fw4_1];FW-1: [CUL - Member] Policy Freeze mechanism disabled, Enabling state ma chine at 4 (time=191678416, caller=fwioctl: FWHA_CUL_POLICY_STATE_FREEZE)
    FW-1: Initializing debugging buffer to size 1023K
    [fw4_0];FW-1: monitor filter unloaded
    [fw4_1];FW-1: monitor filter unloaded
    [fw4_2];FW-1: monitor filter unloaded
    [fw4_0];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_1];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_2];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=191782620)
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=191926770)
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=192071082)

    Nothing significant here either.

    Could the CPU message be linked?

  11. #31
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Okay - had a breakthrough (I think)

    Here me out.

    So, I feel the issue is linked back to the F_INDOM message - domain objects.

    As previously mentioned, we barely use them at all.

    Last night, when we forced the issue, usual DMZ traffic - ie to web servers gets dropped due to F_INDOM.

    Now - during normality, it is allowed through on rule 90 - absolutely fine. Always has been. It is a bog standard rule with network objects containing 3 subnets. No link or relevance to domain objects.

    Secondly,
    Another F_INDOM - Our network to a external server call-home. Dropping constantly last night on F_INDOM but allowed usually on rule 55 - perfectly fine no issues. It goes through on that rule all the time every second.

    Why - during these random occurances and also after a policy push is the firewall treating traffic as domain objects, and ultimately failing? What is causing this?

    I repeat we do not really use domain objects, and all of these instances are non domain objects and have no relation in the rule.

    This F_INDOM had over 5000 logs reported to various IP address (which are normally all allowed and not part of a domain object) over the whole issue.

    Any ideas?

  12. #32
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,088
    Rep Power
    12

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    Okay - had a breakthrough (I think)

    Here me out.

    So, I feel the issue is linked back to the F_INDOM message - domain objects.

    As previously mentioned, we barely use them at all.

    Last night, when we forced the issue, usual DMZ traffic - ie to web servers gets dropped due to F_INDOM.

    Now - during normality, it is allowed through on rule 90 - absolutely fine. Always has been. It is a bog standard rule with network objects containing 3 subnets. No link or relevance to domain objects.

    Secondly,
    Another F_INDOM - Our network to a external server call-home. Dropping constantly last night on F_INDOM but allowed usually on rule 55 - perfectly fine no issues. It goes through on that rule all the time every second.

    Why - during these random occurances and also after a policy push is the firewall treating traffic as domain objects, and ultimately failing? What is causing this?

    I repeat we do not really use domain objects, and all of these instances are non domain objects and have no relation in the rule.

    This F_INDOM had over 5000 logs reported to various IP address (which are normally all allowed and not part of a domain object) over the whole issue.

    Any ideas?
    Domain objects are terrible and can easily cause problems like this, try to move rules using them as far down as possible in the policy to avoid problems. The handling of domain objects was significantly overhauled in R80.10 gateway which is a bit of a hint that there were problems with domain objects in prior releases. Even with the overhaul of domain objects in R80.10 I still plan to avoid them like the plague, way way too many bad experiences.
    --
    My Book "Max Power: Check Point Firewall Performance Optimization"
    Second Edition Coming Soon

  13. #33
    Join Date
    2006-09-26
    Posts
    3,055
    Rep Power
    15

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by ShadowPeak.com View Post
    Domain objects are terrible and can easily cause problems like this, try to move rules using them as far down as possible in the policy to avoid problems. The handling of domain objects was significantly overhauled in R80.10 gateway which is a bit of a hint that there were problems with domain objects in prior releases. Even with the overhaul of domain objects in R80.10 I still plan to avoid them like the plague, way way too many bad experiences.
    If you're going to use domain-objects, which everyone is doing these days because of clouds like AWS and Azure Office 365, go with Palo Alto firewalls.

    I still can't believe after 15+ years, checkpoint still does not have its act together with domain-objects

  14. #34
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Hi all,

    Just bumping this thread - any other suggestions?

  15. #35
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,480
    Rep Power
    8

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Can you disable the domain object for for a period of time? It might help zero in on root cause.

  16. #36
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Hi all

    On our slave firewall in the cluster we have this from dmesg

    [fw4_1];Stopping ClusterXL
    [fw4_1];Starting ClusterXL
    [fw4_1];FW-1: fwha_set_new_local_state: Setting state of fwha_local_id(1) to ACT IVE
    parpdrv ioctl: cmd 2010
    bond10.102: dev_set_promiscuity(master, -1)
    device bond10.102 left promiscuous mode
    parpdrv ioctl: cmd 2010
    bond10.103: dev_set_promiscuity(master, -1)
    device bond10.103 left promiscuous mode
    parpdrv ioctl: cmd 2010
    bond10.101: dev_set_promiscuity(master, -1)
    device eth2 left promiscuous mode
    device eth1 left promiscuous mode
    device bond10 left promiscuous mode
    device bond10.101 left promiscuous mode
    parpdrv ioctl: cmd 2010
    bond20.750: dev_set_promiscuity(master, -1)
    device eth5 left promiscuous mode
    device eth6 left promiscuous mode
    device bond20 left promiscuous mode
    device bond20.750 left promiscuous mode
    Passive ARP hook successfully uninstalled!
    [fw4_0];Global param: set int fw_is_running_on_cbs to '1'
    [fw4_1];Global param: set int fw_is_running_on_cbs to '1'
    [fw4_2];Global param: set int fw_is_running_on_cbs to '1'
    [fw4_0];Global param: set int fwha_cbs_which_member_is_running_gated to '0'
    [fw4_1];Global param: set int fwha_cbs_which_member_is_running_gated to '0'
    [fw4_1];FW-1: fwha_validate_member_running_gated: 0 is not a valid member id
    [fw4_2];Global param: set int fwha_cbs_which_member_is_running_gated to '0'
    [fw4_0];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_1];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_2];fw_kmalloc_impl: alloc_ranges: allocates 0 bytes
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=22174172)
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=22318402)
    [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.
    [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_p olicy_done_time=22462933)

    What does Passive Arp hook successfully installed mean? Could this be related to our ARP issue you all suggested?

  17. #37
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,088
    Rep Power
    12

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    Hi all,

    Just bumping this thread - any other suggestions?
    You've received plenty of them along with a fair amount of speculation. There have been requests to run commands and post their output here and to check certain things, and you've mostly ignored those, posted other things you've found, and requested even more speculation. That is why the thread has ground to a halt.

    If you'd like to get this thread going again, please go back through the entire thread and provide the items that were requested very early on.
    --
    My Book "Max Power: Check Point Firewall Performance Optimization"
    Second Edition Coming Soon

  18. #38
    Join Date
    2017-11-01
    Posts
    20
    Rep Power
    0

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by ShadowPeak.com View Post
    You've received plenty of them along with a fair amount of speculation. There have been requests to run commands and post their output here and to check certain things, and you've mostly ignored those, posted other things you've found, and requested even more speculation. That is why the thread has ground to a halt.

    If you'd like to get this thread going again, please go back through the entire thread and provide the items that were requested very early on.
    Iím not ignoring at all - Iíve just found new stuff that either contradicted or made what I said previously irrelevant hence why I havenít honoured everything you have said. Iíve very appreciative of all your help so please donít be patronising

  19. #39
    Join Date
    2006-09-26
    Posts
    3,055
    Rep Power
    15

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    Iím not ignoring at all - Iíve just found new stuff that either contradicted or made what I said previously irrelevant hence why I havenít honoured everything you have said. Iíve very appreciative of all your help so please donít be patronising
    you didn't do anything wrong. If Checkpoint TAC has not figured out your issue, it is very unlikely that anyone on this forum can figure it out either. Sad but true.

    since Checpoint has not figured your issue, have they talked to you about purchasing Diamond support yet?

  20. #40
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,480
    Rep Power
    8

    Default Re: Freezes/Lock-Out on our firewall that have CP puzzled.

    Quote Originally Posted by JPYDX View Post
    I’m not ignoring at all - I’ve just found new stuff that either contradicted or made what I said previously irrelevant hence why I haven’t honoured everything you have said. I’ve very appreciative of all your help so please don’t be patronising
    There has been a lot of discussion (Well maybe not a lot) about changes that would be helpful for this forum and this thread pretty much encompasses everything I've brought up.

    Just a quick overview,

    a way for someone to request a set of trouble shooting commands to be run, possibly from a template
    file request / attachments that follows all the posts.
    a way to mark requested output at completed
    up / down voting

    That being said what shadow said is correct. To move the issue forward you need to get the command output and I think try disabling domain objects. Everyone has said they can cause problems.

    I think your issue is right not there are still too many possibilities to come up with a root cause.

    Domain objects - possible - would start here at this point
    Something on a given firewall node? - possible just fail the firewall over and see if it keeps happening.
    SecureXL issue - possible - disable via cpconfig and then try policy pushes again - can cause increased CPU FYI.
    Generic Routing / Linux issue - possible

    All this is hard to say without input. I feel like you need a better shopping list for the next time to make an attempt at root cause.
    Last edited by jflemingeds; 3 Weeks Ago at 13:45.

Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. Databse Lock issue using custom script
    By biskit in forum R77.30
    Replies: 9
    Last Post: 2016-12-09, 01:23
  2. 4807 Appliance Webui - Cannot Aquire Lock?
    By cjmiller2 in forum R75.40 (GAiA)
    Replies: 0
    Last Post: 2013-09-19, 15:31
  3. known hard lock on flash based cards.
    By jflemingeds in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 1
    Last Post: 2012-07-28, 17:16
  4. SD Freezes
    By ymhhou in forum IPS Blade (Formerly SmartDefense)
    Replies: 3
    Last Post: 2008-03-09, 15:56
  5. NGX Primary HA active node hard lock
    By l0wkey in forum Management High Availability
    Replies: 5
    Last Post: 2006-11-08, 15:56

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •