CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


I'd like to thank everyone involved for making "The CPUG Challenge" a great success.
We helped a lot of people see and learn a bit more about R80.10, while having some fun.
We will be using this success to try and bring more events to more locations soon. -E

 

Results 1 to 12 of 12

Thread: Cluster stopped passing traffic

  1. #1
    Join Date
    2012-10-03
    Posts
    65
    Rep Power
    6

    Default Cluster stopped passing traffic

    Good evening everyone. I had a 4800 cluster, R77.30 take 216 gaia stop passing traffic tonight. The cluster has 6 locally connected networks, and one interface leads to our private MPLS network. I wasn't on site at the time, here's what I know:

    No locally connected segments could see each other. No traffic in or out to the MPLS cloud.

    The cluster sends logs to two different log servers - one is on a locally connected network. From the outage period, there were no logs on either log server sourced from this cluster

    Nothing in the var/messages.* files look interesting to me

    So, what fixed it..? I had an onsite person power cycle both members, and everything was back to normal.

    Any ideas on where to look for the cause would be appreciated. This cluster is relatively trouble-free, this is the first issue I can recall.

    thanks

  2. #2
    Join Date
    2013-09-25
    Location
    Bucharest
    Posts
    611
    Rep Power
    5

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by DannyW View Post
    Good evening everyone. I had a 4800 cluster, R77.30 take 216 gaia stop passing traffic tonight. The cluster has 6 locally connected networks, and one interface leads to our private MPLS network. I wasn't on site at the time, here's what I know:

    No locally connected segments could see each other. No traffic in or out to the MPLS cloud.

    The cluster sends logs to two different log servers - one is on a locally connected network. From the outage period, there were no logs on either log server sourced from this cluster

    Nothing in the var/messages.* files look interesting to me

    So, what fixed it..? I had an onsite person power cycle both members, and everything was back to normal.

    Any ideas on where to look for the cause would be appreciated. This cluster is relatively trouble-free, this is the first issue I can recall.

    thanks

    Unfortunately, no logs --> no clue. What does monitoring right before the event shows? CPU spike, traffic spike, maybe no of sessions spike?

  3. #3
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,051
    Rep Power
    12

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by DannyW View Post
    Good evening everyone. I had a 4800 cluster, R77.30 take 216 gaia stop passing traffic tonight. The cluster has 6 locally connected networks, and one interface leads to our private MPLS network. I wasn't on site at the time, here's what I know:

    No locally connected segments could see each other. No traffic in or out to the MPLS cloud.

    The cluster sends logs to two different log servers - one is on a locally connected network. From the outage period, there were no logs on either log server sourced from this cluster

    Nothing in the var/messages.* files look interesting to me

    So, what fixed it..? I had an onsite person power cycle both members, and everything was back to normal.

    Any ideas on where to look for the cause would be appreciated. This cluster is relatively trouble-free, this is the first issue I can recall.

    thanks
    Fire up cpview in historical mode zoomed in on the time period immediately before and during the outage. Any strange resource issues like high CPU, low memory, or network errors?

    sar can be used to as well look back at resource usage during a troubled period on the firewall.

    Check $FWDIR/log/fwd.elg and any other *.elg files in that directory on the cluster members for suspicious entries around that time too.

    Are you sure there are no ClusterXL log entries of type "Control" (grey wrench) in SmartView Tracker/SmartLog around the time of the issue?
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  4. #4
    Join Date
    2006-09-26
    Posts
    3,022
    Rep Power
    15

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by ShadowPeak.com View Post
    Fire up cpview in historical mode zoomed in on the time period immediately before and during the outage.
    Can you share with us how you go about doing that?

    Thanx

  5. #5
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,402
    Rep Power
    8

    Default Re: Cluster stopped passing traffic

    cpview -t ddmmmyyyy hh:mm

    then use + / - to move forward / backwards.

  6. #6
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,051
    Rep Power
    12

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by jflemingeds View Post
    cpview -t ddmmmyyyy hh:mm

    then use + / - to move forward / backwards.
    Right so I'd suggest starting 30 minutes before the outage and then start stepping forward one minute at a time watching CPU and network statistics. Kind of hard to describe what to look for specifically, but hopefully you'll know it when you see it.
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  7. #7
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,402
    Rep Power
    8

    Default Re: Cluster stopped passing traffic

    I should really know this, but what is the process for the mgmt server downloading logs stored on the gateway? Is that something you have to do manually? What i'm getting at is if the gateway stopped logging to mgmt server chance are there are logs on the gateway itself that would contain control messages which could help identify what happened. Could also just view the logs with fw log directly on the gateway, just make sure to use the options to disable name and service resolution as these are terribly slow.

  8. #8
    Join Date
    2012-10-03
    Posts
    65
    Rep Power
    6

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by jflemingeds View Post
    I should really know this, but what is the process for the mgmt server downloading logs stored on the gateway? Is that something you have to do manually? What i'm getting at is if the gateway stopped logging to mgmt server chance are there are logs on the gateway itself that would contain control messages which could help identify what happened. Could also just view the logs with fw log directly on the gateway, just make sure to use the options to disable name and service resolution as these are terribly slow.
    I did "fetch" the logs that were locally stored on each gateway.....nothing for the 3 hour period. Going to do try the cpview recommendation - will update everyone if I find something eye catching.

    thx

  9. #9
    NorbertBohusch is offline Junior Member CPUG Challenge Finalist - 2017
    Join Date
    2017-05-18
    Posts
    7
    Rep Power
    0

    Default Re: Cluster stopped passing traffic

    Which blades/functions are active on the cluster? Some of our customer with HTTPS inspection had similar outages with take 216!
    TAC confirmed issue and fixed in 221 and up. But take 216 was not called back.

    Gesendet von meinem Moto G (4) mit Tapatalk

  10. #10
    Join Date
    2012-10-03
    Posts
    65
    Rep Power
    6

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by NorbertBohusch View Post
    Which blades/functions are active on the cluster? Some of our customer with HTTPS inspection had similar outages with take 216!
    TAC confirmed issue and fixed in 221 and up. But take 216 was not called back.

    Gesendet von meinem Moto G (4) mit Tapatalk

    fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot

    Thanks for the info. Seems like they only provided detailed info on fixes\enhancements for GA takes, do you know the verbiage of the take 221 fix - what was the "symptom" description?

    thanks

  11. #11
    NorbertBohusch is offline Junior Member CPUG Challenge Finalist - 2017
    Join Date
    2017-05-18
    Posts
    7
    Rep Power
    0

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by DannyW View Post
    fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot

    Thanks for the info. Seems like they only provided detailed info on fixes\enhancements for GA takes, do you know the verbiage of the take 221 fix - what was the "symptom" description?

    thanks
    Hi, as I don't have all infos at hand, I would just recommend you install take 225 asap!

    Gesendet von meinem Moto G (4) mit Tapatalk

  12. #12
    NorbertBohusch is offline Junior Member CPUG Challenge Finalist - 2017
    Join Date
    2017-05-18
    Posts
    7
    Rep Power
    0

    Default Re: Cluster stopped passing traffic

    Quote Originally Posted by NorbertBohusch View Post
    Hi, as I don't have all infos at hand, I would just recommend you install take 225 asap!

    Gesendet von meinem Moto G (4) mit Tapatalk
    Just checked Jumbo SK and its there under ID 02468493 at take 221 (section "resolved issues for ongoing take")

    Gesendet von meinem Moto G (4) mit Tapatalk

Similar Threads

  1. R75 cluster object corrupt. Cluster not passing traffic
    By jmcgrady in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 0
    Last Post: 2011-12-01, 23:53
  2. Checkpoint firewall stopped forwarding traffic
    By wilkie in forum Miscellaneous
    Replies: 1
    Last Post: 2009-07-01, 11:14
  3. Not passing traffic
    By awalt1279 in forum Topology Issues
    Replies: 8
    Last Post: 2009-06-15, 16:51
  4. Need clarification on SA's : Tunnel up but traffic not passing
    By dreambuddy in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 3
    Last Post: 2008-09-15, 14:54
  5. when adding 2nd node in clusterXL , cluster stops passing traffic
    By chgoIT in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 0
    Last Post: 2006-11-13, 10:41

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •