CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 17 of 17

Thread: High CPU problem on checkpoint gateway

Hybrid View

  1. #1
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default High CPU problem on checkpoint gateway

    A pair of R77.30 with HFA_216 clusterXL in Active/Standby on Dell PowerEdge R710 with 405 license. Only fw blade is enabled as confirmed with "enabled_blades" output.

    I have a 10G interfaces on bond0 and 10G interface on eth1. My encrypted Oracle traffics traverse the firewall on the first rule in the security policy. No NAT involved, just routing. SecureXL is ON. Dynamics dispatching is also ON. This Oracle traffic rule is at the top of security policy. The Oracle traffic enters interface bond0 and out of interface eth1

    To ensure that the Oracle traffic is not fragmented, I set the SDU paramater in Oracle to 1400 to make sure it fits the MTU on the interface which is 1500.

    As soon as the traffics get to 2Gbps, I see one of the CPU is maxed out at 100% utilization. In my case, it was cpu 0. When I run top, I see cpu 0 is pegged with 100% si, like this:

    top - 13:08:58 up 501 days, 12:14, 1 user, load average: 0.00, 0.02, 0.00
    Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie
    Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 03.0%id, 0.0%wa, 0.0%hi, 95.0%si, 0.0%st


    When I check "sim affinity -l", I see this:

    [Expert@gw1:0]# sim affinity -l
    eth0 : 0
    eth1 : 0
    eth11 : 0
    eth13 : 0
    eth3 : 0
    eth4 : 0
    [Expert@gw1:0]#

    When I check with 'fwaccel conns | grep "x.x.x.x" | grep F', all of the Oracle traffic between the source and destination are in accelerated mode.

    How do I go about fixing the high CPU issue? Is this a "known" issue with Checkpoint firewalls? Does it mean I have to manually do the "sim affinity"? What is the downside of manually assigning individual CPUs to interfaces?

    I notice that the same problem with R80.20 management that manages R80.20 gateways as well.

    How do you about fixing this issue?

  2. #2
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default Re: High CPU problem on checkpoint gateway

    still looking for suggestions on this.

  3. #3
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: High CPU problem on checkpoint gateway

    Quote Originally Posted by cciesec2006 View Post
    A pair of R77.30 with HFA_216 clusterXL in Active/Standby on Dell PowerEdge R710 with 405 license. Only fw blade is enabled as confirmed with "enabled_blades" output.

    I have a 10G interfaces on bond0 and 10G interface on eth1. My encrypted Oracle traffics traverse the firewall on the first rule in the security policy. No NAT involved, just routing. SecureXL is ON. Dynamics dispatching is also ON. This Oracle traffic rule is at the top of security policy. The Oracle traffic enters interface bond0 and out of interface eth1

    To ensure that the Oracle traffic is not fragmented, I set the SDU paramater in Oracle to 1400 to make sure it fits the MTU on the interface which is 1500.

    As soon as the traffics get to 2Gbps, I see one of the CPU is maxed out at 100% utilization. In my case, it was cpu 0. When I run top, I see cpu 0 is pegged with 100% si, like this:

    top - 13:08:58 up 501 days, 12:14, 1 user, load average: 0.00, 0.02, 0.00
    Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie
    Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 03.0%id, 0.0%wa, 0.0%hi, 95.0%si, 0.0%st


    When I check "sim affinity -l", I see this:

    [Expert@gw1:0]# sim affinity -l
    eth0 : 0
    eth1 : 0
    eth11 : 0
    eth13 : 0
    eth3 : 0
    eth4 : 0
    [Expert@gw1:0]#

    When I check with 'fwaccel conns | grep "x.x.x.x" | grep F', all of the Oracle traffic between the source and destination are in accelerated mode.

    How do I go about fixing the high CPU issue? Is this a "known" issue with Checkpoint firewalls? Does it mean I have to manually do the "sim affinity"? What is the downside of manually assigning individual CPUs to interfaces?

    I notice that the same problem with R80.20 management that manages R80.20 gateways as well.

    How do you about fixing this issue?
    Because you are licensed for only 4 cores, you probably have the default 1/3 split of SND/IRQ cores to Firewall Worker cores. Please provide output of fw ctl affinity -l -r and fwaccel stats -s to confirm.

    Since it sounds like almost all traffic is accelerated, all of it is only being handled by the one SND/IRQ core. The Dynamic Dispatcher will not help in this case since its job is to balance load among Firewall Worker cores, not SND/IRQ cores. Would suggest reducing the number of Firewall Workers (kernel instances) from 3 to 2 via cpconfig so that there will be a split of 2 SND/IRQ cores vs. 2 Firewall Worker cores; this will double the SND/IRQ processing resources available to handle the accelerated Oracle traffic.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  4. #4
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default Re: High CPU problem on checkpoint gateway

    Quote Originally Posted by ShadowPeak.com View Post
    Because you are licensed for only 4 cores, you probably have the default 1/3 split of SND/IRQ cores to Firewall Worker cores. Please provide output of fw ctl affinity -l -r and fwaccel stats -s to confirm.

    Since it sounds like almost all traffic is accelerated, all of it is only being handled by the one SND/IRQ core. The Dynamic Dispatcher will not help in this case since its job is to balance load among Firewall Worker cores, not SND/IRQ cores. Would suggest reducing the number of Firewall Workers (kernel instances) from 3 to 2 via cpconfig so that there will be a split of 2 SND/IRQ cores vs. 2 Firewall Worker cores; this will double the SND/IRQ processing resources available to handle the accelerated Oracle traffic.

    [Expert@gw-1:0]# fw ctl affinity -l -r
    CPU 0: eth3 eth4 eth11 eth13 eth0 eth1
    CPU 1: fw_2
    CPU 2: fw_1
    CPU 3: fw_0
    CPU 4:
    CPU 5:
    CPU 6:
    CPU 7:
    All: rtmd fwd in.ahclientd mpdaemon in.aclientd lpd cprid cpd
    The current license permits the use of CPUs 0, 1, 2, 3 only.
    [Expert@gw-1:0]# fwaccel stats -s
    Accelerated conns/Total conns : 5199/5228 (99%)
    Accelerated pkts/Total pkts : 10750376389/10862974478 (98%)
    F2Fed pkts/Total pkts : 78461822/10862974478 (0%)
    PXL pkts/Total pkts : 34136267/10862974478 (0%)
    QXL pkts/Total pkts : 0/10862974478 (0%)
    [Expert@gw-1:0]#

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: High CPU problem on checkpoint gateway

    Quote Originally Posted by cciesec2006 View Post
    [Expert@gw-1:0]# fw ctl affinity -l -r
    CPU 0: eth3 eth4 eth11 eth13 eth0 eth1
    CPU 1: fw_2
    CPU 2: fw_1
    CPU 3: fw_0
    CPU 4:
    CPU 5:
    CPU 6:
    CPU 7:
    All: rtmd fwd in.ahclientd mpdaemon in.aclientd lpd cprid cpd
    The current license permits the use of CPUs 0, 1, 2, 3 only.
    [Expert@gw-1:0]# fwaccel stats -s
    Accelerated conns/Total conns : 5199/5228 (99%)
    Accelerated pkts/Total pkts : 10750376389/10862974478 (98%)
    F2Fed pkts/Total pkts : 78461822/10862974478 (0%)
    PXL pkts/Total pkts : 34136267/10862974478 (0%)
    QXL pkts/Total pkts : 0/10862974478 (0%)
    [Expert@gw-1:0]#
    Yup definitely decrease kernel instances from 3 to 2 with cpconfig. Will help a lot.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  6. #6
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default Re: High CPU problem on checkpoint gateway

    Quote Originally Posted by ShadowPeak.com View Post
    Yup definitely decrease kernel instances from 3 to 2 with cpconfig. Will help a lot.
    Actually I just did and it made the problem worse. Now everything is 50% slower :-(. Had to revert back my change.

  7. #7
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: High CPU problem on checkpoint gateway

    Quote Originally Posted by cciesec2006 View Post
    Actually I just did and it made the problem worse. Now everything is 50% slower :-(. Had to revert back my change.
    Huh? That makes no sense, please define what "50% slower" means. If you have a cluster changing the number of kernel instances needs to be handled the same way as code upgrade.

    You may have something else going on, please post the output of these commands:

    fwaccel stat
    grep -c ^processor /proc/cpuinfo
    /sbin/cpuinfo
    fw ctl affinity -l -r
    sim affinity -l
    netstat -ni
    fw ctl multik stat
    free -m
    enabled_blades
    fw ver
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

Similar Threads

  1. IPSEC VPN tunnel problem between checkpoint and Juniper Gateway
    By tofke in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 6
    Last Post: 2014-04-17, 03:51
  2. node gateway vs checkpoint gateway
    By jgarzam in forum SmartDashboard
    Replies: 1
    Last Post: 2013-05-13, 08:51
  3. Is Checkpoint TAC high on crack?
    By cciesec2006 in forum Installing And Upgrading
    Replies: 7
    Last Post: 2010-03-08, 12:06
  4. problem with a SC high availability
    By jvalenzuela in forum Licensing
    Replies: 8
    Last Post: 2009-04-29, 19:25
  5. High CPU, Strange alerts problem
    By ggts2008 in forum Check Point UTM-1 Appliances
    Replies: 1
    Last Post: 2008-07-21, 16:08

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •