CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


I'd like to thank everyone involved for making "The CPUG Challenge" a great success.
We helped a lot of people see and learn a bit more about R80.10, while having some fun.
We will be using this success to try and bring more events to more locations soon. -E

 

Results 1 to 15 of 15

Thread: CoreXL for 4 core system

  1. #1
    Join Date
    2007-07-18
    Posts
    11
    Rep Power
    0

    Default CoreXL for 4 core system

    We have several 4800 clusters running VSX, with 2 virtual systems (1 for firewall and 1 for VPN). Since it is a 4 core box, Id like to ask what the optimum setup is.

    They are running R77.30 and our SE recommended the following:
    CoreXL is disabled on VS0
    Assign 2 cores to Firewall VS and 1 core to VPN VS, leaving 1 core for interfaces
    2 virtual system instances assigned to Firewall VS
    1 virtual system instances assigned to VPN VS

    According to top and cpview, Core 0 is getting hammered with high (si). This happens intermittently and when it does there is high latency. From what I have read, CoreXL instances and SND should not share a core. Separating the internal and external interfaces on separate cores makes sense, but how to allocate enough for FWK? What is max virtual instances we should define per VS?

    Thanks

  2. #2
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    865
    Rep Power
    12

    Default Re: CoreXL for 4 core system

    You have 4 cores in your disposal, so it makes sense to use 1 SND and 3 FWK instances. The recommendation from your SE is reasonable. However, it is unclear what Core0 is doing in your case. Is it SND or not? Do you have any fragmentation issues in the network?
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  3. #3
    Join Date
    2006-09-26
    Posts
    2,917
    Rep Power
    13

    Default Re: CoreXL for 4 core system

    Quote Originally Posted by varera View Post
    You have 4 cores in your disposal, so it makes sense to use 1 SND and 3 FWK instances. The recommendation from your SE is reasonable. However, it is unclear what Core0 is doing in your case. Is it SND or not? Do you have any fragmentation issues in the network?
    Having deal with high CPU issue on the firewall on both the Power-1 appliances and open servers, both 805 licenses (8 cores and 5 blades but I am only using FW blade), I can offer my opinion below:

    1- fragmentation might be an issue but unlikely because your application must be intentionally doing it so it is unlikely the cause,

    2- the traffics traversing the firewall is under some kind of checkpoint enforcement like SQL'net or Microsoft DFS. those are CPU killers.

    3- if you're going to disable CoreXL, I think it will make the situation worse.

    my 2c.

  4. #4
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    865
    Rep Power
    12

    Default Re: CoreXL for 4 core system

    Quote Originally Posted by cciesec2006 View Post
    Having deal with high CPU issue on the firewall on both the Power-1 appliances and open servers, both 805 licenses (8 cores and 5 blades but I am only using FW blade), I can offer my opinion below:

    1- fragmentation might be an issue but unlikely because your application must be intentionally doing it so it is unlikely the cause,

    2- the traffics traversing the firewall is under some kind of checkpoint enforcement like SQL'net or Microsoft DFS. those are CPU killers.

    3- if you're going to disable CoreXL, I think it will make the situation worse.

    my 2c.
    That may only be partially correct.

    Software Interrupts are FW kernel operations or SecureXL processing. Most of FW tasks today are outsourced to FWK instances, but de-fragmentation is not. Deep inspection is done by FWK and would look like user mode CPU utilization.

    Yet, without sim affinity info we are just guessing here. Let's wait till TS posts sim affinity details
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  5. #5
    Join Date
    2007-07-18
    Posts
    11
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    Not planning to disable CoreXL, wondering if a 2/2 split will be fine (2 SND, 2 FWK)

    [Expert@fw-:0]# fw ctl affinity -l
    eth1: CPU 0
    eth7: CPU 0
    Mgmt: CPU 0
    VS_0 fwk: CPU 1 2 3
    VS_1 fwk: CPU 1 2 3
    VS_2 fwk: CPU 1 2 3
    VS_3 fwk: CPU 1 2 3

    [Expert@fw:0]# sim affinity -l
    Mgmt : 0
    eth1 : 0
    eth2 : 0
    eth3 : 0
    eth4 : 0
    eth5 : 0
    eth6 : 0
    eth7 : 0

    top - 17:24:28 up 124 days, 21:42, 3 users, load average: 1.55, 1.31, 1.18
    Tasks: 183 total, 2 running, 181 sleeping, 0 stopped, 0 zombie
    Cpu0 : 32.7%us, 4.3%sy, 0.0%ni, 14.7%id, 0.0%wa, 5.7%hi, 42.7%si, 0.0%st
    Cpu1 : 24.3%us, 9.3%sy, 0.0%ni, 65.1%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st
    Cpu2 : 17.2%us, 3.6%sy, 1.9%ni, 75.1%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st
    Cpu3 : 26.6%us, 6.2%sy, 0.0%ni, 62.7%id, 0.0%wa, 0.0%hi, 4.5%si, 0.0%st
    Mem: 8003244k total, 7556768k used, 446476k free, 643060k buffers
    Swap: 18900464k total, 600k used, 18899864k free, 2485788k cached

  6. #6
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    865
    Rep Power
    12

    Default Re: CoreXL for 4 core system

    Okay, it seems there is an issue with SXL core. The issue is, we do not know what is that, yet. Try getting SXL statistics, look for any acceleration errors, traffic spikes, etc
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  7. #7
    Join Date
    2006-09-26
    Posts
    2,917
    Rep Power
    13

    Default Re: CoreXL for 4 core system

    Quote Originally Posted by varera View Post
    Okay, it seems there is an issue with SXL core. The issue is, we do not know what is that, yet. Try getting SXL statistics, look for any acceleration errors, traffic spikes, etc
    In my situation, see below:

    top - 13:38:58 up 16 days, 16:14, 5 users, load average: 3.02, 1.11, 0.50
    Tasks: 168 total, 6 running, 162 sleeping, 0 stopped, 0 zombie
    Cpu0 : 12.0%us, 2.0%sy, 0.0%ni, 54.0%id, 0.0%wa, 0.0%hi, 32.0%si, 0.0%st
    Cpu1 : 1.9%us, 1.9%sy, 0.0%ni, 86.5%id, 0.0%wa, 0.0%hi, 9.6%si, 0.0%st
    Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 32.0%id, 0.0%wa, 0.0%hi, 68.0%si, 0.0%st
    Cpu3 : 0.0%us, 2.0%sy, 0.0%ni, 34.0%id, 0.0%wa, 0.0%hi, 64.0%si, 0.0%st
    Cpu4 : 0.0%us, 0.0%sy, 0.0%ni, 34.7%id, 0.0%wa, 0.0%hi, 65.3%si, 0.0%st
    Cpu5 : 0.0%us, 2.0%sy, 0.0%ni, 62.0%id, 0.0%wa, 0.0%hi, 36.0%si, 0.0%st
    Cpu6 : 0.0%us, 2.0%sy, 0.0%ni, 60.8%id, 0.0%wa, 0.0%hi, 37.3%si, 0.0%st
    Cpu7 : 0.0%us, 0.0%sy, 0.0%ni, 9.6%id, 0.0%wa, 0.0%hi, 89.4%si, 0.0%st
    Mem: 5960440k total, 3927864k used, 2032576k free, 289404k buffers
    Swap: 14707496k total, 0k used, 14707496k free, 1710600k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    5297 admin 15 0 0 0 0 R 89 0.0 7857:58 fw_worker_0
    5302 admin 15 0 0 0 0 R 67 0.0 8181:43 fw_worker_5
    5300 admin 15 0 0 0 0 S 65 0.0 1565:45 fw_worker_3
    5301 admin 15 0 0 0 0 R 65 0.0 583:03.26 fw_worker_4
    5298 admin 15 0 0 0 0 R 38 0.0 387:51.57 fw_worker_1
    5299 admin 15 0 0 0 0 R 38 0.0 1709:09 fw_worker_2

    Firewall is passing about 800Mbits/sec of SQL'net traffics and I can see latencies with other traffics. "fwaccel stats -s" shows that 99% of the traffics go through F2F which causes the CPU spikes, no packet fragmentation and I also have DD enable on the gateways, and that my SQL'net is at the very top of the rule base:

    Every 2.0s: fwaccel stats -s Mon Apr 17 13:40:59 2017

    Accelerated conns/Total conns : 11/31 (35%)
    Accelerated pkts/Total pkts : 4104/32792559 (0%)
    F2Fed pkts/Total pkts : 32788455/32792559 (99%)
    PXL pkts/Total pkts : 0/32792559 (0%)
    QXL pkts/Total pkts : 0/32792559 (0%)

    fw ctl affinity -l -r -v -a
    CPU 0: Sync (irq 139) Exp2-2 (irq 91) Lan1 (irq 235) Exp1-1 (irq 107)
    CPU 1: Exp2-1 (irq 75) Lan2 (irq 211) Exp1-2 (irq 123)
    CPU 2: fw_5
    CPU 3: fw_4
    CPU 4: fw_3
    CPU 5: fw_2
    CPU 6: fw_1
    CPU 7: fw_0
    All: mpdaemon rtmd fwd cprid cpd

  8. #8
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    865
    Rep Power
    12

    Default Re: CoreXL for 4 core system

    Quote Originally Posted by cciesec2006 View Post
    In my situation, see below:
    It seems to me you are trying to highjack this thread. Just open a new one, would you?
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  9. #9
    Join Date
    2007-07-18
    Posts
    11
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    Varera, here you go:

    [Expert@fw:1]# fwaccel stats -s
    Accelerated conns/Total conns : 332/11550 (2%)
    Delayed conns/(Accelerated conns + PXL conns) : 169/6029 (2%)
    Accelerated pkts/Total pkts : 2921044/202601942 (1%)
    F2Fed pkts/Total pkts : 19855139/202601942 (9%)
    PXL pkts/Total pkts : 179825759/202601942 (88%)
    QXL pkts/Total pkts : 0/202601942 (0%)

    [Expert@fw:1]# fwaccel stats
    Name Value Name Value
    -------------------- --------------- -------------------- ---------------

    Accelerated Path
    ------------------------------------------------------------------------------
    accel packets 2919885 accel bytes 1674697195
    conns created 1282897 conns deleted 826982
    C total conns 11691 C templates 376
    C TCP conns 10134 C delayed TCP conns 175
    C non TCP conns 1557 C delayed nonTCP con 0
    conns from templates 292198 temporary conns 39539
    nat conns 732999 dropped packets 38147
    dropped bytes 51419194 nat templates 0
    port alloc templates 0 conns from nat tmpl 0
    port alloc conns 0 conns auto expired 681966

    Accelerated VPN Path
    ------------------------------------------------------------------------------
    C crypt conns 0 enc bytes 0
    dec bytes 0 ESP enc pkts 0
    ESP enc err 0 ESP dec pkts 0
    ESP dec err 0 ESP other err 0
    AH enc pkts 0 AH enc err 0
    AH dec pkts 0 AH dec err 0
    AH other err 0 espudp enc pkts 0
    espudp enc err 0 espudp dec pkts 0
    espudp dec err 0 espudp other err 0

    Medium Path
    ------------------------------------------------------------------------------
    PXL packets 179560871 PXL async packets 179844554
    PXL bytes 122105304015 C PXL conns 5804
    C PXL templates 331 PXL FF conns 0
    PXL FF packets 0 PXL FF bytes 0
    PXL FF acks 0

    Accelerated QoS Path
    ------------------------------------------------------------------------------
    QXL packets 0 QXL async packets 0
    QXL bytes 0 C QXL conns 0
    C QXL templates 0

    Firewall Path
    ------------------------------------------------------------------------------
    F2F packets 19837892 F2F bytes 11548151109
    C F2F conns 5546 TCP violations 101711
    C partial conns 0 C anticipated conns 0
    port alloc f2f 0

    GTP
    ------------------------------------------------------------------------------
    gtp tunnels created 0 gtp tunnels 0
    gtp accel pkts 0 gtp f2f pkts 0
    gtp spoofed pkts 0 gtp in gtp pkts 0
    gtp signaling pkts 0 gtp tcpopt pkts 0
    gtp apn err pkts 0

    General
    ------------------------------------------------------------------------------
    memory used 0 free memory 0
    C used templates 235 pxl tmpl conns 240070
    C conns from tmpl 1751 C non TCP F2F conns 1170
    C tcp handshake conn 178 C tcp established co 9356
    C tcp closed conns 600 C tcp f2f handshake 112
    C tcp f2f establishe 4220 C tcp f2f closed con 44
    C tcp pxl handshake 62 C tcp pxl establishe 4893
    C tcp pxl closed con 463 outbound packets 2919885
    outbound pxl packets 179554673 outbound f2f packets 18331611
    outbound bytes 1721164081 outbound pxl bytes 124872806273
    outbound f2f bytes 10315699211

  10. #10
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    865
    Rep Power
    12

    Default Re: CoreXL for 4 core system

    one time look does not help. You need to check which stats are changing when you are experiencing higher latency.
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  11. #11
    Join Date
    2016-10-31
    Posts
    46
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    VPN always use CPU0 so assign different CPU to SND.

  12. #12
    Join Date
    2016-10-31
    Posts
    46
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    VWhen CoreXL is enabled, VPN traffic inspection occurs only in global CoreXL FW instance #0 (fw_worker_0). By design, global CoreXL FW instance #0 (fw_worker_0) always runs on the CPU core with highest ID (as allowed by the current CoreXL license) so so assign different CPU to SND.

  13. #13
    Join Date
    2016-10-31
    Posts
    46
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    Can you also share fwaccel stat output which will give idea where securexl is breaking.

  14. #14
    Join Date
    2006-09-26
    Posts
    2,917
    Rep Power
    13

    Default Re: CoreXL for 4 core system

    Quote Originally Posted by ba3113 View Post
    Can you also share fwaccel stat output which will give idea where securexl is breaking.
    Either "watch -d -n 1 fwaccel stats -p" or "watch -d -n 1 fwaccel stats -s"

    Every 1.0s: fwaccel stats -s Thu Apr 27 09:50:23 2017

    Accelerated conns/Total conns : 11/22 (50%)
    Accelerated pkts/Total pkts : 100135/34355268 (0%) --- secureXL is accelerating traffics GOOD !!!!! (but none for me)
    F2Fed pkts/Total pkts : 34246331/34355268 (99%) ---- it is going through the F2F means that SecureXL can not accelerate your traffics BAD !!!!!
    PXL pkts/Total pkts : 8802/34355268 (0%)
    QXL pkts/Total pkts : 0/34355268 (0%)


    Every 2.0s: fwaccel stats -p Thu Apr 27 09:51:52 2017

    F2F packets:
    --------------
    Violation Packets Violation Packets
    -------------------- --------------- -------------------- ---------------
    pkt is a fragment 2615975 pkt has IP options 0
    ICMP miss conn 456178 TCP-SYN miss conn 1248
    TCP-other miss conn 622 UDP miss conn 27379154 this is BAD !!!!!
    other miss conn 20130 VPN returned F2F 0
    ICMP conn is F2Fed 0 TCP conn is F2Fed 9940779 ---- this is BAD !!!!
    UDP conn is F2Fed 863 other conn is F2Fed 0
    uni-directional viol 0 possible spoof viol 1
    TCP state viol 127 out if not def/accl 0
    bridge, src=dst 0 routing decision err 0
    sanity checks failed 0 temp conn expired 0
    fwd to non-pivot 0 broadcast/multicast 0
    cluster message 0 partial conn 0
    PXL returned F2F 0 cluster forward 0
    chain forwarding 0 general reason 0

  15. #15
    Join Date
    2016-10-31
    Posts
    46
    Rep Power
    0

    Default Re: CoreXL for 4 core system

    Can you please share fwaccel stat not the fwaccel stats.

Similar Threads

  1. Replies: 5
    Last Post: 2016-04-21, 13:04
  2. Replies: 3
    Last Post: 2014-11-03, 18:25
  3. How do I check system log files on ipso, to check system health etc
    By carl_t in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 4
    Last Post: 2013-10-25, 09:35
  4. Urgent: Help needed with migrating NGx R65 CMA from Solaris 9 system to SPLAT system
    By cciesec2006 in forum Provider-1 (Multi-Domain Management)
    Replies: 7
    Last Post: 2007-12-26, 09:58
  5. Core XL
    By Porter in forum Miscellaneous
    Replies: 4
    Last Post: 2007-10-31, 21:25

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •