CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


 

Page 3 of 3 FirstFirst 123
Results 41 to 59 of 59

Thread: very slow intervaln communication via checkpoint

  1. #41
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Looks like you have 4 Firewall Worker cores who are having to compete with SND/IRQ processing for the same cores. Your RX-DRP rate is well below 0.1% on all interfaces which is good, although your eth4 and eth5 interfaces are attempting flow control with the switch which indicates they are getting close to overruns in the NICs although it doesn't look like any have actually occurred. Please post the following:

    cat /proc/interrupts
    sar -P ALL (wait until after your firewall's busiest period has passed earlier in the day before running this)
    fwaccel stats -s
    fwaccel stat

    When you say performance is unacceptable, during that period is it due to packet loss (doesn't look likely based on what you posted) or excessive packet latency? (more likely) Run a ping through the firewall when performance is degraded to see if it is latency or loss.

    One thing that is a bit unusual is the nonzero tx_restart_queue counters, this generally indicates that the TX ring buffer is full which is fairly rare. Most problems I've seen are on the RX side. My guess is that you have very high CPU utilization on your 4 cores and may need to consider unlocking more cores with a larger license; the sar output will show us that. SecureXL tuning may help get that high CPU utilization down.

    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.
    Last edited by ShadowPeak.com; 2015-04-15 at 13:42.

  2. #42
    Join Date
    2014-10-03
    Posts
    29
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    [Expert@gate02:0]# cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 2876448406 0 0 0 IO-APIC-edge timer
    1: 3 0 0 0 IO-APIC-edge i8042
    4: 502 0 0 0 IO-APIC-edge serial
    8: 3 0 0 0 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-level acpi
    12: 4 0 0 0 IO-APIC-edge i8042
    51: 11390230 451373469 144753102 45509666 PCI-MSI-X eth3
    59: 261403272 12366201 15013100 182359 PCI-MSI-X eth1
    122: 649 0 0 0 IO-APIC-level ehci_hcd:usb2
    130: 66 0 0 0 IO-APIC-level ehci_hcd:usb3
    146: 80851132 0 0 0 PCI-MSI-X cciss0
    185: 0 0 0 0 IO-APIC-level uhci_hcd:usb1
    202: 11834048 568954070 1295149294 2901189190 PCI-MSI-X eth4-TxRx-0
    210: 1 0 0 0 PCI-MSI-X eth4
    226: 3558238 344913786 1043530074 1527649647 PCI-MSI-X eth5-TxRx-0
    234: 1 0 0 0 PCI-MSI-X eth5
    NMI: 645291 724210 644017 694212
    LOC: 2876329589 2876329706 2876329670 2876326904
    ERR: 0
    MIS: 0

    SAR Command:


    07:40:01 CPU %user %nice %system %iowait %steal %idle
    07:50:01 all 1.97 0.00 16.85 0.12 0.00 81.07
    07:50:01 0 1.56 0.00 15.49 0.34 0.00 82.61
    07:50:01 1 1.95 0.00 11.07 0.08 0.00 86.90
    07:50:01 2 1.91 0.00 20.57 0.02 0.00 77.51
    07:50:01 3 2.45 0.00 20.27 0.04 0.00 77.24
    08:00:01 all 2.08 0.00 18.88 0.14 0.00 78.91
    08:00:01 0 2.14 0.00 14.02 0.36 0.00 83.47
    08:00:01 1 2.25 0.00 13.84 0.04 0.00 83.86
    08:00:01 2 2.02 0.00 22.38 0.03 0.00 75.56
    08:00:01 3 1.89 0.00 25.25 0.12 0.00 72.74
    08:10:01 all 2.33 0.00 22.65 0.02 0.00 75.00
    08:10:01 0 2.26 0.00 20.04 0.00 0.00 77.70
    08:10:01 1 1.92 0.00 20.13 0.03 0.00 77.92
    08:10:01 2 2.71 0.00 22.00 0.02 0.00 75.27
    08:10:01 3 2.41 0.00 28.45 0.03 0.00 69.11
    08:20:01 all 2.75 0.00 22.10 0.15 0.00 75.00
    08:20:01 0 2.82 0.00 16.95 0.25 0.00 79.98
    08:20:01 1 2.18 0.00 20.50 0.07 0.00 77.25
    08:20:01 2 3.31 0.00 23.87 0.08 0.00 72.74
    08:20:01 3 2.70 0.00 27.08 0.19 0.00 70.03
    08:30:01 all 2.07 0.00 24.81 0.08 0.00 73.04
    08:30:01 0 2.29 0.00 18.28 0.24 0.00 79.19
    08:30:01 1 1.95 0.00 21.31 0.05 0.00 76.69
    08:30:01 2 2.38 0.00 25.19 0.03 0.00 72.39
    08:30:01 3 1.65 0.00 34.44 0.02 0.00 63.89
    08:40:01 all 2.19 0.00 24.90 0.01 0.00 72.90
    08:40:01 0 2.21 0.00 19.38 0.00 0.00 78.41
    08:40:01 1 1.61 0.00 23.75 0.01 0.00 74.62
    08:40:01 2 1.88 0.00 32.95 0.00 0.00 65.17
    08:40:01 3 3.08 0.00 23.50 0.03 0.00 73.39
    08:50:05 all 4.94 0.09 31.07 0.05 0.00 63.86
    08:50:05 0 3.83 0.03 28.55 0.03 0.00 67.56
    08:50:05 1 6.03 0.06 23.87 0.13 0.00 69.92
    08:50:05 2 4.69 0.26 33.76 0.02 0.00 61.28
    08:50:05 3 5.22 0.01 38.09 0.01 0.00 56.67
    09:00:11 all 2.43 0.00 33.51 0.07 0.00 63.99
    09:00:11 0 2.69 0.00 26.66 0.16 0.00 70.49
    09:00:11 1 2.22 0.00 34.40 0.05 0.00 63.33
    09:00:11 2 2.88 0.00 29.36 0.06 0.00 67.69
    09:00:11 3 1.91 0.00 43.63 0.00 0.00 54.46
    09:10:01 all 2.29 0.00 36.34 0.03 0.00 61.35
    09:10:01 0 1.77 0.00 36.13 0.00 0.00 62.10
    09:10:01 1 2.93 0.00 31.14 0.05 0.00 65.89
    09:10:01 2 3.32 0.00 31.30 0.02 0.00 65.35
    09:10:01 3 1.15 0.00 46.78 0.03 0.00 52.03
    09:20:02 all 2.79 0.00 34.51 0.12 0.00 62.58
    09:20:02 0 2.30 0.00 30.45 0.21 0.00 67.04
    09:20:02 1 3.42 0.00 35.32 0.15 0.00 61.10
    09:20:02 2 3.95 0.00 30.09 0.07 0.00 65.89
    09:20:02 3 1.48 0.00 42.19 0.03 0.00 56.30
    09:30:01 all 2.24 0.00 38.65 0.08 0.00 59.03
    09:30:01 0 2.64 0.00 25.00 0.16 0.00 72.20
    09:30:01 1 1.88 0.00 41.83 0.10 0.00 56.18
    09:30:01 2 2.88 0.00 34.72 0.03 0.00 62.38
    09:30:01 3 1.54 0.00 53.04 0.04 0.00 45.38
    09:40:01 all 2.32 0.00 34.78 0.01 0.00 62.89
    09:40:01 0 2.11 0.00 27.64 0.00 0.00 70.24
    09:40:01 1 2.05 0.00 37.41 0.01 0.00 60.53
    09:40:01 2 3.08 0.00 33.66 0.04 0.00 63.22
    09:40:01 3 2.04 0.00 40.40 0.00 0.00 57.55
    09:50:01 all 2.35 0.00 32.43 0.06 0.00 65.15
    09:50:01 0 2.31 0.00 23.63 0.18 0.00 73.88
    09:50:01 1 1.76 0.00 40.66 0.00 0.00 57.58
    09:50:01 2 3.35 0.00 33.21 0.03 0.00 63.41
    09:50:01 3 1.99 0.00 32.24 0.03 0.00 65.74
    10:00:01 all 2.33 0.00 36.60 0.07 0.00 61.01
    10:00:01 0 2.37 0.00 27.65 0.16 0.00 69.82
    10:00:01 1 2.13 0.00 45.61 0.04 0.00 52.22
    10:00:01 2 3.18 0.00 36.31 0.02 0.00 60.49
    10:00:01 3 1.62 0.00 36.82 0.05 0.00 61.50
    10:10:01 all 2.49 0.00 34.98 0.01 0.00 62.51
    10:10:01 0 2.60 0.00 27.78 0.00 0.00 69.62
    10:10:01 1 1.84 0.00 39.50 0.02 0.00 58.64
    10:10:01 2 3.35 0.00 33.75 0.02 0.00 62.88
    10:10:01 3 2.17 0.00 38.90 0.01 0.00 58.92
    10:20:01 all 2.25 0.00 32.47 0.12 0.00 65.16
    10:20:01 0 1.91 0.00 30.45 0.27 0.00 67.37
    10:20:01 1 2.39 0.00 27.66 0.09 0.00 69.87
    10:20:01 2 2.86 0.00 31.11 0.11 0.00 65.92
    10:20:01 3 1.85 0.00 40.65 0.03 0.00 57.47
    10:30:01 all 2.16 0.00 34.33 0.40 0.00 63.11
    10:30:01 0 1.84 0.00 29.60 0.25 0.00 68.31
    10:30:01 1 1.91 0.00 38.71 0.41 0.00 58.97
    10:30:01 2 3.32 0.00 30.20 0.49 0.00 65.99
    10:30:01 3 1.55 0.00 38.82 0.46 0.00 59.17
    10:40:01 all 2.13 0.00 35.85 0.01 0.00 62.02
    10:40:01 0 2.11 0.00 28.81 0.00 0.00 69.08
    10:40:01 1 2.05 0.00 47.53 0.01 0.00 50.40
    10:40:01 2 3.17 0.00 28.41 0.01 0.00 68.41
    10:40:01 3 1.18 0.00 38.64 0.01 0.00 60.18
    10:50:01 all 2.19 0.00 37.03 0.07 0.00 60.72
    10:50:01 0 2.35 0.00 28.11 0.19 0.00 69.35
    10:50:01 1 2.58 0.00 30.74 0.04 0.00 66.64
    10:50:01 2 2.10 0.00 44.98 0.00 0.00 52.91
    10:50:01 3 1.72 0.00 44.29 0.03 0.00 53.96
    11:00:01 all 2.08 0.00 30.96 0.10 0.00 66.86
    11:00:01 0 2.28 0.00 21.01 0.28 0.00 76.42
    11:00:01 1 1.74 0.00 27.10 0.10 0.00 71.07
    11:00:01 2 2.99 0.00 28.46 0.02 0.00 68.54
    11:00:01 3 1.29 0.00 47.29 0.01 0.00 51.41
    11:10:01 all 2.03 0.00 25.21 0.01 0.00 72.75
    11:10:01 0 1.71 0.00 24.16 0.00 0.00 74.13
    11:10:01 1 1.80 0.00 19.12 0.01 0.00 79.07
    11:10:01 2 2.34 0.00 25.46 0.03 0.00 72.16
    11:10:01 3 2.26 0.00 32.09 0.01 0.00 65.64
    11:20:01 all 2.13 0.00 25.55 0.10 0.00 72.22
    11:20:01 0 1.99 0.00 23.45 0.29 0.00 74.26
    11:20:01 1 1.94 0.00 21.18 0.03 0.00 76.85
    11:20:01 2 2.04 0.00 31.44 0.04 0.00 66.48
    11:20:01 3 2.53 0.00 26.15 0.02 0.00 71.29
    11:30:01 all 2.02 0.00 21.11 0.09 0.00 76.78
    11:30:01 0 2.00 0.00 16.35 0.30 0.00 81.34
    11:30:01 1 1.30 0.00 24.02 0.03 0.00 74.64
    11:30:01 2 2.32 0.00 22.82 0.03 0.00 74.83
    11:30:01 3 2.46 0.00 21.24 0.01 0.00 76.29

    11:30:01 CPU %user %nice %system %iowait %steal %idle
    11:40:01 all 2.03 0.00 22.19 0.01 0.00 75.77
    11:40:01 0 1.76 0.00 23.37 0.00 0.00 74.86
    11:40:01 1 1.84 0.00 18.97 0.03 0.00 79.16
    11:40:01 2 2.38 0.00 21.73 0.03 0.00 75.87
    11:40:01 3 2.12 0.00 24.71 0.00 0.00 73.18
    11:50:01 all 4.52 0.10 24.54 0.14 0.00 70.70
    11:50:01 0 2.23 0.02 30.37 0.24 0.00 67.14
    11:50:01 1 3.79 0.10 20.06 0.17 0.00 75.88
    11:50:01 2 5.13 0.09 23.37 0.04 0.00 71.36
    11:50:01 3 6.94 0.18 24.33 0.13 0.00 68.41
    12:00:01 all 2.03 0.00 23.32 0.12 0.00 74.53
    12:00:01 0 2.21 0.00 17.86 0.28 0.00 79.65
    12:00:01 1 1.33 0.00 22.82 0.07 0.00 75.78
    12:00:01 2 2.55 0.00 24.62 0.06 0.00 72.78
    12:00:01 3 2.01 0.00 27.98 0.07 0.00 69.93
    12:10:01 all 2.65 0.00 26.93 0.03 0.00 70.39
    12:10:01 0 2.13 0.00 31.23 0.00 0.00 66.64
    12:10:01 1 2.43 0.00 20.13 0.04 0.00 77.40
    12:10:01 2 3.01 0.00 26.90 0.03 0.00 70.06
    12:10:01 3 3.04 0.00 29.46 0.03 0.00 67.46
    12:20:01 all 2.86 0.00 25.19 0.11 0.00 71.84
    12:20:01 0 2.96 0.00 22.96 0.33 0.00 73.74
    12:20:01 1 1.99 0.00 22.23 0.03 0.00 75.75
    12:20:01 2 3.67 0.00 24.20 0.05 0.00 72.08
    12:20:01 3 2.80 0.00 31.38 0.02 0.00 65.80
    12:30:01 all 2.38 0.00 26.45 0.13 0.00 71.04
    12:30:01 0 2.59 0.00 20.15 0.30 0.00 76.96
    12:30:01 1 1.65 0.00 26.77 0.02 0.00 71.56
    12:30:01 2 2.91 0.00 26.71 0.18 0.00 70.20
    12:30:01 3 2.39 0.00 32.16 0.01 0.00 65.44
    12:40:01 all 2.32 0.00 28.57 0.06 0.00 69.06
    12:40:01 0 1.97 0.00 28.84 0.00 0.00 69.20
    12:40:01 1 1.74 0.00 26.75 0.13 0.00 71.37
    12:40:01 2 2.78 0.00 27.28 0.06 0.00 69.87
    12:40:01 3 2.78 0.00 31.40 0.03 0.00 65.79
    12:50:01 all 2.24 0.00 30.34 0.12 0.00 67.29
    12:50:01 0 2.08 0.00 28.76 0.20 0.00 68.96
    12:50:01 1 1.32 0.00 31.39 0.05 0.00 67.23
    12:50:01 2 2.73 0.00 30.42 0.09 0.00 66.76
    12:50:01 3 2.85 0.00 30.81 0.13 0.00 66.20
    Average: all 2.05 0.00 20.67 0.10 0.00 77.17
    Average: 0 2.16 0.00 15.17 0.21 0.00 82.46
    Average: 1 1.43 0.00 25.58 0.05 0.00 72.93
    Average: 2 2.15 0.01 20.57 0.07 0.00 77.21
    Average: 3 2.47 0.01 21.38 0.07 0.00 76.08


    [Expert@gate02:0]# fwaccel stats -s
    Accelerated conns/Total conns : 45/10203 (0%)
    Accelerated pkts/Total pkts : 30463/33121932 (0%)
    F2Fed pkts/Total pkts : 790706/33121932 (2%)
    PXL pkts/Total pkts : 32300763/33121932 (97%)
    QXL pkts/Total pkts : 0/33121932 (0%)

    [Expert@gate02:0]# fwaccel stat
    Accelerator Status : on
    Accept Templates : disabled by Firewall
    disabled from rule #243
    Drop Templates : disabled
    NAT Templates : disabled by user

    Accelerator Features : Accounting, NAT, Cryptography, Routing,
    HasClock, Templates, Synchronous, IdleDetection,
    Sequencing, TcpStateDetect, AutoExpire,
    DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
    WireMode, DropTemplates, NatTemplates,
    Streaming, MultiFW, AntiSpoofing, ViolationStats,
    Nac, AsychronicNotif, ERDOS, McastRoutingV2
    Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
    3DES, DES, CAST, CAST-40, AES-128, AES-256,
    ESP, LinkSelection, DynamicVPN, NatTraversal,
    EncRouting, AES-XCBC, SHA256

    Bad performance in my case means that when i copy large files between the servers in different vlans i get only 10-20MB/s which is very slow as i think.

  3. #43
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Well it definitely does not appear to be a firewall CPU bottleneck and automatic affinity is doing a reasonable job of spreading around IRQ processing which tracks with a low RX-DRP rate. The fact that very few connections are able to be templated by SecureXL is a little strange, but tuning in that area just helps save CPU on the Firewall Worker cores which are not overloaded to begin with. This assumes of course that the provided sar output is showing the firewall's busiest period. Almost all your traffic is being processed in the Medium Path (PXL) which is not that unusual and it looks like the Firewall Worker cores are able to keep up with the load. Try this:

    • Start a continuous ping across the firewall for a few minutes and note the average latency numbers and any loss.
    • Now start one of your slow file transfers. What happens to the latency and/or loss on your continuous ping? We still need to figure out if it is latency or loss that is slowing you down.
    • While the transfer is running, run top on the firewall and then hit a numeric "1". Any individual CPUs below 25% idle? My guess is no.

    During a maintenance window, you could try disabling features one by one that force use of the Medium Path on the firewall (based on your previous posts it looks like these would be App control, IPS detect mode, URL filtering for your site) and see if performance dramatically improves right after one of them is turned off. Frankly I'm not sure even turning them all off will make much of a difference. If it does you could have a firewall topology issue that is forcing traffic between internal networks to get processed by APCL/URLF which is a big no-no. Could definitely be something in IPS as well even if it is in detect mode.

    If you see sudden spikes of latency or loss in your ping when a slow transfer is running, but can't seem to correlate it to observable issues with the firewall, that may indicate an issue elsewhere in the network path. These can be notoriously difficult to find and will highly depend on the architecture of your network. The first few chapters of my book take you through discovering the path through your network and checking any network devices external to the firewall for the following five main things:

    • Excessive CPU Load
    • Low Free Memory
    • Network Interface Errors
    • Network Buffer Processing Errors
    • Suspiciously Short Device Uptime
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  4. #44
    Join Date
    2017-03-01
    Posts
    1
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by Opera View Post
    [Expert@gate02:0]# cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 2876448406 0 0 0 IO-APIC-edge timer
    1: 3 0 0 0 IO-APIC-edge i8042
    4: 502 0 0 0 IO-APIC-edge serial
    8: 3 0 0 0 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-level acpi
    12: 4 0 0 0 IO-APIC-edge i8042
    51: 11390230 451373469 144753102 45509666 PCI-MSI-X eth3
    59: 261403272 12366201 15013100 182359 PCI-MSI-X eth1
    122: 649 0 0 0 IO-APIC-level ehci_hcd:usb2
    130: 66 0 0 0 IO-APIC-level ehci_hcd:usb3
    146: 80851132 0 0 0 PCI-MSI-X cciss0
    185: 0 0 0 0 IO-APIC-level uhci_hcd:usb1
    202: 11834048 568954070 1295149294 2901189190 PCI-MSI-X eth4-TxRx-0
    210: 1 0 0 0 PCI-MSI-X eth4
    226: 3558238 344913786 1043530074 1527649647 PCI-MSI-X eth5-TxRx-0
    234: 1 0 0 0 PCI-MSI-X eth5
    NMI: 645291 724210 644017 694212
    LOC: 2876329589 2876329706 2876329670 2876326904
    ERR: 0
    MIS: 0

    SAR Command:


    07:40:01 CPU %user %nice %system %iowait %steal %idle
    07:50:01 all 1.97 0.00 16.85 0.12 0.00 81.07
    07:50:01 0 1.56 0.00 15.49 0.34 0.00 82.61
    07:50:01 1 1.95 0.00 11.07 0.08 0.00 86.90
    07:50:01 2 1.91 0.00 20.57 0.02 0.00 77.51
    07:50:01 3 2.45 0.00 20.27 0.04 0.00 77.24
    08:00:01 all 2.08 0.00 18.88 0.14 0.00 78.91
    08:00:01 0 2.14 0.00 14.02 0.36 0.00 83.47
    08:00:01 1 2.25 0.00 13.84 0.04 0.00 83.86
    08:00:01 2 2.02 0.00 22.38 0.03 0.00 75.56
    08:00:01 3 1.89 0.00 25.25 0.12 0.00 72.74
    08:10:01 all 2.33 0.00 22.65 0.02 0.00 75.00
    08:10:01 0 2.26 0.00 20.04 0.00 0.00 77.70
    08:10:01 1 1.92 0.00 20.13 0.03 0.00 77.92
    08:10:01 2 2.71 0.00 22.00 0.02 0.00 75.27
    08:10:01 3 2.41 0.00 28.45 0.03 0.00 69.11
    08:20:01 all 2.75 0.00 22.10 0.15 0.00 75.00
    08:20:01 0 2.82 0.00 16.95 0.25 0.00 79.98
    08:20:01 1 2.18 0.00 20.50 0.07 0.00 77.25
    08:20:01 2 3.31 0.00 23.87 0.08 0.00 72.74
    08:20:01 3 2.70 0.00 27.08 0.19 0.00 70.03
    08:30:01 all 2.07 0.00 24.81 0.08 0.00 73.04
    08:30:01 0 2.29 0.00 18.28 0.24 0.00 79.19
    08:30:01 1 1.95 0.00 21.31 0.05 0.00 76.69
    08:30:01 2 2.38 0.00 25.19 0.03 0.00 72.39
    08:30:01 3 1.65 0.00 34.44 0.02 0.00 63.89
    08:40:01 all 2.19 0.00 24.90 0.01 0.00 72.90
    08:40:01 0 2.21 0.00 19.38 0.00 0.00 78.41
    08:40:01 1 1.61 0.00 23.75 0.01 0.00 74.62
    08:40:01 2 1.88 0.00 32.95 0.00 0.00 65.17
    08:40:01 3 3.08 0.00 23.50 0.03 0.00 73.39
    08:50:05 all 4.94 0.09 31.07 0.05 0.00 63.86
    08:50:05 0 3.83 0.03 28.55 0.03 0.00 67.56
    08:50:05 1 6.03 0.06 23.87 0.13 0.00 69.92
    08:50:05 2 4.69 0.26 33.76 0.02 0.00 61.28
    08:50:05 3 5.22 0.01 38.09 0.01 0.00 56.67
    09:00:11 all 2.43 0.00 33.51 0.07 0.00 63.99
    09:00:11 0 2.69 0.00 26.66 0.16 0.00 70.49
    09:00:11 1 2.22 0.00 34.40 0.05 0.00 63.33
    09:00:11 2 2.88 0.00 29.36 0.06 0.00 67.69
    09:00:11 3 1.91 0.00 43.63 0.00 0.00 54.46
    09:10:01 all 2.29 0.00 36.34 0.03 0.00 61.35
    09:10:01 0 1.77 0.00 36.13 0.00 0.00 62.10
    09:10:01 1 2.93 0.00 31.14 0.05 0.00 65.89
    09:10:01 2 3.32 0.00 31.30 0.02 0.00 65.35
    09:10:01 3 1.15 0.00 46.78 0.03 0.00 52.03
    09:20:02 all 2.79 0.00 34.51 0.12 0.00 62.58
    09:20:02 0 2.30 0.00 30.45 0.21 0.00 67.04
    09:20:02 1 3.42 0.00 35.32 0.15 0.00 61.10
    09:20:02 2 3.95 0.00 30.09 0.07 0.00 65.89
    09:20:02 3 1.48 0.00 42.19 0.03 0.00 56.30
    09:30:01 all 2.24 0.00 38.65 0.08 0.00 59.03
    09:30:01 0 2.64 0.00 25.00 0.16 0.00 72.20
    09:30:01 1 1.88 0.00 41.83 0.10 0.00 56.18
    09:30:01 2 2.88 0.00 34.72 0.03 0.00 62.38
    09:30:01 3 1.54 0.00 53.04 0.04 0.00 45.38
    09:40:01 all 2.32 0.00 34.78 0.01 0.00 62.89
    09:40:01 0 2.11 0.00 27.64 0.00 0.00 70.24
    09:40:01 1 2.05 0.00 37.41 0.01 0.00 60.53
    09:40:01 2 3.08 0.00 33.66 0.04 0.00 63.22
    09:40:01 3 2.04 0.00 40.40 0.00 0.00 57.55
    09:50:01 all 2.35 0.00 32.43 0.06 0.00 65.15
    09:50:01 0 2.31 0.00 23.63 0.18 0.00 73.88
    09:50:01 1 1.76 0.00 40.66 0.00 0.00 57.58
    09:50:01 2 3.35 0.00 33.21 0.03 0.00 63.41
    09:50:01 3 1.99 0.00 32.24 0.03 0.00 65.74
    10:00:01 all 2.33 0.00 36.60 0.07 0.00 61.01
    10:00:01 0 2.37 0.00 27.65 0.16 0.00 69.82
    10:00:01 1 2.13 0.00 45.61 0.04 0.00 52.22
    10:00:01 2 3.18 0.00 36.31 0.02 0.00 60.49
    10:00:01 3 1.62 0.00 36.82 0.05 0.00 61.50
    10:10:01 all 2.49 0.00 34.98 0.01 0.00 62.51
    10:10:01 0 2.60 0.00 27.78 0.00 0.00 69.62
    10:10:01 1 1.84 0.00 39.50 0.02 0.00 58.64
    10:10:01 2 3.35 0.00 33.75 0.02 0.00 62.88
    10:10:01 3 2.17 0.00 38.90 0.01 0.00 58.92
    10:20:01 all 2.25 0.00 32.47 0.12 0.00 65.16
    10:20:01 0 1.91 0.00 30.45 0.27 0.00 67.37
    10:20:01 1 2.39 0.00 27.66 0.09 0.00 69.87
    10:20:01 2 2.86 0.00 31.11 0.11 0.00 65.92
    10:20:01 3 1.85 0.00 40.65 0.03 0.00 57.47
    10:30:01 all 2.16 0.00 34.33 0.40 0.00 63.11
    10:30:01 0 1.84 0.00 29.60 0.25 0.00 68.31
    10:30:01 1 1.91 0.00 38.71 0.41 0.00 58.97
    10:30:01 2 3.32 0.00 30.20 0.49 0.00 65.99
    10:30:01 3 1.55 0.00 38.82 0.46 0.00 59.17
    10:40:01 all 2.13 0.00 35.85 0.01 0.00 62.02
    10:40:01 0 2.11 0.00 28.81 0.00 0.00 69.08
    10:40:01 1 2.05 0.00 47.53 0.01 0.00 50.40
    10:40:01 2 3.17 0.00 28.41 0.01 0.00 68.41
    10:40:01 3 1.18 0.00 38.64 0.01 0.00 60.18
    10:50:01 all 2.19 0.00 37.03 0.07 0.00 60.72
    10:50:01 0 2.35 0.00 28.11 0.19 0.00 69.35
    10:50:01 1 2.58 0.00 30.74 0.04 0.00 66.64
    10:50:01 2 2.10 0.00 44.98 0.00 0.00 52.91
    10:50:01 3 1.72 0.00 44.29 0.03 0.00 53.96
    11:00:01 all 2.08 0.00 30.96 0.10 0.00 66.86
    11:00:01 0 2.28 0.00 21.01 0.28 0.00 76.42
    11:00:01 1 1.74 0.00 27.10 0.10 0.00 71.07
    11:00:01 2 2.99 0.00 28.46 0.02 0.00 68.54
    11:00:01 3 1.29 0.00 47.29 0.01 0.00 51.41
    11:10:01 all 2.03 0.00 25.21 0.01 0.00 72.75
    11:10:01 0 1.71 0.00 24.16 0.00 0.00 74.13
    11:10:01 1 1.80 0.00 19.12 0.01 0.00 79.07
    11:10:01 2 2.34 0.00 25.46 0.03 0.00 72.16
    11:10:01 3 2.26 0.00 32.09 0.01 0.00 65.64
    11:20:01 all 2.13 0.00 25.55 0.10 0.00 72.22
    11:20:01 0 1.99 0.00 23.45 0.29 0.00 74.26
    11:20:01 1 1.94 0.00 21.18 0.03 0.00 76.85
    11:20:01 2 2.04 0.00 31.44 0.04 0.00 66.48
    11:20:01 3 2.53 0.00 26.15 0.02 0.00 71.29
    11:30:01 all 2.02 0.00 21.11 0.09 0.00 76.78
    11:30:01 0 2.00 0.00 16.35 0.30 0.00 81.34
    11:30:01 1 1.30 0.00 24.02 0.03 0.00 74.64
    11:30:01 2 2.32 0.00 22.82 0.03 0.00 74.83
    11:30:01 3 2.46 0.00 21.24 0.01 0.00 76.29

    11:30:01 CPU %user %nice %system %iowait %steal %idle
    11:40:01 all 2.03 0.00 22.19 0.01 0.00 75.77
    11:40:01 0 1.76 0.00 23.37 0.00 0.00 74.86
    11:40:01 1 1.84 0.00 18.97 0.03 0.00 79.16
    11:40:01 2 2.38 0.00 21.73 0.03 0.00 75.87
    11:40:01 3 2.12 0.00 24.71 0.00 0.00 73.18
    11:50:01 all 4.52 0.10 24.54 0.14 0.00 70.70
    11:50:01 0 2.23 0.02 30.37 0.24 0.00 67.14
    11:50:01 1 3.79 0.10 20.06 0.17 0.00 75.88
    11:50:01 2 5.13 0.09 23.37 0.04 0.00 71.36
    11:50:01 3 6.94 0.18 24.33 0.13 0.00 68.41
    12:00:01 all 2.03 0.00 23.32 0.12 0.00 74.53
    12:00:01 0 2.21 0.00 17.86 0.28 0.00 79.65
    12:00:01 1 1.33 0.00 22.82 0.07 0.00 75.78
    12:00:01 2 2.55 0.00 24.62 0.06 0.00 72.78
    12:00:01 3 2.01 0.00 27.98 0.07 0.00 69.93
    12:10:01 all 2.65 0.00 26.93 0.03 0.00 70.39
    12:10:01 0 2.13 0.00 31.23 0.00 0.00 66.64
    12:10:01 1 2.43 0.00 20.13 0.04 0.00 77.40
    12:10:01 2 3.01 0.00 26.90 0.03 0.00 70.06
    12:10:01 3 3.04 0.00 29.46 0.03 0.00 67.46
    12:20:01 all 2.86 0.00 25.19 0.11 0.00 71.84
    12:20:01 0 2.96 0.00 22.96 0.33 0.00 73.74
    12:20:01 1 1.99 0.00 22.23 0.03 0.00 75.75
    12:20:01 2 3.67 0.00 24.20 0.05 0.00 72.08
    12:20:01 3 2.80 0.00 31.38 0.02 0.00 65.80
    12:30:01 all 2.38 0.00 26.45 0.13 0.00 71.04
    12:30:01 0 2.59 0.00 20.15 0.30 0.00 76.96
    12:30:01 1 1.65 0.00 26.77 0.02 0.00 71.56
    12:30:01 2 2.91 0.00 26.71 0.18 0.00 70.20
    12:30:01 3 2.39 0.00 32.16 0.01 0.00 65.44
    12:40:01 all 2.32 0.00 28.57 0.06 0.00 69.06
    12:40:01 0 1.97 0.00 28.84 0.00 0.00 69.20
    12:40:01 1 1.74 0.00 26.75 0.13 0.00 71.37
    12:40:01 2 2.78 0.00 27.28 0.06 0.00 69.87
    12:40:01 3 2.78 0.00 31.40 0.03 0.00 65.79
    12:50:01 all 2.24 0.00 30.34 0.12 0.00 67.29
    12:50:01 0 2.08 0.00 28.76 0.20 0.00 68.96
    12:50:01 1 1.32 0.00 31.39 0.05 0.00 67.23
    12:50:01 2 2.73 0.00 30.42 0.09 0.00 66.76
    12:50:01 3 2.85 0.00 30.81 0.13 0.00 66.20
    Average: all 2.05 0.00 20.67 0.10 0.00 77.17
    Average: 0 2.16 0.00 15.17 0.21 0.00 82.46
    Average: 1 1.43 0.00 25.58 0.05 0.00 72.93
    Average: 2 2.15 0.01 20.57 0.07 0.00 77.21
    Average: 3 2.47 0.01 21.38 0.07 0.00 76.08


    [Expert@gate02:0]# fwaccel stats -s
    Accelerated conns/Total conns : 45/10203 (0%)
    Accelerated pkts/Total pkts : 30463/33121932 (0%)
    F2Fed pkts/Total pkts : 790706/33121932 (2%)
    PXL pkts/Total pkts : 32300763/33121932 (97%)
    QXL pkts/Total pkts : 0/33121932 (0%)

    [Expert@gate02:0]# fwaccel stat
    Accelerator Status : on
    Accept Templates : disabled by Firewall
    disabled from rule #243
    Drop Templates : disabled
    NAT Templates : disabled by user

    Accelerator Features : Accounting, NAT, Cryptography, Routing,
    HasClock, Templates, Synchronous, IdleDetection,
    Sequencing, TcpStateDetect, AutoExpire,
    DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
    WireMode, DropTemplates, NatTemplates,
    Streaming, MultiFW, AntiSpoofing, ViolationStats,
    Nac, AsychronicNotif, ERDOS, McastRoutingV2
    Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
    3DES, DES, CAST, CAST-40, AES-128, AES-256,
    ESP, LinkSelection, DynamicVPN, NatTraversal,
    EncRouting, AES-XCBC, SHA256

    Bad performance in my case means that when i copy large files between the servers in different vlans i get only 10-20MB/s which is very slow as i think.
    Hi Opera,

    I am troubleshooting a performance issue that appears to be very similar to yours. On the R77.10 VSX gateway, I am only averaging 10-40MB/s copying large files of approx. 100M between two VLANs connected to the same VS. I get the occasional fast transfer (90-100MB/s). Did you ever find a solution or improve performance?

    Thanks,


    Jonne.

  5. #45
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    hello
    i have same issue with my network
    2 gw are working on network cluster xl with cisco nexus (vpc )
    i see 400 500 mb/ps when i test by ipref tool on local network but test result is 20 30 mb/ps between local and dmz network

    all gw have 10 gb inteface card

    how did you solve this problem

  6. #46
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    hello
    i have same issue with my network
    2 gw are working on network cluster xl with cisco nexus (vpc )
    i see 400 500 mb/ps when i test by ipref tool on local network but test result is 20 30 mb/ps between local and dmz network

    all gw have 10 gb inteface card

    how did you solve this problem
    What is the hardware type of the firewalls?

    Please provide output of the following "Super Seven" performance assessment commands run on the active firewall member during a long slow transfer:

    fwaccel stat
    fwaccel stats -s
    grep -c ^processor /proc/cpuinfo
    fw ctl affinity -l -r
    netstat -ni
    fw ctl multik stat
    cpstat os -f multi_cpu -o 1
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  7. #47
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    Hello

    command resul is below

    [Expert@CPGW02:0]# fwaccel stat
    Accelerator Status : on
    Accept Templates : enabled
    Drop Templates : disabled
    NAT Templates : disabled by user

    Accelerator Features : Accounting, NAT, Cryptography, Routing,
    HasClock, Templates, Synchronous, IdleDetection,
    Sequencing, TcpStateDetect, AutoExpire,
    DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
    WireMode, DropTemplates, NatTemplates,
    Streaming, MultiFW, AntiSpoofing, Nac,
    ViolationStats, AsychronicNotif, ERDOS,
    NAT64, GTPAcceleration, SCTPAcceleration,
    McastRoutingV2
    Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
    3DES, DES, CAST, CAST-40, AES-128, AES-256,
    ESP, LinkSelection, DynamicVPN, NatTraversal,
    EncRouting, AES-XCBC, SHA256
    [Expert@CPGW02:0]# fwaccel stats -s
    Accelerated conns/Total conns : 892/22374 (3%)
    Accelerated pkts/Total pkts : 406144/3996782 (10%)
    F2Fed pkts/Total pkts : 986669/3996782 (24%)
    PXL pkts/Total pkts : 2603969/3996782 (65%)
    QXL pkts/Total pkts : 0/3996782 (0%)
    [Expert@CPGW02:0]# grep -c ^processor /proc/cpuinfo
    8
    [Expert@CPGW02:0]# fw ctl affinity -l -r
    CPU 0: eth8 eth0 eth1 eth2
    CPU 1: eth4 eth5 eth11 eth3
    CPU 2: fw_5
    CPU 3: fw_4
    CPU 4: fw_3
    CPU 5: fw_2
    CPU 6: fw_1
    CPU 7: fw_0
    All: in.msd vpnd pdpd usrchkd in.acapd pepd rad fwd mpdaemon fwpushd rtmd cpd cprid
    [Expert@CPGW02:0]# netstat -ni
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    bond1 1500 0 979512121 0 2378 0 1297527898 0 0 0 BMmRU
    bond1.20 1500 0 931034529 0 0 0 1297547505 0 0 0 BMmRU
    bond2 1500 0 346477067 0 82 0 306145365 0 0 0 BMmRU
    bond2.34 1500 0 345027255 0 0 0 302250627 0 0 0 BMmRU
    bond2.106 1500 0 521579 0 0 0 1330266 0 0 0 BMmRU
    bond2.107 1500 0 496730 0 0 0 2542306 0 0 0 BMmRU
    eth0 1500 0 190769070 0 82 0 145162388 0 0 0 BMsRU
    eth1 1500 0 967930979 0 2378 0 843804189 0 0 0 BMsRU
    eth2 1500 0 155708002 0 0 0 160982985 0 0 0 BMsRU
    eth3 1500 0 11581246 0 0 0 453723874 0 0 0 BMsRU
    eth4 1500 0 27844208 0 0 0 19852352 0 0 0 BMRU
    eth5 1500 0 980368184 0 302432 302432 774786946 0 0 0 BMRU
    eth8 1500 0 297716 0 0 0 4886 0 0 0 BMRU
    eth11 1500 0 20419612 0 165 165 78261723 0 0 0 BMRU
    lo 16436 0 26736535 0 0 0 26736535 0 0 0 LRU
    [Expert@CPGW02:0]# fw ctl multik stat
    ID | Active | CPU | Connections | Peak
    ----------------------------------------------
    0 | Yes | 7 | 5157 | 7597
    1 | Yes | 6 | 4872 | 5966
    2 | Yes | 5 | 4848 | 7426
    3 | Yes | 4 | 4665 | 6166
    4 | Yes | 3 | 5137 | 8813
    5 | Yes | 2 | 6313 | 7535
    [Expert@CPGW02:0]# cpstat os -f multi_cpu -o 1



    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 2| 6| 92| 8| ?| 0|
    | 2| 2| 6| 91| 9| ?| 0|
    | 3| 1| 12| 87| 13| ?| 0|
    | 4| 1| 13| 86| 14| ?| 0|
    | 5| 1| 13| 87| 13| ?| 0|
    | 6| 1| 13| 86| 14| ?| 0|
    | 7| 1| 14| 85| 15| ?| 0|
    | 8| 1| 14| 84| 16| ?| 0|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 2| 6| 92| 8| ?| 0|
    | 2| 2| 6| 91| 9| ?| 0|
    | 3| 1| 12| 87| 13| ?| 0|
    | 4| 1| 13| 86| 14| ?| 0|
    | 5| 1| 13| 87| 13| ?| 0|
    | 6| 1| 13| 86| 14| ?| 0|
    | 7| 1| 14| 85| 15| ?| 0|
    | 8| 1| 14| 84| 16| ?| 0|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 12| 19| 70| 30| ?| 28665|
    | 2| 11| 28| 62| 38| ?| 28667|
    | 3| 2| 62| 36| 64| ?| 28669|
    | 4| 1| 71| 28| 72| ?| 28672|
    | 5| 5| 38| 58| 42| ?| 28675|
    | 6| 3| 70| 27| 73| ?| 28676|
    | 7| 5| 54| 41| 59| ?| 28677|
    | 8| 6| 52| 42| 58| ?| 57360|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 12| 19| 70| 30| ?| 28665|
    | 2| 11| 28| 62| 38| ?| 28667|
    | 3| 2| 62| 36| 64| ?| 28669|
    | 4| 1| 71| 28| 72| ?| 28672|
    | 5| 5| 38| 58| 42| ?| 28675|
    | 6| 3| 70| 27| 73| ?| 28676|
    | 7| 5| 54| 41| 59| ?| 28677|
    | 8| 6| 52| 42| 58| ?| 57360|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 8| 16| 76| 24| ?| 35472|
    | 2| 8| 17| 75| 25| ?| 35473|
    | 3| 4| 49| 47| 53| ?| 35475|
    | 4| 2| 86| 13| 87| ?| 35478|
    | 5| 4| 62| 34| 66| ?| 35479|
    | 6| 2| 77| 21| 79| ?| 70965|
    | 7| 5| 66| 30| 70| ?| 35486|
    | 8| 4| 64| 32| 68| ?| 35490|
    ---------------------------------------------------------------------------------



    [Expert@CPGW02:0]#

  8. #48
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    Hello

    all gw open server

    [Expert@CPGW02:0]# fwaccel stat
    Accelerator Status : on
    Accept Templates : enabled
    Drop Templates : disabled
    NAT Templates : disabled by user

    Accelerator Features : Accounting, NAT, Cryptography, Routing,
    HasClock, Templates, Synchronous, IdleDetection,
    Sequencing, TcpStateDetect, AutoExpire,
    DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
    WireMode, DropTemplates, NatTemplates,
    Streaming, MultiFW, AntiSpoofing, Nac,
    ViolationStats, AsychronicNotif, ERDOS,
    NAT64, GTPAcceleration, SCTPAcceleration,
    McastRoutingV2
    Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
    3DES, DES, CAST, CAST-40, AES-128, AES-256,
    ESP, LinkSelection, DynamicVPN, NatTraversal,
    EncRouting, AES-XCBC, SHA256
    [Expert@CPGW02:0]# fwaccel stats -s
    Accelerated conns/Total conns : 224/28099 (0%)
    Accelerated pkts/Total pkts : 515038/183162461 (0%)
    F2Fed pkts/Total pkts : 50622616/183162461 (27%)
    PXL pkts/Total pkts : 132024807/183162461 (72%)
    QXL pkts/Total pkts : 0/183162461 (0%)
    [Expert@CPGW02:0]# grep -c ^processor /proc/cpuinfo
    8
    [Expert@CPGW02:0]# fw ctl affinity -l -r
    CPU 0: eth8 eth1 eth2 eth3
    CPU 1: eth4 eth5 eth11 eth0
    CPU 2: fw_5
    CPU 3: fw_4
    CPU 4: fw_3
    CPU 5: fw_2
    CPU 6: fw_1
    CPU 7: fw_0
    All: vpnd pdpd usrchkd pepd rad fwd mpdaemon fwpushd rtmd cpd cprid
    [Expert@CPGW02:0]# netstat -ni
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    bond1 1500 0 1177769705 0 5571 0 1595064431 0 0 0 BMmRU
    bond1.20 1500 0 1123219949 0 0 0 1595074770 0 0 0 BMmRU
    bond2 1500 0 385861325 0 82 0 338242808 0 0 0 BMmRU
    bond2.34 1500 0 384249023 0 0 0 333929993 0 0 0 BMmRU
    bond2.106 1500 0 611273 0 0 0 1548826 0 0 0 BMmRU
    bond2.107 1500 0 535440 0 0 0 2739597 0 0 0 BMmRU
    eth0 1500 0 217613515 0 82 0 159256106 0 0 0 BMsRU
    eth1 1500 0 1164444468 0 5571 0 1055916459 0 0 0 BMsRU
    eth2 1500 0 168247819 0 0 0 178986713 0 0 0 BMsRU
    eth3 1500 0 13325335 0 0 0 539148111 0 0 0 BMsRU
    eth4 1500 0 33919836 0 0 0 24502300 0 0 0 BMRU
    eth5 1500 0 1239838999 0 375124 375124 979502286 0 0 0 BMRU
    eth8 1500 0 326042 0 0 0 5292 0 0 0 BMRU
    eth11 1500 0 22006335 0 212 212 95669018 0 0 0 BMRU
    lo 16436 0 34467492 0 0 0 34467492 0 0 0 LRU
    [Expert@CPGW02:0]# fw ctl multik stat
    ID | Active | CPU | Connections | Peak
    ----------------------------------------------
    0 | Yes | 7 | 4382 | 7597
    1 | Yes | 6 | 4359 | 5966
    2 | Yes | 5 | 4222 | 7426
    3 | Yes | 4 | 4078 | 6166
    4 | Yes | 3 | 4472 | 8813
    5 | Yes | 2 | 5245 | 7535
    [Expert@CPGW02:0]# cpstat os -f multi_cpu -o 1



    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 6| 14| 80| 20| ?| 0|
    | 2| 6| 16| 78| 22| ?| 0|
    | 3| 2| 30| 68| 32| ?| 0|
    | 4| 3| 31| 66| 34| ?| 0|
    | 5| 2| 31| 67| 33| ?| 0|
    | 6| 2| 31| 67| 33| ?| 0|
    | 7| 3| 33| 64| 36| ?| 0|
    | 8| 3| 35| 62| 38| ?| 0|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 6| 14| 80| 20| ?| 0|
    | 2| 6| 16| 78| 22| ?| 0|
    | 3| 2| 30| 68| 32| ?| 0|
    | 4| 3| 31| 66| 34| ?| 0|
    | 5| 2| 31| 67| 33| ?| 0|
    | 6| 2| 31| 67| 33| ?| 0|
    | 7| 3| 33| 64| 36| ?| 0|
    | 8| 3| 35| 62| 38| ?| 0|
    ---------------------------------------------------------------------------------





    Processors load
    ---------------------------------------------------------------------------------
    |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
    ---------------------------------------------------------------------------------
    | 1| 5| 9| 85| 15| ?| 33042|
    | 2| 7| 13| 80| 20| ?| 33044|
    | 3| 3| 42| 55| 45| ?| 33045|
    | 4| 2| 47| 52| 48| ?| 66096|
    | 5| 1| 31| 69| 31| ?| 33050|
    | 6| 3| 26| 71| 29| ?| 33052|
    | 7| 3| 34| 63| 37| ?| 33054|
    | 8| 2| 38| 60| 40| ?| 66111|
    ---------------------------------------------------------------------------------



    [Expert@CPGW02:0]#

  9. #49
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Everything looks fairly well balanced here, the firewall just looks quite busy. The only possible thing I see here is this:

    Quote Originally Posted by coskun_ist View Post

    eth5 1500 0 1239838999 0 375124 375124 979502286 0 0 0 BMRU
    1) You are taking overruns & drops on your eth5 interface, by chance is this interface involved with the slow communication? Do the RX-DRP/RX-OVR counters for this interface increment during a slow transfer?

    2) Please provide the output of these further commands on the active cluster member:

    fw ver
    fw ctl multik get_mode
    enabled_blades (if that doesn't work try enabled_blades.sh)
    free -m
    ethtool -i eth5
    ethtool -S eth5
    ethtool -i eth1
    ethtool -S eth1
    /sbin/cpuinfo

    I hope you are not using Broadcom or Emulex NICs on your open hardware system as their performance will be terrible no matter what you do...

    3) Assuming the above all check out, it will be time to look at the surrounding switchports and host NICs involved with the slow communication for errors. Your firewall is actually in much better shape than most as far as optimization.
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  10. #50
    Join Date
    2006-09-26
    Posts
    2,827
    Rep Power
    13

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    [Expert@CPGW02:0]# fwaccel stats -s
    Accelerated conns/Total conns : 224/28099 (0%)
    Accelerated pkts/Total pkts : 515038/183162461 (0%)
    F2Fed pkts/Total pkts : 50622616/183162461 (27%)
    PXL pkts/Total pkts : 132024807/183162461 (72%)
    QXL pkts/Total pkts : 0/183162461 (0%)
    The reason your firewall is in terrible shape is because none of the traffics is being accelerated by SecureXL. 72% of the traffic is "partially" accelerated by the firewall. That is something you need to look into. Might be the cause of the slowness.

    This is what you should be seeing:

    Accelerated conns/Total conns : 7/26 (26%)
    Accelerated pkts/Total pkts : 2241723/2249700 (99%)
    F2Fed pkts/Total pkts : 7977/2249700 (0%)
    PXL pkts/Total pkts : 0/2249700 (0%)
    QXL pkts/Total pkts : 0/2249700 (0%)

  11. #51
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by cciesec2006 View Post
    The reason your firewall is in terrible shape is because none of the traffics is being accelerated by SecureXL. 72% of the traffic is "partially" accelerated by the firewall. That is something you need to look into. Might be the cause of the slowness.

    This is what you should be seeing:

    Accelerated conns/Total conns : 7/26 (26%)
    Accelerated pkts/Total pkts : 2241723/2249700 (99%)
    F2Fed pkts/Total pkts : 7977/2249700 (0%)
    PXL pkts/Total pkts : 0/2249700 (0%)
    QXL pkts/Total pkts : 0/2249700 (0%)
    Maybe, depends on which blades are enabled. I have requested that info.
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  12. #52
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    hello
    1 ) eth5 is my internet interface. we stilll have slow communication when we shutdown this interface.
    2 ) you can find the outputs you asked below

    [Expert@CPGW02:0]# fw ver
    This is Check Point's software version R77.30 - Build 034

    [Expert@CPGW02:0]# fw ctl multik get_mode
    Current mode is On

    [Expert@CPGW02:0]# enabled_blades
    fw vpn cvpn urlf appi ips identityServer SSL_INSPECT

    [Expert@CPGW02:0]# free -m
    total used free shared buffers cached
    Mem: 144811 20289 124521 0 447 1739
    -/+ buffers/cache: 18103 126708
    Swap: 12284 0 12284

    [Expert@CPGW02:0]# ethtool -i eth1
    driver: ixgbe
    version: 3.9.15-NAPI
    firmware-version: 0x80000868
    bus-info: 0000:1a:00.1

    [Expert@CPGW02:0]# ethtool -i eth3
    driver: ixgbe
    version: 3.9.15-NAPI
    firmware-version: 0x80000868
    bus-info: 0000:24:00.1

    [Expert@CPGW02:0]# ethtool -i eth0
    driver: ixgbe
    version: 3.9.15-NAPI
    firmware-version: 0x80000868
    bus-info: 0000:1a:00.0

    [Expert@CPGW02:0]# ethtool -i eth2
    driver: ixgbe
    version: 3.9.15-NAPI
    firmware-version: 0x80000868
    bus-info: 0000:24:00.0
    [Expert@CPGW02:0]# ethtool -i eth5
    driver: igb
    version: 4.1.2
    firmware-version: 3.0, 0x61a30001
    bus-info: 0000:15:00.1
    [Expert@CPGW02:0]# ethtool -S eth5
    NIC statistics:
    rx_packets: 551372217
    tx_packets: 438921035
    rx_bytes: 660948804668
    tx_bytes: 115098787091
    rx_broadcast: 381513
    tx_broadcast: 5039
    rx_multicast: 908636
    tx_multicast: 850432
    multicast: 908636
    collisions: 0
    rx_crc_errors: 0
    rx_no_buffer_count: 161
    rx_missed_errors: 16809
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_window_errors: 0
    tx_abort_late_coll: 0
    tx_deferred_ok: 0
    tx_single_coll_ok: 0
    tx_multi_coll_ok: 0
    tx_timeout_count: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    rx_align_errors: 0
    tx_tcp_seg_good: 0
    tx_tcp_seg_failed: 0
    rx_flow_control_xon: 0
    rx_flow_control_xoff: 0
    tx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_long_byte_count: 660948804668
    tx_dma_out_of_sync: 0
    lro_aggregated: 0
    lro_flushed: 0
    lro_recycled: 0
    tx_smbus: 0
    rx_smbus: 0
    dropped_smbus: 0
    os2bmc_rx_by_bmc: 0
    os2bmc_tx_by_bmc: 0
    os2bmc_tx_by_host: 0
    os2bmc_rx_by_host: 0
    rx_errors: 0
    tx_errors: 0
    tx_dropped: 0
    rx_length_errors: 0
    rx_over_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 16809
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    tx_queue_0_packets: 438921035
    tx_queue_0_bytes: 112248339969
    tx_queue_0_restart: 0
    rx_queue_0_packets: 551372217
    rx_queue_0_bytes: 658743315800
    rx_queue_0_drops: 0
    rx_queue_0_csum_err: 153587
    rx_queue_0_alloc_failed: 0
    [Expert@CPGW02:0]#


    [Expert@CPGW02:0]# ethtool -S eth1
    NIC statistics:
    rx_packets: 480694151
    tx_packets: 469714252
    rx_bytes: 138326521201
    tx_bytes: 516766853042
    rx_errors: 0
    tx_errors: 0
    rx_dropped: 0
    tx_dropped: 0
    multicast: 9189101
    collisions: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 0
    rx_missed_errors: 1488
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    rx_pkts_nic: 480694151
    tx_pkts_nic: 469714253
    rx_bytes_nic: 142171766339
    tx_bytes_nic: 520615914739
    lsc_int: 3
    tx_busy: 0
    non_eop_descs: 0
    broadcast: 7003242
    rx_no_buffer_count: 0
    tx_timeout_count: 0
    tx_restart_queue: 4
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    tx_flow_control_xon: 2
    rx_flow_control_xon: 0
    tx_flow_control_xoff: 791
    rx_flow_control_xoff: 0
    rx_csum_offload_errors: 452
    alloc_rx_page_failed: 0
    alloc_rx_buff_failed: 0
    rx_no_dma_resources: 0
    hw_rsc_aggregated: 0
    hw_rsc_flushed: 0
    fdir_match: 0
    fdir_miss: 0
    fdir_overflow: 0
    os2bmc_rx_by_bmc: 0
    os2bmc_tx_by_bmc: 0
    os2bmc_tx_by_host: 0
    os2bmc_rx_by_host: 0
    tx_queue_0_packets: 469714253
    tx_queue_0_bytes: 516766853108
    rx_queue_0_packets: 480694151
    rx_queue_0_bytes: 138326521201
    [Expert@CPGW02:0]#

    [Expert@CPGW02:0]# ethtool -S eth3
    NIC statistics:
    rx_packets: 5246700
    tx_packets: 231302446
    rx_bytes: 449074177
    tx_bytes: 258956587338
    rx_errors: 0
    tx_errors: 0
    rx_dropped: 0
    tx_dropped: 0
    multicast: 5149044
    collisions: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 0
    rx_missed_errors: 0
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    rx_pkts_nic: 5246700
    tx_pkts_nic: 231302446
    rx_bytes_nic: 490962405
    tx_bytes_nic: 260834179150
    lsc_int: 5
    tx_busy: 0
    non_eop_descs: 0
    broadcast: 97215
    rx_no_buffer_count: 0
    tx_timeout_count: 0
    tx_restart_queue: 1
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    tx_flow_control_xon: 0
    rx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_flow_control_xoff: 0
    rx_csum_offload_errors: 0
    alloc_rx_page_failed: 0
    alloc_rx_buff_failed: 0
    rx_no_dma_resources: 0
    hw_rsc_aggregated: 0
    hw_rsc_flushed: 0
    fdir_match: 0
    fdir_miss: 0
    fdir_overflow: 0
    os2bmc_rx_by_bmc: 0
    os2bmc_tx_by_bmc: 0
    os2bmc_tx_by_host: 0
    os2bmc_rx_by_host: 0
    tx_queue_0_packets: 231302446
    tx_queue_0_bytes: 258956587338
    rx_queue_0_packets: 5246700
    rx_queue_0_bytes: 449074177
    [Expert@CPGW02:0]#


    [Expert@CPGW02:0]# ethtool -S eth0
    NIC statistics:
    rx_packets: 78462184
    tx_packets: 56154414
    rx_bytes: 43594922619
    tx_bytes: 25547031021
    rx_errors: 0
    tx_errors: 0
    rx_dropped: 0
    tx_dropped: 0
    multicast: 1058434
    collisions: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 0
    rx_missed_errors: 0
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    rx_pkts_nic: 78462184
    tx_pkts_nic: 56154414
    rx_bytes_nic: 44222519943
    tx_bytes_nic: 26015093618
    lsc_int: 3
    tx_busy: 0
    non_eop_descs: 0
    broadcast: 72179
    rx_no_buffer_count: 0
    tx_timeout_count: 0
    tx_restart_queue: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    tx_flow_control_xon: 0
    rx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_flow_control_xoff: 0
    rx_csum_offload_errors: 0
    alloc_rx_page_failed: 0
    alloc_rx_buff_failed: 0
    rx_no_dma_resources: 0
    hw_rsc_aggregated: 0
    hw_rsc_flushed: 0
    fdir_match: 0
    fdir_miss: 0
    fdir_overflow: 0
    os2bmc_rx_by_bmc: 0
    os2bmc_tx_by_bmc: 0
    os2bmc_tx_by_host: 0
    os2bmc_rx_by_host: 0
    tx_queue_0_packets: 56154414
    tx_queue_0_bytes: 25547031021
    rx_queue_0_packets: 78462184
    rx_queue_0_bytes: 43594922619
    [Expert@CPGW02:0]#
    [Expert@CPGW02:0]# ethtool -S eth2
    NIC statistics:
    rx_packets: 85218744
    tx_packets: 65502377
    rx_bytes: 100387796187
    tx_bytes: 27947313866
    rx_errors: 0
    tx_errors: 0
    rx_dropped: 0
    tx_dropped: 0
    multicast: 488186
    collisions: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 0
    rx_missed_errors: 0
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    rx_pkts_nic: 85218744
    tx_pkts_nic: 65502377
    rx_bytes_nic: 101069224559
    tx_bytes_nic: 28491873002
    lsc_int: 3
    tx_busy: 0
    non_eop_descs: 0
    broadcast: 5769
    rx_no_buffer_count: 0
    tx_timeout_count: 0
    tx_restart_queue: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    tx_flow_control_xon: 0
    rx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_flow_control_xoff: 0
    rx_csum_offload_errors: 0
    alloc_rx_page_failed: 0
    alloc_rx_buff_failed: 0
    rx_no_dma_resources: 0
    hw_rsc_aggregated: 0
    hw_rsc_flushed: 0
    fdir_match: 0
    fdir_miss: 0
    fdir_overflow: 0
    os2bmc_rx_by_bmc: 0
    os2bmc_tx_by_bmc: 0
    os2bmc_tx_by_host: 0
    os2bmc_rx_by_host: 0
    tx_queue_0_packets: 65502377
    tx_queue_0_bytes: 27947313866
    rx_queue_0_packets: 85218744
    rx_queue_0_bytes: 100387796187
    [Expert@CPGW02:0]#
    [Expert@CPGW02:0]# /sbin/cpuinfo
    HyperThreading=disabled
    [Expert@CPGW02:0]#



    the card we use is of INTEL

    3 ) eth1 and eth3 form bond1 , which is our Local Network , and eth0 and eth2 form bond2 , which is our DMZ Network

    both bond1 and bond2 are connected to their own cisco nexus switches and VPC runs on these switches
    my topology is full mesh Redundancy

  13. #53
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    [Expert@CPGW02:0]# enabled_blades
    fw vpn cvpn urlf appi ips identityServer SSL_INSPECT
    A few observations:

    1) Check the topology screen of your firewall object. Is the External interface correctly defined? Are ALL other Internal interfaces of the firewall completely and correctly defined? Any missing interfaces defined in the Gaia OS but not the objects's topology will be assumed to be External and get traffic to/from them sucked into the PXL path at Gigabit speeds, not good at all for performance. Rectifying a topology issue like this may well fix everything else in the list below.

    2) The presence of HTTPS Inspection explains why 27% of your traffic is F2F. Check your HTTPS Inspection policy, in the Destination column of all HTTPS Inspection rules with an action of Inspect, are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" in the destination will cause HTTPS traffic between internal interfaces (including DMZs) to get pulled into the F2F path at LAN speeds.

    3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

    4) If there is a "Any Internet Any Log" rule at the end of your APCL/URLF Inspection policy (or even worse a "Any Any Any Log" rule), try removing it. Removing it will not cause an outage, the implied rule at the end of APCL/URLF policy is Allow.

    5) What is the setting on the IPS screen of the cluster for Protection Scope? If other steps above have not resolved the problem, run "ips off" on the active gateway, wait 60 seconds, then start a transfer in a completely new connection. Does performance improve? Don't forget to run "ips on" (or reinstall policy) when done!
    Last edited by ShadowPeak.com; 2 Weeks Ago at 19:38.
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  14. #54
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    hello
    your advice has worked and the problem is solved
    thanks for your support

  15. #55
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    hello
    your advice has worked and the problem is solved
    thanks for your support
    Thanks for the followup, can you please elaborate on which of the tips above solved the problem? Did you just need to correct/update the topology settings on your firewall object or were some policy adjustments required as well?
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

  16. #56
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    763
    Rep Power
    12

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by ShadowPeak.com View Post
    Thanks for the followup, can you please elaborate on which of the tips above solved the problem? Did you just need to correct/update the topology settings on your firewall object or were some policy adjustments required as well?
    Second that, would be nice to know which particular advice helped
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  17. #57
    Join Date
    2017-03-06
    Posts
    6
    Rep Power
    0

    Default Re: very slow intervaln communication via checkpoint

    hello
    when we sellect internet icon on destination column on application and url and https inspection

    3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

  18. #58
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    763
    Rep Power
    12

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    hello
    when we sellect internet icon on destination column on application and url and https inspection

    3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.
    Clear now, thanks
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  19. #59
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    1,911
    Rep Power
    10

    Default Re: very slow intervaln communication via checkpoint

    Quote Originally Posted by coskun_ist View Post
    hello
    when we sellect internet icon on destination column on application and url and https inspection

    3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.
    Great, thank you for the clarification.
    --
    My book "Max Power: Check Point Firewall Performance Optimization"
    now available via http://maxpowerfirewalls.com.

Page 3 of 3 FirstFirst 123

Similar Threads

  1. Replies: 5
    Last Post: 2014-06-27, 14:13
  2. ICA and SIC communication
    By Palanivel in forum Intermediate
    Replies: 3
    Last Post: 2013-09-10, 22:41
  3. IP addresses using to communication
    By ppawlo in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 4
    Last Post: 2010-06-10, 10:26
  4. SIC communication fail
    By d31jan in forum Check Point SecurePlatform (SPLAT)
    Replies: 3
    Last Post: 2008-08-03, 15:16
  5. PPTP Communication
    By roadrunner in forum Services (TCP, UDP, ICMP, etc.)
    Replies: 0
    Last Post: 2005-08-14, 12:09

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •