very slow intervaln communication via checkpoint

[ShadowPeak.com]

Looks like you have 4 Firewall Worker cores who are having to compete with SND/IRQ processing for the same cores. Your RX-DRP rate is well below 0.1% on all interfaces which is good, although your eth4 and eth5 interfaces are attempting flow control with the switch which indicates they are getting close to overruns in the NICs although it doesn't look like any have actually occurred. Please post the following:

cat /proc/interrupts
sar -P ALL (wait until after your firewall's busiest period has passed earlier in the day before running this)
fwaccel stats -s
fwaccel stat

When you say performance is unacceptable, during that period is it due to packet loss (doesn't look likely based on what you posted) or excessive packet latency? (more likely) Run a ping through the firewall when performance is degraded to see if it is latency or loss.

One thing that is a bit unusual is the nonzero tx_restart_queue counters, this generally indicates that the TX ring buffer is full which is fairly rare. Most problems I've seen are on the RX side. My guess is that you have very high CPU utilization on your 4 cores and may need to consider unlocking more cores with a larger license; the sar output will show us that. SecureXL tuning may help get that high CPU utilization down.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Opera]

[Expert@gate02:0]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 2876448406 0 0 0 IO-APIC-edge timer
1: 3 0 0 0 IO-APIC-edge i8042
4: 502 0 0 0 IO-APIC-edge serial
8: 3 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-level acpi
12: 4 0 0 0 IO-APIC-edge i8042
51: 11390230 451373469 144753102 45509666 PCI-MSI-X eth3
59: 261403272 12366201 15013100 182359 PCI-MSI-X eth1
122: 649 0 0 0 IO-APIC-level ehci_hcd:usb2
130: 66 0 0 0 IO-APIC-level ehci_hcd:usb3
146: 80851132 0 0 0 PCI-MSI-X cciss0
185: 0 0 0 0 IO-APIC-level uhci_hcd:usb1
202: 11834048 568954070 1295149294 2901189190 PCI-MSI-X eth4-TxRx-0
210: 1 0 0 0 PCI-MSI-X eth4
226: 3558238 344913786 1043530074 1527649647 PCI-MSI-X eth5-TxRx-0
234: 1 0 0 0 PCI-MSI-X eth5
NMI: 645291 724210 644017 694212
LOC: 2876329589 2876329706 2876329670 2876326904
ERR: 0
MIS: 0

SAR Command:


07:40:01 CPU %user %nice %system %iowait %steal %idle
07:50:01 all 1.97 0.00 16.85 0.12 0.00 81.07
07:50:01 0 1.56 0.00 15.49 0.34 0.00 82.61
07:50:01 1 1.95 0.00 11.07 0.08 0.00 86.90
07:50:01 2 1.91 0.00 20.57 0.02 0.00 77.51
07:50:01 3 2.45 0.00 20.27 0.04 0.00 77.24
08:00:01 all 2.08 0.00 18.88 0.14 0.00 78.91
08:00:01 0 2.14 0.00 14.02 0.36 0.00 83.47
08:00:01 1 2.25 0.00 13.84 0.04 0.00 83.86
08:00:01 2 2.02 0.00 22.38 0.03 0.00 75.56
08:00:01 3 1.89 0.00 25.25 0.12 0.00 72.74
08:10:01 all 2.33 0.00 22.65 0.02 0.00 75.00
08:10:01 0 2.26 0.00 20.04 0.00 0.00 77.70
08:10:01 1 1.92 0.00 20.13 0.03 0.00 77.92
08:10:01 2 2.71 0.00 22.00 0.02 0.00 75.27
08:10:01 3 2.41 0.00 28.45 0.03 0.00 69.11
08:20:01 all 2.75 0.00 22.10 0.15 0.00 75.00
08:20:01 0 2.82 0.00 16.95 0.25 0.00 79.98
08:20:01 1 2.18 0.00 20.50 0.07 0.00 77.25
08:20:01 2 3.31 0.00 23.87 0.08 0.00 72.74
08:20:01 3 2.70 0.00 27.08 0.19 0.00 70.03
08:30:01 all 2.07 0.00 24.81 0.08 0.00 73.04
08:30:01 0 2.29 0.00 18.28 0.24 0.00 79.19
08:30:01 1 1.95 0.00 21.31 0.05 0.00 76.69
08:30:01 2 2.38 0.00 25.19 0.03 0.00 72.39
08:30:01 3 1.65 0.00 34.44 0.02 0.00 63.89
08:40:01 all 2.19 0.00 24.90 0.01 0.00 72.90
08:40:01 0 2.21 0.00 19.38 0.00 0.00 78.41
08:40:01 1 1.61 0.00 23.75 0.01 0.00 74.62
08:40:01 2 1.88 0.00 32.95 0.00 0.00 65.17
08:40:01 3 3.08 0.00 23.50 0.03 0.00 73.39
08:50:05 all 4.94 0.09 31.07 0.05 0.00 63.86
08:50:05 0 3.83 0.03 28.55 0.03 0.00 67.56
08:50:05 1 6.03 0.06 23.87 0.13 0.00 69.92
08:50:05 2 4.69 0.26 33.76 0.02 0.00 61.28
08:50:05 3 5.22 0.01 38.09 0.01 0.00 56.67
09:00:11 all 2.43 0.00 33.51 0.07 0.00 63.99
09:00:11 0 2.69 0.00 26.66 0.16 0.00 70.49
09:00:11 1 2.22 0.00 34.40 0.05 0.00 63.33
09:00:11 2 2.88 0.00 29.36 0.06 0.00 67.69
09:00:11 3 1.91 0.00 43.63 0.00 0.00 54.46
09:10:01 all 2.29 0.00 36.34 0.03 0.00 61.35
09:10:01 0 1.77 0.00 36.13 0.00 0.00 62.10
09:10:01 1 2.93 0.00 31.14 0.05 0.00 65.89
09:10:01 2 3.32 0.00 31.30 0.02 0.00 65.35
09:10:01 3 1.15 0.00 46.78 0.03 0.00 52.03
09:20:02 all 2.79 0.00 34.51 0.12 0.00 62.58
09:20:02 0 2.30 0.00 30.45 0.21 0.00 67.04
09:20:02 1 3.42 0.00 35.32 0.15 0.00 61.10
09:20:02 2 3.95 0.00 30.09 0.07 0.00 65.89
09:20:02 3 1.48 0.00 42.19 0.03 0.00 56.30
09:30:01 all 2.24 0.00 38.65 0.08 0.00 59.03
09:30:01 0 2.64 0.00 25.00 0.16 0.00 72.20
09:30:01 1 1.88 0.00 41.83 0.10 0.00 56.18
09:30:01 2 2.88 0.00 34.72 0.03 0.00 62.38
09:30:01 3 1.54 0.00 53.04 0.04 0.00 45.38
09:40:01 all 2.32 0.00 34.78 0.01 0.00 62.89
09:40:01 0 2.11 0.00 27.64 0.00 0.00 70.24
09:40:01 1 2.05 0.00 37.41 0.01 0.00 60.53
09:40:01 2 3.08 0.00 33.66 0.04 0.00 63.22
09:40:01 3 2.04 0.00 40.40 0.00 0.00 57.55
09:50:01 all 2.35 0.00 32.43 0.06 0.00 65.15
09:50:01 0 2.31 0.00 23.63 0.18 0.00 73.88
09:50:01 1 1.76 0.00 40.66 0.00 0.00 57.58
09:50:01 2 3.35 0.00 33.21 0.03 0.00 63.41
09:50:01 3 1.99 0.00 32.24 0.03 0.00 65.74
10:00:01 all 2.33 0.00 36.60 0.07 0.00 61.01
10:00:01 0 2.37 0.00 27.65 0.16 0.00 69.82
10:00:01 1 2.13 0.00 45.61 0.04 0.00 52.22
10:00:01 2 3.18 0.00 36.31 0.02 0.00 60.49
10:00:01 3 1.62 0.00 36.82 0.05 0.00 61.50
10:10:01 all 2.49 0.00 34.98 0.01 0.00 62.51
10:10:01 0 2.60 0.00 27.78 0.00 0.00 69.62
10:10:01 1 1.84 0.00 39.50 0.02 0.00 58.64
10:10:01 2 3.35 0.00 33.75 0.02 0.00 62.88
10:10:01 3 2.17 0.00 38.90 0.01 0.00 58.92
10:20:01 all 2.25 0.00 32.47 0.12 0.00 65.16
10:20:01 0 1.91 0.00 30.45 0.27 0.00 67.37
10:20:01 1 2.39 0.00 27.66 0.09 0.00 69.87
10:20:01 2 2.86 0.00 31.11 0.11 0.00 65.92
10:20:01 3 1.85 0.00 40.65 0.03 0.00 57.47
10:30:01 all 2.16 0.00 34.33 0.40 0.00 63.11
10:30:01 0 1.84 0.00 29.60 0.25 0.00 68.31
10:30:01 1 1.91 0.00 38.71 0.41 0.00 58.97
10:30:01 2 3.32 0.00 30.20 0.49 0.00 65.99
10:30:01 3 1.55 0.00 38.82 0.46 0.00 59.17
10:40:01 all 2.13 0.00 35.85 0.01 0.00 62.02
10:40:01 0 2.11 0.00 28.81 0.00 0.00 69.08
10:40:01 1 2.05 0.00 47.53 0.01 0.00 50.40
10:40:01 2 3.17 0.00 28.41 0.01 0.00 68.41
10:40:01 3 1.18 0.00 38.64 0.01 0.00 60.18
10:50:01 all 2.19 0.00 37.03 0.07 0.00 60.72
10:50:01 0 2.35 0.00 28.11 0.19 0.00 69.35
10:50:01 1 2.58 0.00 30.74 0.04 0.00 66.64
10:50:01 2 2.10 0.00 44.98 0.00 0.00 52.91
10:50:01 3 1.72 0.00 44.29 0.03 0.00 53.96
11:00:01 all 2.08 0.00 30.96 0.10 0.00 66.86
11:00:01 0 2.28 0.00 21.01 0.28 0.00 76.42
11:00:01 1 1.74 0.00 27.10 0.10 0.00 71.07
11:00:01 2 2.99 0.00 28.46 0.02 0.00 68.54
11:00:01 3 1.29 0.00 47.29 0.01 0.00 51.41
11:10:01 all 2.03 0.00 25.21 0.01 0.00 72.75
11:10:01 0 1.71 0.00 24.16 0.00 0.00 74.13
11:10:01 1 1.80 0.00 19.12 0.01 0.00 79.07
11:10:01 2 2.34 0.00 25.46 0.03 0.00 72.16
11:10:01 3 2.26 0.00 32.09 0.01 0.00 65.64
11:20:01 all 2.13 0.00 25.55 0.10 0.00 72.22
11:20:01 0 1.99 0.00 23.45 0.29 0.00 74.26
11:20:01 1 1.94 0.00 21.18 0.03 0.00 76.85
11:20:01 2 2.04 0.00 31.44 0.04 0.00 66.48
11:20:01 3 2.53 0.00 26.15 0.02 0.00 71.29
11:30:01 all 2.02 0.00 21.11 0.09 0.00 76.78
11:30:01 0 2.00 0.00 16.35 0.30 0.00 81.34
11:30:01 1 1.30 0.00 24.02 0.03 0.00 74.64
11:30:01 2 2.32 0.00 22.82 0.03 0.00 74.83
11:30:01 3 2.46 0.00 21.24 0.01 0.00 76.29

11:30:01 CPU %user %nice %system %iowait %steal %idle
11:40:01 all 2.03 0.00 22.19 0.01 0.00 75.77
11:40:01 0 1.76 0.00 23.37 0.00 0.00 74.86
11:40:01 1 1.84 0.00 18.97 0.03 0.00 79.16
11:40:01 2 2.38 0.00 21.73 0.03 0.00 75.87
11:40:01 3 2.12 0.00 24.71 0.00 0.00 73.18
11:50:01 all 4.52 0.10 24.54 0.14 0.00 70.70
11:50:01 0 2.23 0.02 30.37 0.24 0.00 67.14
11:50:01 1 3.79 0.10 20.06 0.17 0.00 75.88
11:50:01 2 5.13 0.09 23.37 0.04 0.00 71.36
11:50:01 3 6.94 0.18 24.33 0.13 0.00 68.41
12:00:01 all 2.03 0.00 23.32 0.12 0.00 74.53
12:00:01 0 2.21 0.00 17.86 0.28 0.00 79.65
12:00:01 1 1.33 0.00 22.82 0.07 0.00 75.78
12:00:01 2 2.55 0.00 24.62 0.06 0.00 72.78
12:00:01 3 2.01 0.00 27.98 0.07 0.00 69.93
12:10:01 all 2.65 0.00 26.93 0.03 0.00 70.39
12:10:01 0 2.13 0.00 31.23 0.00 0.00 66.64
12:10:01 1 2.43 0.00 20.13 0.04 0.00 77.40
12:10:01 2 3.01 0.00 26.90 0.03 0.00 70.06
12:10:01 3 3.04 0.00 29.46 0.03 0.00 67.46
12:20:01 all 2.86 0.00 25.19 0.11 0.00 71.84
12:20:01 0 2.96 0.00 22.96 0.33 0.00 73.74
12:20:01 1 1.99 0.00 22.23 0.03 0.00 75.75
12:20:01 2 3.67 0.00 24.20 0.05 0.00 72.08
12:20:01 3 2.80 0.00 31.38 0.02 0.00 65.80
12:30:01 all 2.38 0.00 26.45 0.13 0.00 71.04
12:30:01 0 2.59 0.00 20.15 0.30 0.00 76.96
12:30:01 1 1.65 0.00 26.77 0.02 0.00 71.56
12:30:01 2 2.91 0.00 26.71 0.18 0.00 70.20
12:30:01 3 2.39 0.00 32.16 0.01 0.00 65.44
12:40:01 all 2.32 0.00 28.57 0.06 0.00 69.06
12:40:01 0 1.97 0.00 28.84 0.00 0.00 69.20
12:40:01 1 1.74 0.00 26.75 0.13 0.00 71.37
12:40:01 2 2.78 0.00 27.28 0.06 0.00 69.87
12:40:01 3 2.78 0.00 31.40 0.03 0.00 65.79
12:50:01 all 2.24 0.00 30.34 0.12 0.00 67.29
12:50:01 0 2.08 0.00 28.76 0.20 0.00 68.96
12:50:01 1 1.32 0.00 31.39 0.05 0.00 67.23
12:50:01 2 2.73 0.00 30.42 0.09 0.00 66.76
12:50:01 3 2.85 0.00 30.81 0.13 0.00 66.20
Average: all 2.05 0.00 20.67 0.10 0.00 77.17
Average: 0 2.16 0.00 15.17 0.21 0.00 82.46
Average: 1 1.43 0.00 25.58 0.05 0.00 72.93
Average: 2 2.15 0.01 20.57 0.07 0.00 77.21
Average: 3 2.47 0.01 21.38 0.07 0.00 76.08


[Expert@gate02:0]# fwaccel stats -s
Accelerated conns/Total conns : 45/10203 (0%)
Accelerated pkts/Total pkts : 30463/33121932 (0%)
F2Fed pkts/Total pkts : 790706/33121932 (2%)
PXL pkts/Total pkts : 32300763/33121932 (97%)
QXL pkts/Total pkts : 0/33121932 (0%)

[Expert@gate02:0]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #243
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, ViolationStats,
Nac, AsychronicNotif, ERDOS, McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

Bad performance in my case means that when i copy large files between the servers in different vlans i get only 10-20MB/s which is very slow as i think.

[ShadowPeak.com]

Well it definitely does not appear to be a firewall CPU bottleneck and automatic affinity is doing a reasonable job of spreading around IRQ processing which tracks with a low RX-DRP rate. The fact that very few connections are able to be templated by SecureXL is a little strange, but tuning in that area just helps save CPU on the Firewall Worker cores which are not overloaded to begin with. This assumes of course that the provided sar output is showing the firewall's busiest period. Almost all your traffic is being processed in the Medium Path (PXL) which is not that unusual and it looks like the Firewall Worker cores are able to keep up with the load. Try this:

• Start a continuous ping across the firewall for a few minutes and note the average latency numbers and any loss.
• Now start one of your slow file transfers. What happens to the latency and/or loss on your continuous ping? We still need to figure out if it is latency or loss that is slowing you down.
• While the transfer is running, run top on the firewall and then hit a numeric "1". Any individual CPUs below 25% idle? My guess is no.

During a maintenance window, you could try disabling features one by one that force use of the Medium Path on the firewall (based on your previous posts it looks like these would be App control, IPS detect mode, URL filtering for your site) and see if performance dramatically improves right after one of them is turned off. Frankly I'm not sure even turning them all off will make much of a difference. If it does you could have a firewall topology issue that is forcing traffic between internal networks to get processed by APCL/URLF which is a big no-no. Could definitely be something in IPS as well even if it is in detect mode.

If you see sudden spikes of latency or loss in your ping when a slow transfer is running, but can't seem to correlate it to observable issues with the firewall, that may indicate an issue elsewhere in the network path. These can be notoriously difficult to find and will highly depend on the architecture of your network. The first few chapters of my book take you through discovering the path through your network and checking any network devices external to the firewall for the following five main things:

• Excessive CPU Load
• Low Free Memory
• Network Interface Errors
• Network Buffer Processing Errors
• Suspiciously Short Device Uptime

[jonne]

[Expert@gate02:0]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 2876448406 0 0 0 IO-APIC-edge timer
1: 3 0 0 0 IO-APIC-edge i8042
4: 502 0 0 0 IO-APIC-edge serial
8: 3 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-level acpi
12: 4 0 0 0 IO-APIC-edge i8042
51: 11390230 451373469 144753102 45509666 PCI-MSI-X eth3
59: 261403272 12366201 15013100 182359 PCI-MSI-X eth1
122: 649 0 0 0 IO-APIC-level ehci_hcd:usb2
130: 66 0 0 0 IO-APIC-level ehci_hcd:usb3
146: 80851132 0 0 0 PCI-MSI-X cciss0
185: 0 0 0 0 IO-APIC-level uhci_hcd:usb1
202: 11834048 568954070 1295149294 2901189190 PCI-MSI-X eth4-TxRx-0
210: 1 0 0 0 PCI-MSI-X eth4
226: 3558238 344913786 1043530074 1527649647 PCI-MSI-X eth5-TxRx-0
234: 1 0 0 0 PCI-MSI-X eth5
NMI: 645291 724210 644017 694212
LOC: 2876329589 2876329706 2876329670 2876326904
ERR: 0
MIS: 0

SAR Command:


07:40:01 CPU %user %nice %system %iowait %steal %idle
07:50:01 all 1.97 0.00 16.85 0.12 0.00 81.07
07:50:01 0 1.56 0.00 15.49 0.34 0.00 82.61
07:50:01 1 1.95 0.00 11.07 0.08 0.00 86.90
07:50:01 2 1.91 0.00 20.57 0.02 0.00 77.51
07:50:01 3 2.45 0.00 20.27 0.04 0.00 77.24
08:00:01 all 2.08 0.00 18.88 0.14 0.00 78.91
08:00:01 0 2.14 0.00 14.02 0.36 0.00 83.47
08:00:01 1 2.25 0.00 13.84 0.04 0.00 83.86
08:00:01 2 2.02 0.00 22.38 0.03 0.00 75.56
08:00:01 3 1.89 0.00 25.25 0.12 0.00 72.74
08:10:01 all 2.33 0.00 22.65 0.02 0.00 75.00
08:10:01 0 2.26 0.00 20.04 0.00 0.00 77.70
08:10:01 1 1.92 0.00 20.13 0.03 0.00 77.92
08:10:01 2 2.71 0.00 22.00 0.02 0.00 75.27
08:10:01 3 2.41 0.00 28.45 0.03 0.00 69.11
08:20:01 all 2.75 0.00 22.10 0.15 0.00 75.00
08:20:01 0 2.82 0.00 16.95 0.25 0.00 79.98
08:20:01 1 2.18 0.00 20.50 0.07 0.00 77.25
08:20:01 2 3.31 0.00 23.87 0.08 0.00 72.74
08:20:01 3 2.70 0.00 27.08 0.19 0.00 70.03
08:30:01 all 2.07 0.00 24.81 0.08 0.00 73.04
08:30:01 0 2.29 0.00 18.28 0.24 0.00 79.19
08:30:01 1 1.95 0.00 21.31 0.05 0.00 76.69
08:30:01 2 2.38 0.00 25.19 0.03 0.00 72.39
08:30:01 3 1.65 0.00 34.44 0.02 0.00 63.89
08:40:01 all 2.19 0.00 24.90 0.01 0.00 72.90
08:40:01 0 2.21 0.00 19.38 0.00 0.00 78.41
08:40:01 1 1.61 0.00 23.75 0.01 0.00 74.62
08:40:01 2 1.88 0.00 32.95 0.00 0.00 65.17
08:40:01 3 3.08 0.00 23.50 0.03 0.00 73.39
08:50:05 all 4.94 0.09 31.07 0.05 0.00 63.86
08:50:05 0 3.83 0.03 28.55 0.03 0.00 67.56
08:50:05 1 6.03 0.06 23.87 0.13 0.00 69.92
08:50:05 2 4.69 0.26 33.76 0.02 0.00 61.28
08:50:05 3 5.22 0.01 38.09 0.01 0.00 56.67
09:00:11 all 2.43 0.00 33.51 0.07 0.00 63.99
09:00:11 0 2.69 0.00 26.66 0.16 0.00 70.49
09:00:11 1 2.22 0.00 34.40 0.05 0.00 63.33
09:00:11 2 2.88 0.00 29.36 0.06 0.00 67.69
09:00:11 3 1.91 0.00 43.63 0.00 0.00 54.46
09:10:01 all 2.29 0.00 36.34 0.03 0.00 61.35
09:10:01 0 1.77 0.00 36.13 0.00 0.00 62.10
09:10:01 1 2.93 0.00 31.14 0.05 0.00 65.89
09:10:01 2 3.32 0.00 31.30 0.02 0.00 65.35
09:10:01 3 1.15 0.00 46.78 0.03 0.00 52.03
09:20:02 all 2.79 0.00 34.51 0.12 0.00 62.58
09:20:02 0 2.30 0.00 30.45 0.21 0.00 67.04
09:20:02 1 3.42 0.00 35.32 0.15 0.00 61.10
09:20:02 2 3.95 0.00 30.09 0.07 0.00 65.89
09:20:02 3 1.48 0.00 42.19 0.03 0.00 56.30
09:30:01 all 2.24 0.00 38.65 0.08 0.00 59.03
09:30:01 0 2.64 0.00 25.00 0.16 0.00 72.20
09:30:01 1 1.88 0.00 41.83 0.10 0.00 56.18
09:30:01 2 2.88 0.00 34.72 0.03 0.00 62.38
09:30:01 3 1.54 0.00 53.04 0.04 0.00 45.38
09:40:01 all 2.32 0.00 34.78 0.01 0.00 62.89
09:40:01 0 2.11 0.00 27.64 0.00 0.00 70.24
09:40:01 1 2.05 0.00 37.41 0.01 0.00 60.53
09:40:01 2 3.08 0.00 33.66 0.04 0.00 63.22
09:40:01 3 2.04 0.00 40.40 0.00 0.00 57.55
09:50:01 all 2.35 0.00 32.43 0.06 0.00 65.15
09:50:01 0 2.31 0.00 23.63 0.18 0.00 73.88
09:50:01 1 1.76 0.00 40.66 0.00 0.00 57.58
09:50:01 2 3.35 0.00 33.21 0.03 0.00 63.41
09:50:01 3 1.99 0.00 32.24 0.03 0.00 65.74
10:00:01 all 2.33 0.00 36.60 0.07 0.00 61.01
10:00:01 0 2.37 0.00 27.65 0.16 0.00 69.82
10:00:01 1 2.13 0.00 45.61 0.04 0.00 52.22
10:00:01 2 3.18 0.00 36.31 0.02 0.00 60.49
10:00:01 3 1.62 0.00 36.82 0.05 0.00 61.50
10:10:01 all 2.49 0.00 34.98 0.01 0.00 62.51
10:10:01 0 2.60 0.00 27.78 0.00 0.00 69.62
10:10:01 1 1.84 0.00 39.50 0.02 0.00 58.64
10:10:01 2 3.35 0.00 33.75 0.02 0.00 62.88
10:10:01 3 2.17 0.00 38.90 0.01 0.00 58.92
10:20:01 all 2.25 0.00 32.47 0.12 0.00 65.16
10:20:01 0 1.91 0.00 30.45 0.27 0.00 67.37
10:20:01 1 2.39 0.00 27.66 0.09 0.00 69.87
10:20:01 2 2.86 0.00 31.11 0.11 0.00 65.92
10:20:01 3 1.85 0.00 40.65 0.03 0.00 57.47
10:30:01 all 2.16 0.00 34.33 0.40 0.00 63.11
10:30:01 0 1.84 0.00 29.60 0.25 0.00 68.31
10:30:01 1 1.91 0.00 38.71 0.41 0.00 58.97
10:30:01 2 3.32 0.00 30.20 0.49 0.00 65.99
10:30:01 3 1.55 0.00 38.82 0.46 0.00 59.17
10:40:01 all 2.13 0.00 35.85 0.01 0.00 62.02
10:40:01 0 2.11 0.00 28.81 0.00 0.00 69.08
10:40:01 1 2.05 0.00 47.53 0.01 0.00 50.40
10:40:01 2 3.17 0.00 28.41 0.01 0.00 68.41
10:40:01 3 1.18 0.00 38.64 0.01 0.00 60.18
10:50:01 all 2.19 0.00 37.03 0.07 0.00 60.72
10:50:01 0 2.35 0.00 28.11 0.19 0.00 69.35
10:50:01 1 2.58 0.00 30.74 0.04 0.00 66.64
10:50:01 2 2.10 0.00 44.98 0.00 0.00 52.91
10:50:01 3 1.72 0.00 44.29 0.03 0.00 53.96
11:00:01 all 2.08 0.00 30.96 0.10 0.00 66.86
11:00:01 0 2.28 0.00 21.01 0.28 0.00 76.42
11:00:01 1 1.74 0.00 27.10 0.10 0.00 71.07
11:00:01 2 2.99 0.00 28.46 0.02 0.00 68.54
11:00:01 3 1.29 0.00 47.29 0.01 0.00 51.41
11:10:01 all 2.03 0.00 25.21 0.01 0.00 72.75
11:10:01 0 1.71 0.00 24.16 0.00 0.00 74.13
11:10:01 1 1.80 0.00 19.12 0.01 0.00 79.07
11:10:01 2 2.34 0.00 25.46 0.03 0.00 72.16
11:10:01 3 2.26 0.00 32.09 0.01 0.00 65.64
11:20:01 all 2.13 0.00 25.55 0.10 0.00 72.22
11:20:01 0 1.99 0.00 23.45 0.29 0.00 74.26
11:20:01 1 1.94 0.00 21.18 0.03 0.00 76.85
11:20:01 2 2.04 0.00 31.44 0.04 0.00 66.48
11:20:01 3 2.53 0.00 26.15 0.02 0.00 71.29
11:30:01 all 2.02 0.00 21.11 0.09 0.00 76.78
11:30:01 0 2.00 0.00 16.35 0.30 0.00 81.34
11:30:01 1 1.30 0.00 24.02 0.03 0.00 74.64
11:30:01 2 2.32 0.00 22.82 0.03 0.00 74.83
11:30:01 3 2.46 0.00 21.24 0.01 0.00 76.29

11:30:01 CPU %user %nice %system %iowait %steal %idle
11:40:01 all 2.03 0.00 22.19 0.01 0.00 75.77
11:40:01 0 1.76 0.00 23.37 0.00 0.00 74.86
11:40:01 1 1.84 0.00 18.97 0.03 0.00 79.16
11:40:01 2 2.38 0.00 21.73 0.03 0.00 75.87
11:40:01 3 2.12 0.00 24.71 0.00 0.00 73.18
11:50:01 all 4.52 0.10 24.54 0.14 0.00 70.70
11:50:01 0 2.23 0.02 30.37 0.24 0.00 67.14
11:50:01 1 3.79 0.10 20.06 0.17 0.00 75.88
11:50:01 2 5.13 0.09 23.37 0.04 0.00 71.36
11:50:01 3 6.94 0.18 24.33 0.13 0.00 68.41
12:00:01 all 2.03 0.00 23.32 0.12 0.00 74.53
12:00:01 0 2.21 0.00 17.86 0.28 0.00 79.65
12:00:01 1 1.33 0.00 22.82 0.07 0.00 75.78
12:00:01 2 2.55 0.00 24.62 0.06 0.00 72.78
12:00:01 3 2.01 0.00 27.98 0.07 0.00 69.93
12:10:01 all 2.65 0.00 26.93 0.03 0.00 70.39
12:10:01 0 2.13 0.00 31.23 0.00 0.00 66.64
12:10:01 1 2.43 0.00 20.13 0.04 0.00 77.40
12:10:01 2 3.01 0.00 26.90 0.03 0.00 70.06
12:10:01 3 3.04 0.00 29.46 0.03 0.00 67.46
12:20:01 all 2.86 0.00 25.19 0.11 0.00 71.84
12:20:01 0 2.96 0.00 22.96 0.33 0.00 73.74
12:20:01 1 1.99 0.00 22.23 0.03 0.00 75.75
12:20:01 2 3.67 0.00 24.20 0.05 0.00 72.08
12:20:01 3 2.80 0.00 31.38 0.02 0.00 65.80
12:30:01 all 2.38 0.00 26.45 0.13 0.00 71.04
12:30:01 0 2.59 0.00 20.15 0.30 0.00 76.96
12:30:01 1 1.65 0.00 26.77 0.02 0.00 71.56
12:30:01 2 2.91 0.00 26.71 0.18 0.00 70.20
12:30:01 3 2.39 0.00 32.16 0.01 0.00 65.44
12:40:01 all 2.32 0.00 28.57 0.06 0.00 69.06
12:40:01 0 1.97 0.00 28.84 0.00 0.00 69.20
12:40:01 1 1.74 0.00 26.75 0.13 0.00 71.37
12:40:01 2 2.78 0.00 27.28 0.06 0.00 69.87
12:40:01 3 2.78 0.00 31.40 0.03 0.00 65.79
12:50:01 all 2.24 0.00 30.34 0.12 0.00 67.29
12:50:01 0 2.08 0.00 28.76 0.20 0.00 68.96
12:50:01 1 1.32 0.00 31.39 0.05 0.00 67.23
12:50:01 2 2.73 0.00 30.42 0.09 0.00 66.76
12:50:01 3 2.85 0.00 30.81 0.13 0.00 66.20
Average: all 2.05 0.00 20.67 0.10 0.00 77.17
Average: 0 2.16 0.00 15.17 0.21 0.00 82.46
Average: 1 1.43 0.00 25.58 0.05 0.00 72.93
Average: 2 2.15 0.01 20.57 0.07 0.00 77.21
Average: 3 2.47 0.01 21.38 0.07 0.00 76.08


[Expert@gate02:0]# fwaccel stats -s
Accelerated conns/Total conns : 45/10203 (0%)
Accelerated pkts/Total pkts : 30463/33121932 (0%)
F2Fed pkts/Total pkts : 790706/33121932 (2%)
PXL pkts/Total pkts : 32300763/33121932 (97%)
QXL pkts/Total pkts : 0/33121932 (0%)

[Expert@gate02:0]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #243
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, ViolationStats,
Nac, AsychronicNotif, ERDOS, McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

Bad performance in my case means that when i copy large files between the servers in different vlans i get only 10-20MB/s which is very slow as i think.

Hi Opera,

I am troubleshooting a performance issue that appears to be very similar to yours. On the R77.10 VSX gateway, I am only averaging 10-40MB/s copying large files of approx. 100M between two VLANs connected to the same VS. I get the occasional fast transfer (90-100MB/s). Did you ever find a solution or improve performance?

Thanks,


Jonne.

[coskun_ist]

hello
i have same issue with my network
2 gw are working on network cluster xl with cisco nexus (vpc )
i see 400 500 mb/ps when i test by ipref tool on local network but test result is 20 30 mb/ps between local and dmz network

all gw have 10 gb inteface card

how did you solve this problem

[ShadowPeak.com]

hello
i have same issue with my network
2 gw are working on network cluster xl with cisco nexus (vpc )
i see 400 500 mb/ps when i test by ipref tool on local network but test result is 20 30 mb/ps between local and dmz network

all gw have 10 gb inteface card

how did you solve this problem

What is the hardware type of the firewalls?

Please provide output of the following "Super Seven" performance assessment commands run on the active firewall member during a long slow transfer:

fwaccel stat
fwaccel stats -s
grep -c ^processor /proc/cpuinfo
fw ctl affinity -l -r
netstat -ni
fw ctl multik stat
cpstat os -f multi_cpu -o 1

[coskun_ist]

Hello

command resul is below

[Expert@CPGW02:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@CPGW02:0]# fwaccel stats -s
Accelerated conns/Total conns : 892/22374 (3%)
Accelerated pkts/Total pkts : 406144/3996782 (10%)
F2Fed pkts/Total pkts : 986669/3996782 (24%)
PXL pkts/Total pkts : 2603969/3996782 (65%)
QXL pkts/Total pkts : 0/3996782 (0%)
[Expert@CPGW02:0]# grep -c ^processor /proc/cpuinfo
8
[Expert@CPGW02:0]# fw ctl affinity -l -r
CPU 0: eth8 eth0 eth1 eth2
CPU 1: eth4 eth5 eth11 eth3
CPU 2: fw_5
CPU 3: fw_4
CPU 4: fw_3
CPU 5: fw_2
CPU 6: fw_1
CPU 7: fw_0
All: in.msd vpnd pdpd usrchkd in.acapd pepd rad fwd mpdaemon fwpushd rtmd cpd cprid
[Expert@CPGW02:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond1 1500 0 979512121 0 2378 0 1297527898 0 0 0 BMmRU
bond1.20 1500 0 931034529 0 0 0 1297547505 0 0 0 BMmRU
bond2 1500 0 346477067 0 82 0 306145365 0 0 0 BMmRU
bond2.34 1500 0 345027255 0 0 0 302250627 0 0 0 BMmRU
bond2.106 1500 0 521579 0 0 0 1330266 0 0 0 BMmRU
bond2.107 1500 0 496730 0 0 0 2542306 0 0 0 BMmRU
eth0 1500 0 190769070 0 82 0 145162388 0 0 0 BMsRU
eth1 1500 0 967930979 0 2378 0 843804189 0 0 0 BMsRU
eth2 1500 0 155708002 0 0 0 160982985 0 0 0 BMsRU
eth3 1500 0 11581246 0 0 0 453723874 0 0 0 BMsRU
eth4 1500 0 27844208 0 0 0 19852352 0 0 0 BMRU
eth5 1500 0 980368184 0 302432 302432 774786946 0 0 0 BMRU
eth8 1500 0 297716 0 0 0 4886 0 0 0 BMRU
eth11 1500 0 20419612 0 165 165 78261723 0 0 0 BMRU
lo 16436 0 26736535 0 0 0 26736535 0 0 0 LRU
[Expert@CPGW02:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 7 | 5157 | 7597
1 | Yes | 6 | 4872 | 5966
2 | Yes | 5 | 4848 | 7426
3 | Yes | 4 | 4665 | 6166
4 | Yes | 3 | 5137 | 8813
5 | Yes | 2 | 6313 | 7535
[Expert@CPGW02:0]# cpstat os -f multi_cpu -o 1



Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 6| 92| 8| ?| 0|
| 2| 2| 6| 91| 9| ?| 0|
| 3| 1| 12| 87| 13| ?| 0|
| 4| 1| 13| 86| 14| ?| 0|
| 5| 1| 13| 87| 13| ?| 0|
| 6| 1| 13| 86| 14| ?| 0|
| 7| 1| 14| 85| 15| ?| 0|
| 8| 1| 14| 84| 16| ?| 0|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 6| 92| 8| ?| 0|
| 2| 2| 6| 91| 9| ?| 0|
| 3| 1| 12| 87| 13| ?| 0|
| 4| 1| 13| 86| 14| ?| 0|
| 5| 1| 13| 87| 13| ?| 0|
| 6| 1| 13| 86| 14| ?| 0|
| 7| 1| 14| 85| 15| ?| 0|
| 8| 1| 14| 84| 16| ?| 0|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 12| 19| 70| 30| ?| 28665|
| 2| 11| 28| 62| 38| ?| 28667|
| 3| 2| 62| 36| 64| ?| 28669|
| 4| 1| 71| 28| 72| ?| 28672|
| 5| 5| 38| 58| 42| ?| 28675|
| 6| 3| 70| 27| 73| ?| 28676|
| 7| 5| 54| 41| 59| ?| 28677|
| 8| 6| 52| 42| 58| ?| 57360|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 12| 19| 70| 30| ?| 28665|
| 2| 11| 28| 62| 38| ?| 28667|
| 3| 2| 62| 36| 64| ?| 28669|
| 4| 1| 71| 28| 72| ?| 28672|
| 5| 5| 38| 58| 42| ?| 28675|
| 6| 3| 70| 27| 73| ?| 28676|
| 7| 5| 54| 41| 59| ?| 28677|
| 8| 6| 52| 42| 58| ?| 57360|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 8| 16| 76| 24| ?| 35472|
| 2| 8| 17| 75| 25| ?| 35473|
| 3| 4| 49| 47| 53| ?| 35475|
| 4| 2| 86| 13| 87| ?| 35478|
| 5| 4| 62| 34| 66| ?| 35479|
| 6| 2| 77| 21| 79| ?| 70965|
| 7| 5| 66| 30| 70| ?| 35486|
| 8| 4| 64| 32| 68| ?| 35490|
---------------------------------------------------------------------------------



[Expert@CPGW02:0]#

[coskun_ist]

Hello

all gw open server

[Expert@CPGW02:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@CPGW02:0]# fwaccel stats -s
Accelerated conns/Total conns : 224/28099 (0%)
Accelerated pkts/Total pkts : 515038/183162461 (0%)
F2Fed pkts/Total pkts : 50622616/183162461 (27%)
PXL pkts/Total pkts : 132024807/183162461 (72%)
QXL pkts/Total pkts : 0/183162461 (0%)
[Expert@CPGW02:0]# grep -c ^processor /proc/cpuinfo
8
[Expert@CPGW02:0]# fw ctl affinity -l -r
CPU 0: eth8 eth1 eth2 eth3
CPU 1: eth4 eth5 eth11 eth0
CPU 2: fw_5
CPU 3: fw_4
CPU 4: fw_3
CPU 5: fw_2
CPU 6: fw_1
CPU 7: fw_0
All: vpnd pdpd usrchkd pepd rad fwd mpdaemon fwpushd rtmd cpd cprid
[Expert@CPGW02:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond1 1500 0 1177769705 0 5571 0 1595064431 0 0 0 BMmRU
bond1.20 1500 0 1123219949 0 0 0 1595074770 0 0 0 BMmRU
bond2 1500 0 385861325 0 82 0 338242808 0 0 0 BMmRU
bond2.34 1500 0 384249023 0 0 0 333929993 0 0 0 BMmRU
bond2.106 1500 0 611273 0 0 0 1548826 0 0 0 BMmRU
bond2.107 1500 0 535440 0 0 0 2739597 0 0 0 BMmRU
eth0 1500 0 217613515 0 82 0 159256106 0 0 0 BMsRU
eth1 1500 0 1164444468 0 5571 0 1055916459 0 0 0 BMsRU
eth2 1500 0 168247819 0 0 0 178986713 0 0 0 BMsRU
eth3 1500 0 13325335 0 0 0 539148111 0 0 0 BMsRU
eth4 1500 0 33919836 0 0 0 24502300 0 0 0 BMRU
eth5 1500 0 1239838999 0 375124 375124 979502286 0 0 0 BMRU
eth8 1500 0 326042 0 0 0 5292 0 0 0 BMRU
eth11 1500 0 22006335 0 212 212 95669018 0 0 0 BMRU
lo 16436 0 34467492 0 0 0 34467492 0 0 0 LRU
[Expert@CPGW02:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 7 | 4382 | 7597
1 | Yes | 6 | 4359 | 5966
2 | Yes | 5 | 4222 | 7426
3 | Yes | 4 | 4078 | 6166
4 | Yes | 3 | 4472 | 8813
5 | Yes | 2 | 5245 | 7535
[Expert@CPGW02:0]# cpstat os -f multi_cpu -o 1



Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 6| 14| 80| 20| ?| 0|
| 2| 6| 16| 78| 22| ?| 0|
| 3| 2| 30| 68| 32| ?| 0|
| 4| 3| 31| 66| 34| ?| 0|
| 5| 2| 31| 67| 33| ?| 0|
| 6| 2| 31| 67| 33| ?| 0|
| 7| 3| 33| 64| 36| ?| 0|
| 8| 3| 35| 62| 38| ?| 0|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 6| 14| 80| 20| ?| 0|
| 2| 6| 16| 78| 22| ?| 0|
| 3| 2| 30| 68| 32| ?| 0|
| 4| 3| 31| 66| 34| ?| 0|
| 5| 2| 31| 67| 33| ?| 0|
| 6| 2| 31| 67| 33| ?| 0|
| 7| 3| 33| 64| 36| ?| 0|
| 8| 3| 35| 62| 38| ?| 0|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 5| 9| 85| 15| ?| 33042|
| 2| 7| 13| 80| 20| ?| 33044|
| 3| 3| 42| 55| 45| ?| 33045|
| 4| 2| 47| 52| 48| ?| 66096|
| 5| 1| 31| 69| 31| ?| 33050|
| 6| 3| 26| 71| 29| ?| 33052|
| 7| 3| 34| 63| 37| ?| 33054|
| 8| 2| 38| 60| 40| ?| 66111|
---------------------------------------------------------------------------------



[Expert@CPGW02:0]#

[ShadowPeak.com]

Everything looks fairly well balanced here, the firewall just looks quite busy. The only possible thing I see here is this:

eth5 1500 0 1239838999 0 375124 375124 979502286 0 0 0 BMRU

1) You are taking overruns & drops on your eth5 interface, by chance is this interface involved with the slow communication? Do the RX-DRP/RX-OVR counters for this interface increment during a slow transfer?

2) Please provide the output of these further commands on the active cluster member:

fw ver
fw ctl multik get_mode
enabled_blades (if that doesn't work try enabled_blades.sh)
free -m
ethtool -i eth5
ethtool -S eth5
ethtool -i eth1
ethtool -S eth1
/sbin/cpuinfo

I hope you are not using Broadcom or Emulex NICs on your open hardware system as their performance will be terrible no matter what you do...

3) Assuming the above all check out, it will be time to look at the surrounding switchports and host NICs involved with the slow communication for errors. Your firewall is actually in much better shape than most as far as optimization.

[cciesec2006]

[Expert@CPGW02:0]# fwaccel stats -s
Accelerated conns/Total conns : 224/28099 (0%)
Accelerated pkts/Total pkts : 515038/183162461 (0%)
F2Fed pkts/Total pkts : 50622616/183162461 (27%)
PXL pkts/Total pkts : 132024807/183162461 (72%)
QXL pkts/Total pkts : 0/183162461 (0%)

The reason your firewall is in terrible shape is because none of the traffics is being accelerated by SecureXL. 72% of the traffic is "partially" accelerated by the firewall. That is something you need to look into. Might be the cause of the slowness.

This is what you should be seeing:

Accelerated conns/Total conns : 7/26 (26%)
Accelerated pkts/Total pkts : 2241723/2249700 (99%)
F2Fed pkts/Total pkts : 7977/2249700 (0%)
PXL pkts/Total pkts : 0/2249700 (0%)
QXL pkts/Total pkts : 0/2249700 (0%)

[ShadowPeak.com]

The reason your firewall is in terrible shape is because none of the traffics is being accelerated by SecureXL. 72% of the traffic is "partially" accelerated by the firewall. That is something you need to look into. Might be the cause of the slowness.

This is what you should be seeing:

Accelerated conns/Total conns : 7/26 (26%)
Accelerated pkts/Total pkts : 2241723/2249700 (99%)
F2Fed pkts/Total pkts : 7977/2249700 (0%)
PXL pkts/Total pkts : 0/2249700 (0%)
QXL pkts/Total pkts : 0/2249700 (0%)

Maybe, depends on which blades are enabled. I have requested that info.

[coskun_ist]

hello
1 ) eth5 is my internet interface. we stilll have slow communication when we shutdown this interface.
2 ) you can find the outputs you asked below

[Expert@CPGW02:0]# fw ver
This is Check Point's software version R77.30 - Build 034

[Expert@CPGW02:0]# fw ctl multik get_mode
Current mode is On

[Expert@CPGW02:0]# enabled_blades
fw vpn cvpn urlf appi ips identityServer SSL_INSPECT

[Expert@CPGW02:0]# free -m
total used free shared buffers cached
Mem: 144811 20289 124521 0 447 1739
-/+ buffers/cache: 18103 126708
Swap: 12284 0 12284

[Expert@CPGW02:0]# ethtool -i eth1
driver: ixgbe
version: 3.9.15-NAPI
firmware-version: 0x80000868
bus-info: 0000:1a:00.1

[Expert@CPGW02:0]# ethtool -i eth3
driver: ixgbe
version: 3.9.15-NAPI
firmware-version: 0x80000868
bus-info: 0000:24:00.1

[Expert@CPGW02:0]# ethtool -i eth0
driver: ixgbe
version: 3.9.15-NAPI
firmware-version: 0x80000868
bus-info: 0000:1a:00.0

[Expert@CPGW02:0]# ethtool -i eth2
driver: ixgbe
version: 3.9.15-NAPI
firmware-version: 0x80000868
bus-info: 0000:24:00.0
[Expert@CPGW02:0]# ethtool -i eth5
driver: igb
version: 4.1.2
firmware-version: 3.0, 0x61a30001
bus-info: 0000:15:00.1
[Expert@CPGW02:0]# ethtool -S eth5
NIC statistics:
rx_packets: 551372217
tx_packets: 438921035
rx_bytes: 660948804668
tx_bytes: 115098787091
rx_broadcast: 381513
tx_broadcast: 5039
rx_multicast: 908636
tx_multicast: 850432
multicast: 908636
collisions: 0
rx_crc_errors: 0
rx_no_buffer_count: 161
rx_missed_errors: 16809
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 660948804668
tx_dma_out_of_sync: 0
lro_aggregated: 0
lro_flushed: 0
lro_recycled: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 16809
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_queue_0_packets: 438921035
tx_queue_0_bytes: 112248339969
tx_queue_0_restart: 0
rx_queue_0_packets: 551372217
rx_queue_0_bytes: 658743315800
rx_queue_0_drops: 0
rx_queue_0_csum_err: 153587
rx_queue_0_alloc_failed: 0
[Expert@CPGW02:0]#


[Expert@CPGW02:0]# ethtool -S eth1
NIC statistics:
rx_packets: 480694151
tx_packets: 469714252
rx_bytes: 138326521201
tx_bytes: 516766853042
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 9189101
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 1488
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 480694151
tx_pkts_nic: 469714253
rx_bytes_nic: 142171766339
tx_bytes_nic: 520615914739
lsc_int: 3
tx_busy: 0
non_eop_descs: 0
broadcast: 7003242
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 4
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 2
rx_flow_control_xon: 0
tx_flow_control_xoff: 791
rx_flow_control_xoff: 0
rx_csum_offload_errors: 452
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 469714253
tx_queue_0_bytes: 516766853108
rx_queue_0_packets: 480694151
rx_queue_0_bytes: 138326521201
[Expert@CPGW02:0]#

[Expert@CPGW02:0]# ethtool -S eth3
NIC statistics:
rx_packets: 5246700
tx_packets: 231302446
rx_bytes: 449074177
tx_bytes: 258956587338
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 5149044
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 5246700
tx_pkts_nic: 231302446
rx_bytes_nic: 490962405
tx_bytes_nic: 260834179150
lsc_int: 5
tx_busy: 0
non_eop_descs: 0
broadcast: 97215
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 1
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 231302446
tx_queue_0_bytes: 258956587338
rx_queue_0_packets: 5246700
rx_queue_0_bytes: 449074177
[Expert@CPGW02:0]#


[Expert@CPGW02:0]# ethtool -S eth0
NIC statistics:
rx_packets: 78462184
tx_packets: 56154414
rx_bytes: 43594922619
tx_bytes: 25547031021
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 1058434
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 78462184
tx_pkts_nic: 56154414
rx_bytes_nic: 44222519943
tx_bytes_nic: 26015093618
lsc_int: 3
tx_busy: 0
non_eop_descs: 0
broadcast: 72179
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 56154414
tx_queue_0_bytes: 25547031021
rx_queue_0_packets: 78462184
rx_queue_0_bytes: 43594922619
[Expert@CPGW02:0]#
[Expert@CPGW02:0]# ethtool -S eth2
NIC statistics:
rx_packets: 85218744
tx_packets: 65502377
rx_bytes: 100387796187
tx_bytes: 27947313866
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 488186
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 85218744
tx_pkts_nic: 65502377
rx_bytes_nic: 101069224559
tx_bytes_nic: 28491873002
lsc_int: 3
tx_busy: 0
non_eop_descs: 0
broadcast: 5769
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 65502377
tx_queue_0_bytes: 27947313866
rx_queue_0_packets: 85218744
rx_queue_0_bytes: 100387796187
[Expert@CPGW02:0]#
[Expert@CPGW02:0]# /sbin/cpuinfo
HyperThreading=disabled
[Expert@CPGW02:0]#



the card we use is of INTEL

3 ) eth1 and eth3 form bond1 , which is our Local Network , and eth0 and eth2 form bond2 , which is our DMZ Network

both bond1 and bond2 are connected to their own cisco nexus switches and VPC runs on these switches
my topology is full mesh Redundancy

[ShadowPeak.com]

[Expert@CPGW02:0]# enabled_blades
fw vpn cvpn urlf appi ips identityServer SSL_INSPECT

A few observations:

1) Check the topology screen of your firewall object. Is the External interface correctly defined? Are ALL other Internal interfaces of the firewall completely and correctly defined? Any missing interfaces defined in the Gaia OS but not the objects's topology will be assumed to be External and get traffic to/from them sucked into the PXL path at Gigabit speeds, not good at all for performance. Rectifying a topology issue like this may well fix everything else in the list below.

2) The presence of HTTPS Inspection explains why 27% of your traffic is F2F. Check your HTTPS Inspection policy, in the Destination column of all HTTPS Inspection rules with an action of Inspect, are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" in the destination will cause HTTPS traffic between internal interfaces (including DMZs) to get pulled into the F2F path at LAN speeds.

3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

4) If there is a "Any Internet Any Log" rule at the end of your APCL/URLF Inspection policy (or even worse a "Any Any Any Log" rule), try removing it. Removing it will not cause an outage, the implied rule at the end of APCL/URLF policy is Allow.

5) What is the setting on the IPS screen of the cluster for Protection Scope? If other steps above have not resolved the problem, run "ips off" on the active gateway, wait 60 seconds, then start a transfer in a completely new connection. Does performance improve? Don't forget to run "ips on" (or reinstall policy) when done!

[coskun_ist]

hello
your advice has worked and the problem is solved
thanks for your support

[ShadowPeak.com]

hello
your advice has worked and the problem is solved
thanks for your support

Thanks for the followup, can you please elaborate on which of the tips above solved the problem? Did you just need to correct/update the topology settings on your firewall object or were some policy adjustments required as well?

[varera]

Thanks for the followup, can you please elaborate on which of the tips above solved the problem? Did you just need to correct/update the topology settings on your firewall object or were some policy adjustments required as well?

Second that, would be nice to know which particular advice helped

[coskun_ist]

hello
when we sellect internet icon on destination column on application and url and https inspection

3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

[varera]

hello
when we sellect internet icon on destination column on application and url and https inspection

3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

Clear now, thanks

[ShadowPeak.com]

hello
when we sellect internet icon on destination column on application and url and https inspection

3) Check your APCL/URLF Inspection policy, in the Destination of any rule are you using anything other than "Internet" (Cloud Icon)? Not using "Internet" (or especially using "Any" in the Destination) causes traffic between internal interfaces (including DMZs) to get sucked into the PXL path at LAN speeds.

Great, thank you for the clarification.

[juniorra22]

DUP