Hello everyone!
I'm facing a problem with packet loss on our two Checkpoint 4400 configured in HA mode.
Example: I'm connected via the Endpoint Security VPN Client in my home office and try to work on some servers via RDP.
The RDP session hangs randomly "Connection timeout / trying to reconnect". If I run a PING to serveral internal hosts I can reproduce that aswell (timeout).
After a few seconds the connection stabilizes and the session is restored.
On the currently active firewall member it looks like that:
[Expert@CP4400-1:0]# installed_jumbo_take
R77.30 Jumbo Hotfix Accumulator take_292 is installed, see sk106162.
[Expert@CP4400-1:0]# enabled_blades
fw vpn urlf av appi ips SSL_INSPECT anti_bot ThreatEmulation
[Expert@CP4400-1:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/356 (0%)
Accelerated pkts/Total pkts : 10/1001667 (0%)
F2Fed pkts/Total pkts : 652236/1001667 (65%)
PXL pkts/Total pkts : 349421/1001667 (34%)
QXL pkts/Total pkts : 150218/1001667 (14%)
[Expert@CP4400-1:0]# fwaccel stats
Name Value Name Value
-------------------- --------------- -------------------- ---------------
Accelerated Path
------------------------------------------------------------------------------
accel packets 10 accel bytes 400
conns created 32064 conns deleted 27329
C total conns 635 C templates 1
C TCP conns 300 C delayed TCP conns 0
C non TCP conns 335 C delayed nonTCP con 0
conns from templates 2 temporary conns 0
nat conns 31444 dropped packets 305
dropped bytes 451178 nat templates 0
port alloc templates 0 conns from nat tmpl 0
port alloc conns 0 conns auto expired 4102
Accelerated VPN Path
------------------------------------------------------------------------------
C crypt conns 27 enc bytes 6599632
dec bytes 3244880 ESP enc pkts 11891
ESP enc err 139 ESP dec pkts 12305
ESP dec err 0 ESP other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0
Medium Path
------------------------------------------------------------------------------
PXL packets 354678 PXL async packets 355202
PXL bytes 264837898 C PXL conns 238
C PXL templates 1 PXL FF conns 0
PXL FF packets 0 PXL FF bytes 0
PXL FF acks 0
Accelerated QoS Path
------------------------------------------------------------------------------
QXL packets 154011 QXL async packets 138697
QXL bytes 97369195 C QXL conns 191
C QXL templates 0
Firewall Path
------------------------------------------------------------------------------
F2F packets 667611 F2F bytes 298634218
C F2F conns 397 TCP violations 407
C partial conns 0 C anticipated conns 0
port alloc f2f 0
GTP
------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0
General
------------------------------------------------------------------------------
memory used 0 free memory 0
C used templates 0 pxl tmpl conns 2
C conns from tmpl 0 C non TCP F2F conns 167
C tcp handshake conn 0 C tcp established co 217
C tcp closed conns 83 C tcp f2f handshake 0
C tcp f2f establishe 159 C tcp f2f closed con 71
C tcp pxl handshake 0 C tcp pxl establishe 58
C tcp pxl closed con 12 outbound packets 10
outbound pxl packets 352889 outbound f2f packets 600813
outbound bytes 600 outbound pxl bytes 270192491
outbound f2f bytes 318174680
(*) Statistics marked with C refer to current value, others refer to total value
[Expert@CP4400-1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 2 pkt has IP options 119
ICMP miss conn 2811 TCP-SYN miss conn 15412
TCP-other miss conn 9821 UDP miss conn 216972
other miss conn 0 VPN returned F2F 26
ICMP conn is F2Fed 0 TCP conn is F2Fed 455617
UDP conn is F2Fed 1747 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 0
TCP state viol 587 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 542 cluster forward 0
chain forwarding 0 general reason 0
[Expert@CP4400-1:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 53938464 0 1369 0 53659044 0 0 0 BMRU
eth1 1500 0 94223593 0 9845 0 78306516 0 0 0 BMRU
eth2 1500 0 109361623 0 396 0 112538316 0 0 0 BMRU
eth3 1500 0 195959635 1 111417 0 258283032 0 0 0 BMRU
eth4 1500 0 193285104 0 87299 0 243397320 0 0 0 BMRU
eth7 1500 0 183311404 0 11326 0 178746301 0 0 0 BMRU
lo 16436 0 58270633 0 0 0 58270633 0 0 0 LRU
[Expert@CP4400-1:0]# ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth4
Settings for eth4:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth7
Settings for eth7:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# fw ctl affinity -l -r
CPU 0: eth3 Mgmt eth4
fw_1
CPU 1: eth1 eth2 eth7
fw_0
All: in.emaild.mta fwd scrubd lpd mpdaemon usrchkd scanengine_b in.msd fgd50 vpnd scrub_cp_file_convertd scanengine_k rad in.acapd cpd cprid
[Expert@CP4400-1:0]# sim affinity -l
Mgmt : 0
eth1 : 1
eth2 : 1
eth3 : 0
eth4 : 0
eth7 : 1
[Expert@CP4400-1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 305 | 4689
1 | Yes | 0 | 208 | 3256
[Expert@CP4400-1:0]# fw ctl multik get_mode
Current mode is On
[Expert@CP4400-1:0]# fw ctl pstat
System Capacity Summary:
Memory used: 28% (447 MB out of 1587 MB) - below watermark
Concurrent Connections: 3% (830 out of 24900) - below watermark
Aggressive Aging is not active
Hash kernel memory (hmem) statistics:
Total memory allocated: 289406976 bytes in 70656 (4096 bytes) blocks using 92 pools
Initial memory allocated: 163577856 bytes (Hash memory extended by 125829120 bytes)
Memory allocation limit: 289406976 bytes using 512 pools
Total memory bytes used: 152051496 unused: 137355480 (47.46%) peak: 304472140
Total memory blocks used: 45787 unused: 24869 (35%) peak: 74841
Allocations: 3217346246 alloc, 12702722 failed alloc, 3214660748 free
System kernel memory (smem) statistics:
Total memory bytes used: 476525088 peak: 599410556
Total memory bytes wasted: 48499443
Blocking memory bytes used: 12501556 peak: 42940064
Non-Blocking memory bytes used: 464023532 peak: 556470492
Allocations: 25479776 alloc, 45 failed alloc, 25165757 free, 0 failed free
vmalloc bytes used: 25996204 expensive: yes
Kernel memory (kmem) statistics:
Total memory bytes used: 333440172 peak: 560623508
Allocations: 3242789433 alloc, 0 failed alloc
3239791102 free, 0 failed free
External Allocations: 419564 for packets, 44415062 for SXL
Cookies:
1026822863 total, 22794219 alloc, 22794219 free,
4277711 dup, 3749029114 get, 444976521 put,
2182698671 len, 177981 cached len, 0 chain alloc,
0 chain free
Connections:
12169983 total, 4101610 TCP, 7826209 UDP, 153164 ICMP,
89000 other, 135 anticipated, 0 recovered, 832 concurrent,
6499 peak concurrent
Fragments:
163735 fragments, 80587 packets, 35 expired, 0 short,
0 large, 0 duplicates, 0 failures
NAT:
97591026/0 forw, 96012168/0 bckw, 91830813 tcpudp,
31164 icmp, 11446506-24997123 alloc
Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 32819740, retransmitted : 2203, retrans reqs : 2941, acks : 1191308
Sync packets received:
total : 38378083, were queued : 38511, dropped by net : 2622
retrans reqs : 1296, received 1398442 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 1527
Callback statistics: handled 1383828 cb, average delay : 1, max delay : 40
Thank you for your help in advance!
Volker
Bookmarks