Problem with Packet Loss

[VolkerM]

Hello everyone!

I'm facing a problem with packet loss on our two Checkpoint 4400 configured in HA mode.

Example: I'm connected via the Endpoint Security VPN Client in my home office and try to work on some servers via RDP.
The RDP session hangs randomly "Connection timeout / trying to reconnect". If I run a PING to serveral internal hosts I can reproduce that aswell (timeout).
After a few seconds the connection stabilizes and the session is restored.

On the currently active firewall member it looks like that:


[Expert@CP4400-1:0]# installed_jumbo_take
R77.30 Jumbo Hotfix Accumulator take_292 is installed, see sk106162.


[Expert@CP4400-1:0]# enabled_blades
fw vpn urlf av appi ips SSL_INSPECT anti_bot ThreatEmulation


[Expert@CP4400-1:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/356 (0%)
Accelerated pkts/Total pkts : 10/1001667 (0%)
F2Fed pkts/Total pkts : 652236/1001667 (65%)
PXL pkts/Total pkts : 349421/1001667 (34%)
QXL pkts/Total pkts : 150218/1001667 (14%)


[Expert@CP4400-1:0]# fwaccel stats
Name Value Name Value
-------------------- --------------- -------------------- ---------------

Accelerated Path
------------------------------------------------------------------------------
accel packets 10 accel bytes 400
conns created 32064 conns deleted 27329
C total conns 635 C templates 1
C TCP conns 300 C delayed TCP conns 0
C non TCP conns 335 C delayed nonTCP con 0
conns from templates 2 temporary conns 0
nat conns 31444 dropped packets 305
dropped bytes 451178 nat templates 0
port alloc templates 0 conns from nat tmpl 0
port alloc conns 0 conns auto expired 4102

Accelerated VPN Path
------------------------------------------------------------------------------
C crypt conns 27 enc bytes 6599632
dec bytes 3244880 ESP enc pkts 11891
ESP enc err 139 ESP dec pkts 12305
ESP dec err 0 ESP other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0

Medium Path
------------------------------------------------------------------------------
PXL packets 354678 PXL async packets 355202
PXL bytes 264837898 C PXL conns 238
C PXL templates 1 PXL FF conns 0
PXL FF packets 0 PXL FF bytes 0
PXL FF acks 0

Accelerated QoS Path
------------------------------------------------------------------------------
QXL packets 154011 QXL async packets 138697
QXL bytes 97369195 C QXL conns 191
C QXL templates 0

Firewall Path
------------------------------------------------------------------------------
F2F packets 667611 F2F bytes 298634218
C F2F conns 397 TCP violations 407
C partial conns 0 C anticipated conns 0
port alloc f2f 0

GTP
------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
------------------------------------------------------------------------------
memory used 0 free memory 0
C used templates 0 pxl tmpl conns 2
C conns from tmpl 0 C non TCP F2F conns 167
C tcp handshake conn 0 C tcp established co 217
C tcp closed conns 83 C tcp f2f handshake 0
C tcp f2f establishe 159 C tcp f2f closed con 71
C tcp pxl handshake 0 C tcp pxl establishe 58
C tcp pxl closed con 12 outbound packets 10
outbound pxl packets 352889 outbound f2f packets 600813
outbound bytes 600 outbound pxl bytes 270192491
outbound f2f bytes 318174680

(*) Statistics marked with C refer to current value, others refer to total value


[Expert@CP4400-1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 2 pkt has IP options 119
ICMP miss conn 2811 TCP-SYN miss conn 15412
TCP-other miss conn 9821 UDP miss conn 216972
other miss conn 0 VPN returned F2F 26
ICMP conn is F2Fed 0 TCP conn is F2Fed 455617
UDP conn is F2Fed 1747 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 0
TCP state viol 587 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 542 cluster forward 0
chain forwarding 0 general reason 0



[Expert@CP4400-1:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 53938464 0 1369 0 53659044 0 0 0 BMRU
eth1 1500 0 94223593 0 9845 0 78306516 0 0 0 BMRU
eth2 1500 0 109361623 0 396 0 112538316 0 0 0 BMRU
eth3 1500 0 195959635 1 111417 0 258283032 0 0 0 BMRU
eth4 1500 0 193285104 0 87299 0 243397320 0 0 0 BMRU
eth7 1500 0 183311404 0 11326 0 178746301 0 0 0 BMRU
lo 16436 0 58270633 0 0 0 58270633 0 0 0 LRU


[Expert@CP4400-1:0]# ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth4
Settings for eth4:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
[Expert@CP4400-1:0]# ethtool eth7
Settings for eth7:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes


[Expert@CP4400-1:0]# fw ctl affinity -l -r
CPU 0: eth3 Mgmt eth4
fw_1
CPU 1: eth1 eth2 eth7
fw_0
All: in.emaild.mta fwd scrubd lpd mpdaemon usrchkd scanengine_b in.msd fgd50 vpnd scrub_cp_file_convertd scanengine_k rad in.acapd cpd cprid


[Expert@CP4400-1:0]# sim affinity -l
Mgmt : 0
eth1 : 1
eth2 : 1
eth3 : 0
eth4 : 0
eth7 : 1


[Expert@CP4400-1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 305 | 4689
1 | Yes | 0 | 208 | 3256


[Expert@CP4400-1:0]# fw ctl multik get_mode
Current mode is On


[Expert@CP4400-1:0]# fw ctl pstat

System Capacity Summary:
Memory used: 28% (447 MB out of 1587 MB) - below watermark
Concurrent Connections: 3% (830 out of 24900) - below watermark
Aggressive Aging is not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 289406976 bytes in 70656 (4096 bytes) blocks using 92 pools
Initial memory allocated: 163577856 bytes (Hash memory extended by 125829120 bytes)
Memory allocation limit: 289406976 bytes using 512 pools
Total memory bytes used: 152051496 unused: 137355480 (47.46%) peak: 304472140
Total memory blocks used: 45787 unused: 24869 (35%) peak: 74841
Allocations: 3217346246 alloc, 12702722 failed alloc, 3214660748 free

System kernel memory (smem) statistics:
Total memory bytes used: 476525088 peak: 599410556
Total memory bytes wasted: 48499443
Blocking memory bytes used: 12501556 peak: 42940064
Non-Blocking memory bytes used: 464023532 peak: 556470492
Allocations: 25479776 alloc, 45 failed alloc, 25165757 free, 0 failed free
vmalloc bytes used: 25996204 expensive: yes

Kernel memory (kmem) statistics:
Total memory bytes used: 333440172 peak: 560623508
Allocations: 3242789433 alloc, 0 failed alloc
3239791102 free, 0 failed free
External Allocations: 419564 for packets, 44415062 for SXL

Cookies:
1026822863 total, 22794219 alloc, 22794219 free,
4277711 dup, 3749029114 get, 444976521 put,
2182698671 len, 177981 cached len, 0 chain alloc,
0 chain free

Connections:
12169983 total, 4101610 TCP, 7826209 UDP, 153164 ICMP,
89000 other, 135 anticipated, 0 recovered, 832 concurrent,
6499 peak concurrent

Fragments:
163735 fragments, 80587 packets, 35 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
97591026/0 forw, 96012168/0 bckw, 91830813 tcpudp,
31164 icmp, 11446506-24997123 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 32819740, retransmitted : 2203, retrans reqs : 2941, acks : 1191308
Sync packets received:
total : 38378083, were queued : 38511, dropped by net : 2622
retrans reqs : 1296, received 1398442 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 1527
Callback statistics: handled 1383828 cb, average delay : 1, max delay : 40


Thank you for your help in advance!

Volker

[ShadowPeak.com]

Is this a Full HA configuration? In other words do you not have a separate SMS that you connect into with the SmartDhasboard and the two 4400's are basically self-managed? If so the two boxes are almost certainly starved for memory given the number of blades you have enabled and the hash memory allocation failures, please provide output of free -m and the cpconfig menu. The number of blades you have enabled is pretty high for such a limited box like the 4400.

[VolkerM]

Is this a Full HA configuration? In other words do you not have a separate SMS that you connect into with the SmartDhasboard and the two 4400's are basically self-managed? If so the two boxes are almost certainly starved for memory given the number of blades you have enabled and the hash memory allocation failures, please provide output of free -m and the cpconfig menu. The number of blades you have enabled is pretty high for such a limited box like the 4400.

Yes, we changed it to a standalone management VM and reinstalled the firewalls not long ago. Formerly the management was also running on the firewalls; it was nearly impossible to install the policy without trying it 3 times or more (totally out of memory and disk space/performance).
Now it is running smoothly from a management/policy installation perspective.


[Expert@CP4400-1:0]# free -m
total used free shared buffers cached
Mem: 3948 3808 140 0 4 380
-/+ buffers/cache: 3423 525
Swap: 10268 2558 7709


[Expert@CP4400-1:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.


Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Disable Check Point SecureXL
(10) Check Point CoreXL
(11) Automatic start of Check Point Products

(12) Exit

[abusharif]

That's a lot of active blades for 4400, as Shadowpeak says, you have memory issues, no doubt about that.

[ShadowPeak.com]

Yes, we changed it to a standalone management VM and reinstalled the firewalls not long ago. Formerly the management was also running on the firewalls; it was nearly impossible to install the policy without trying it 3 times or more (totally out of memory and disk space/performance).
Now it is running smoothly from a management/policy installation perspective.


[Expert@CP4400-1:0]# free -m
total used free shared buffers cached
Mem: 3948 3808 140 0 4 380
-/+ buffers/cache: 3423 525
Swap: 10268 2558 7709


[Expert@CP4400-1:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.


Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Disable Check Point SecureXL
(10) Check Point CoreXL
(11) Automatic start of Check Point Products

(12) Exit

Your firewall is 2.5GB into swap space against RAM of only 4GB. Upgrading to 8GB of RAM will definitely help. A lot.

[VolkerM]

Thank you for giving me a hint!
The problem is that the RAM of the 4400 boxes is not expandable. So it looks like that I have to buy two new ones.

[ShadowPeak.com]

Thank you for giving me a hint!
The problem is that the RAM of the 4400 boxes is not expandable. So it looks like that I have to buy two new ones.

If you weren't tipped so far over into swap space there might be some memory optimizations that could be performed to reduce memory utilization, but that is probably a lost cause given the number of blades you have enabled. :-)

Any new appliance model 3100+ will ship with a minimum of 8GB of RAM so you should be all set there. 5000 models and higher can have their memory expanded beyond 8GB.

Given the number of blades enabled, I'd strongly recommend obtaining a replacement appliance with 4 cores if you can swing it from a budget perspective. Quad-core models start at the 5600 series. The 5800 is quad-core as well but also commences support for SMT/Hyperthreading, which could give you the extra kick of 8 logical cores should you need it down the road. You can check out appliance specs here:

https://lwf.fink.sh/2017/02/28/check...ary-28st-2017/

Make sure your reseller gives you a trade-in credit for your 4400's, and also quotes the -HA licensing option on your secondary firewall. Good luck!