CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 7 of 7

Thread: Problem with Packet Loss

  1. #1
    Join Date
    2018-02-15
    Posts
    3
    Rep Power
    0

    Default Problem with Packet Loss

    Hello everyone!

    I'm facing a problem with packet loss on our two Checkpoint 4400 configured in HA mode.

    Example: I'm connected via the Endpoint Security VPN Client in my home office and try to work on some servers via RDP.
    The RDP session hangs randomly "Connection timeout / trying to reconnect". If I run a PING to serveral internal hosts I can reproduce that aswell (timeout).
    After a few seconds the connection stabilizes and the session is restored.

    On the currently active firewall member it looks like that:


    [Expert@CP4400-1:0]# installed_jumbo_take
    R77.30 Jumbo Hotfix Accumulator take_292 is installed, see sk106162.


    [Expert@CP4400-1:0]# enabled_blades
    fw vpn urlf av appi ips SSL_INSPECT anti_bot ThreatEmulation


    [Expert@CP4400-1:0]# fwaccel stats -s
    Accelerated conns/Total conns : 0/356 (0%)
    Accelerated pkts/Total pkts : 10/1001667 (0%)
    F2Fed pkts/Total pkts : 652236/1001667 (65%)
    PXL pkts/Total pkts : 349421/1001667 (34%)
    QXL pkts/Total pkts : 150218/1001667 (14%)


    [Expert@CP4400-1:0]# fwaccel stats
    Name Value Name Value
    -------------------- --------------- -------------------- ---------------

    Accelerated Path
    ------------------------------------------------------------------------------
    accel packets 10 accel bytes 400
    conns created 32064 conns deleted 27329
    C total conns 635 C templates 1
    C TCP conns 300 C delayed TCP conns 0
    C non TCP conns 335 C delayed nonTCP con 0
    conns from templates 2 temporary conns 0
    nat conns 31444 dropped packets 305
    dropped bytes 451178 nat templates 0
    port alloc templates 0 conns from nat tmpl 0
    port alloc conns 0 conns auto expired 4102

    Accelerated VPN Path
    ------------------------------------------------------------------------------
    C crypt conns 27 enc bytes 6599632
    dec bytes 3244880 ESP enc pkts 11891
    ESP enc err 139 ESP dec pkts 12305
    ESP dec err 0 ESP other err 0
    AH enc pkts 0 AH enc err 0
    AH dec pkts 0 AH dec err 0
    AH other err 0 espudp enc pkts 0
    espudp enc err 0 espudp dec pkts 0
    espudp dec err 0 espudp other err 0

    Medium Path
    ------------------------------------------------------------------------------
    PXL packets 354678 PXL async packets 355202
    PXL bytes 264837898 C PXL conns 238
    C PXL templates 1 PXL FF conns 0
    PXL FF packets 0 PXL FF bytes 0
    PXL FF acks 0

    Accelerated QoS Path
    ------------------------------------------------------------------------------
    QXL packets 154011 QXL async packets 138697
    QXL bytes 97369195 C QXL conns 191
    C QXL templates 0

    Firewall Path
    ------------------------------------------------------------------------------
    F2F packets 667611 F2F bytes 298634218
    C F2F conns 397 TCP violations 407
    C partial conns 0 C anticipated conns 0
    port alloc f2f 0

    GTP
    ------------------------------------------------------------------------------
    gtp tunnels created 0 gtp tunnels 0
    gtp accel pkts 0 gtp f2f pkts 0
    gtp spoofed pkts 0 gtp in gtp pkts 0
    gtp signaling pkts 0 gtp tcpopt pkts 0
    gtp apn err pkts 0

    General
    ------------------------------------------------------------------------------
    memory used 0 free memory 0
    C used templates 0 pxl tmpl conns 2
    C conns from tmpl 0 C non TCP F2F conns 167
    C tcp handshake conn 0 C tcp established co 217
    C tcp closed conns 83 C tcp f2f handshake 0
    C tcp f2f establishe 159 C tcp f2f closed con 71
    C tcp pxl handshake 0 C tcp pxl establishe 58
    C tcp pxl closed con 12 outbound packets 10
    outbound pxl packets 352889 outbound f2f packets 600813
    outbound bytes 600 outbound pxl bytes 270192491
    outbound f2f bytes 318174680

    (*) Statistics marked with C refer to current value, others refer to total value


    [Expert@CP4400-1:0]# fwaccel stats -p
    F2F packets:
    --------------
    Violation Packets Violation Packets
    -------------------- --------------- -------------------- ---------------
    pkt is a fragment 2 pkt has IP options 119
    ICMP miss conn 2811 TCP-SYN miss conn 15412
    TCP-other miss conn 9821 UDP miss conn 216972
    other miss conn 0 VPN returned F2F 26
    ICMP conn is F2Fed 0 TCP conn is F2Fed 455617
    UDP conn is F2Fed 1747 other conn is F2Fed 0
    uni-directional viol 0 possible spoof viol 0
    TCP state viol 587 out if not def/accl 0
    bridge, src=dst 0 routing decision err 0
    sanity checks failed 0 temp conn expired 0
    fwd to non-pivot 0 broadcast/multicast 0
    cluster message 0 partial conn 0
    PXL returned F2F 542 cluster forward 0
    chain forwarding 0 general reason 0



    [Expert@CP4400-1:0]# netstat -ni
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    Mgmt 1500 0 53938464 0 1369 0 53659044 0 0 0 BMRU
    eth1 1500 0 94223593 0 9845 0 78306516 0 0 0 BMRU
    eth2 1500 0 109361623 0 396 0 112538316 0 0 0 BMRU
    eth3 1500 0 195959635 1 111417 0 258283032 0 0 0 BMRU
    eth4 1500 0 193285104 0 87299 0 243397320 0 0 0 BMRU
    eth7 1500 0 183311404 0 11326 0 178746301 0 0 0 BMRU
    lo 16436 0 58270633 0 0 0 58270633 0 0 0 LRU


    [Expert@CP4400-1:0]# ethtool eth1
    Settings for eth1:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
    Link detected: yes
    [Expert@CP4400-1:0]# ethtool eth2
    Settings for eth2:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
    Link detected: yes
    [Expert@CP4400-1:0]# ethtool eth3
    Settings for eth3:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
    Link detected: yes
    [Expert@CP4400-1:0]# ethtool eth4
    Settings for eth4:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
    Link detected: yes
    [Expert@CP4400-1:0]# ethtool eth7
    Settings for eth7:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
    Link detected: yes


    [Expert@CP4400-1:0]# fw ctl affinity -l -r
    CPU 0: eth3 Mgmt eth4
    fw_1
    CPU 1: eth1 eth2 eth7
    fw_0
    All: in.emaild.mta fwd scrubd lpd mpdaemon usrchkd scanengine_b in.msd fgd50 vpnd scrub_cp_file_convertd scanengine_k rad in.acapd cpd cprid


    [Expert@CP4400-1:0]# sim affinity -l
    Mgmt : 0
    eth1 : 1
    eth2 : 1
    eth3 : 0
    eth4 : 0
    eth7 : 1


    [Expert@CP4400-1:0]# fw ctl multik stat
    ID | Active | CPU | Connections | Peak
    ----------------------------------------------
    0 | Yes | 1 | 305 | 4689
    1 | Yes | 0 | 208 | 3256


    [Expert@CP4400-1:0]# fw ctl multik get_mode
    Current mode is On


    [Expert@CP4400-1:0]# fw ctl pstat

    System Capacity Summary:
    Memory used: 28% (447 MB out of 1587 MB) - below watermark
    Concurrent Connections: 3% (830 out of 24900) - below watermark
    Aggressive Aging is not active

    Hash kernel memory (hmem) statistics:
    Total memory allocated: 289406976 bytes in 70656 (4096 bytes) blocks using 92 pools
    Initial memory allocated: 163577856 bytes (Hash memory extended by 125829120 bytes)
    Memory allocation limit: 289406976 bytes using 512 pools
    Total memory bytes used: 152051496 unused: 137355480 (47.46%) peak: 304472140
    Total memory blocks used: 45787 unused: 24869 (35%) peak: 74841
    Allocations: 3217346246 alloc, 12702722 failed alloc, 3214660748 free

    System kernel memory (smem) statistics:
    Total memory bytes used: 476525088 peak: 599410556
    Total memory bytes wasted: 48499443
    Blocking memory bytes used: 12501556 peak: 42940064
    Non-Blocking memory bytes used: 464023532 peak: 556470492
    Allocations: 25479776 alloc, 45 failed alloc, 25165757 free, 0 failed free
    vmalloc bytes used: 25996204 expensive: yes

    Kernel memory (kmem) statistics:
    Total memory bytes used: 333440172 peak: 560623508
    Allocations: 3242789433 alloc, 0 failed alloc
    3239791102 free, 0 failed free
    External Allocations: 419564 for packets, 44415062 for SXL

    Cookies:
    1026822863 total, 22794219 alloc, 22794219 free,
    4277711 dup, 3749029114 get, 444976521 put,
    2182698671 len, 177981 cached len, 0 chain alloc,
    0 chain free

    Connections:
    12169983 total, 4101610 TCP, 7826209 UDP, 153164 ICMP,
    89000 other, 135 anticipated, 0 recovered, 832 concurrent,
    6499 peak concurrent

    Fragments:
    163735 fragments, 80587 packets, 35 expired, 0 short,
    0 large, 0 duplicates, 0 failures

    NAT:
    97591026/0 forw, 96012168/0 bckw, 91830813 tcpudp,
    31164 icmp, 11446506-24997123 alloc

    Sync:
    Version: new
    Status: Able to Send/Receive sync packets
    Sync packets sent:
    total : 32819740, retransmitted : 2203, retrans reqs : 2941, acks : 1191308
    Sync packets received:
    total : 38378083, were queued : 38511, dropped by net : 2622
    retrans reqs : 1296, received 1398442 acks
    retrans reqs for illegal seq : 0
    dropped updates as a result of sync overload: 1527
    Callback statistics: handled 1383828 cb, average delay : 1, max delay : 40


    Thank you for your help in advance!

    Volker

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,216
    Rep Power
    13

    Default Re: Problem with Packet Loss

    Is this a Full HA configuration? In other words do you not have a separate SMS that you connect into with the SmartDhasboard and the two 4400's are basically self-managed? If so the two boxes are almost certainly starved for memory given the number of blades you have enabled and the hash memory allocation failures, please provide output of free -m and the cpconfig menu. The number of blades you have enabled is pretty high for such a limited box like the 4400.
    Last edited by ShadowPeak.com; 2018-03-27 at 16:15.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2018-02-15
    Posts
    3
    Rep Power
    0

    Default Re: Problem with Packet Loss

    Quote Originally Posted by ShadowPeak.com View Post
    Is this a Full HA configuration? In other words do you not have a separate SMS that you connect into with the SmartDhasboard and the two 4400's are basically self-managed? If so the two boxes are almost certainly starved for memory given the number of blades you have enabled and the hash memory allocation failures, please provide output of free -m and the cpconfig menu. The number of blades you have enabled is pretty high for such a limited box like the 4400.
    Yes, we changed it to a standalone management VM and reinstalled the firewalls not long ago. Formerly the management was also running on the firewalls; it was nearly impossible to install the policy without trying it 3 times or more (totally out of memory and disk space/performance).
    Now it is running smoothly from a management/policy installation perspective.


    [Expert@CP4400-1:0]# free -m
    total used free shared buffers cached
    Mem: 3948 3808 140 0 4 380
    -/+ buffers/cache: 3423 525
    Swap: 10268 2558 7709


    [Expert@CP4400-1:0]# cpconfig
    This program will let you re-configure
    your Check Point products configuration.


    Configuration Options:
    ----------------------
    (1) Licenses and contracts
    (2) SNMP Extension
    (3) PKCS#11 Token
    (4) Random Pool
    (5) Secure Internal Communication
    (6) Disable cluster membership for this gateway
    (7) Enable Check Point Per Virtual System State
    (8) Enable Check Point ClusterXL for Bridge Active/Standby
    (9) Disable Check Point SecureXL
    (10) Check Point CoreXL
    (11) Automatic start of Check Point Products

    (12) Exit

  4. #4
    Join Date
    2006-04-27
    Location
    Twillight zone
    Posts
    1,006
    Rep Power
    14

    Default Re: Problem with Packet Loss

    That's a lot of active blades for 4400, as Shadowpeak says, you have memory issues, no doubt about that.

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,216
    Rep Power
    13

    Default Re: Problem with Packet Loss

    Quote Originally Posted by VolkerM View Post
    Yes, we changed it to a standalone management VM and reinstalled the firewalls not long ago. Formerly the management was also running on the firewalls; it was nearly impossible to install the policy without trying it 3 times or more (totally out of memory and disk space/performance).
    Now it is running smoothly from a management/policy installation perspective.


    [Expert@CP4400-1:0]# free -m
    total used free shared buffers cached
    Mem: 3948 3808 140 0 4 380
    -/+ buffers/cache: 3423 525
    Swap: 10268 2558 7709


    [Expert@CP4400-1:0]# cpconfig
    This program will let you re-configure
    your Check Point products configuration.


    Configuration Options:
    ----------------------
    (1) Licenses and contracts
    (2) SNMP Extension
    (3) PKCS#11 Token
    (4) Random Pool
    (5) Secure Internal Communication
    (6) Disable cluster membership for this gateway
    (7) Enable Check Point Per Virtual System State
    (8) Enable Check Point ClusterXL for Bridge Active/Standby
    (9) Disable Check Point SecureXL
    (10) Check Point CoreXL
    (11) Automatic start of Check Point Products

    (12) Exit
    Your firewall is 2.5GB into swap space against RAM of only 4GB. Upgrading to 8GB of RAM will definitely help. A lot.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  6. #6
    Join Date
    2018-02-15
    Posts
    3
    Rep Power
    0

    Default Re: Problem with Packet Loss

    Thank you for giving me a hint!
    The problem is that the RAM of the 4400 boxes is not expandable. So it looks like that I have to buy two new ones.

  7. #7
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,216
    Rep Power
    13

    Default Re: Problem with Packet Loss

    Quote Originally Posted by VolkerM View Post
    Thank you for giving me a hint!
    The problem is that the RAM of the 4400 boxes is not expandable. So it looks like that I have to buy two new ones.
    If you weren't tipped so far over into swap space there might be some memory optimizations that could be performed to reduce memory utilization, but that is probably a lost cause given the number of blades you have enabled. :-)

    Any new appliance model 3100+ will ship with a minimum of 8GB of RAM so you should be all set there. 5000 models and higher can have their memory expanded beyond 8GB.

    Given the number of blades enabled, I'd strongly recommend obtaining a replacement appliance with 4 cores if you can swing it from a budget perspective. Quad-core models start at the 5600 series. The 5800 is quad-core as well but also commences support for SMT/Hyperthreading, which could give you the extra kick of 8 logical cores should you need it down the road. You can check out appliance specs here:

    https://lwf.fink.sh/2017/02/28/check...ary-28st-2017/

    Make sure your reseller gives you a trade-in credit for your 4400's, and also quotes the -HA licensing option on your secondary firewall. Good luck!
    Last edited by ShadowPeak.com; 2018-04-04 at 09:52.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

Similar Threads

  1. UTM-1 Edge packet loss when cent. mgt.
    By dhoobler76 in forum Check Point UTM-1 Edge Appliances
    Replies: 1
    Last Post: 2009-08-25, 23:42
  2. Replies: 4
    Last Post: 2008-04-18, 04:41
  3. Packet loss on one of interfaces
    By Varyag in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 2
    Last Post: 2008-03-13, 03:24
  4. Crazy Securemote Packet Loss problem
    By safetyboy in forum SecureClient/SecuRemote
    Replies: 1
    Last Post: 2006-11-10, 11:42

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •