CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Page 1 of 2 12 LastLast
Results 1 to 20 of 33

Thread: MTU issues: packets are always fragmented by firewall!

  1. #1
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default MTU issues: our R7720 and R8810 behaves differently cocnerning fragmentation

    Hey,

    Today we had MTU problems biting us in the ass. It turns out our old appliances running R77.20 do not seem to care about the DF bit, and they were always fragmenting the packets. Why?

    (This becomes bad when you switch to something new which works more correctly, but the ICMP requests to please fragment at the source don't reach parts of your network)

    This is the communication sent by the server:

    17:01:59.427027 IP (tos 0x0, ttl 64, id 57226, offset 0, flags [DF], proto TCP (6), length 4156) 172.16.123.101.443 > 192.168.130.136.23258: . 178:4282(4104) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.427039 IP (tos 0x0, ttl 64, id 57229, offset 0, flags [DF], proto TCP (6), length 161) 172.16.123.101.443 > 192.168.130.136.23258: P 4282:4391(109) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.427776 IP (tos 0x0, ttl 64, id 57230, offset 0, flags [DF], proto TCP (6), length 121) 172.16.123.101.443 > 192.168.130.136.23258: P 4391:4460(69) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>

    This is the communication sent out the other side of the firewall (after source NAT) on an interface with lesser MTU than the ingress interface:

    17:01:59.456952 IP (tos 0x0, ttl 62, id 24458, offset 0, flags [DF], proto: TCP (6), length: 1420) 192.168.1.1.https > 192.168.130.136.23258: . 178:1546(1368) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.456964 IP (tos 0x0, ttl 62, id 24459, offset 0, flags [DF], proto: TCP (6), length: 1420) 192.168.1.1.https > 192.168.130.136.23258: . 1546:2914(1368) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.456968 IP (tos 0x0, ttl 62, id 24460, offset 0, flags [DF], proto: TCP (6), length: 1420) 192.168.1.1.https > 192.168.130.136.23258: . 2914:4282(1368) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.456974 IP (tos 0x0, ttl 62, id 57229, offset 0, flags [DF], proto: TCP (6), length: 161) 192.168.1.1.https > 192.168.130.136.23258: P 4282:4391(109) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>
    17:01:59.457595 IP (tos 0x0, ttl 62, id 57230, offset 0, flags [DF], proto: TCP (6), length: 121) 192.168.1.1.https > 192.168.130.136.23258: P 4391:4460(69) ack 17588 win 331 <nop,nop,timestamp 24432792 1397191767>

    You see that it doesn't care about the DF flag being set and fragments the packets anyway. Why would that be? If you can configure anything like this, where would you look?

    Thanks.

    marki
    Last edited by jeronimo; 2018-02-03 at 09:51.

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,229
    Rep Power
    13

    Default Re: MTU issues: packets are always fragmented by firewall!

    Assuming your tcpdump output is accurate, IP did not fragment the packets because the offset field for all the packets you think are fragmented is zero. My guess is the TCP segments within were reassembled smaller based on TCP MSS by some kind of firewall feature like IPS or HTTPS Inspection, please provide the output of enabled_blades, also fw ctl pstat.

    Also the port numbers did not change as a result of the NAT operation so a static NAT is in use here, correct?
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    There you go.

    Code:
    > fw ctl pstat
    
    System Capacity Summary:
      Memory used: 6% (106 MB out of 1617 MB) - below watermark
      Concurrent Connections: 29% (13096 out of 44900) - below watermark
      Aggressive Aging is not active
    
    Hash kernel memory (hmem) statistics:
      Total memory allocated: 167772160 bytes in 40960 (4096 bytes) blocks using 40 pools
      Total memory bytes  used: 19492220   unused: 148279940 (88.38%)   peak: 40790380
      Total memory blocks used:     7024   unused:    33936 (82%)   peak:    10837
      Allocations: 4076060574 alloc, 0 failed alloc, 4075862621 free
    
    System kernel memory (smem) statistics:
      Total memory  bytes  used: 228316012   peak: 231904992
      Total memory bytes wasted:  1883011
        Blocking  memory  bytes   used:  2562888   peak:  5989608
        Non-Blocking memory bytes used: 225753124   peak: 225915384
      Allocations: 17183351 alloc, 0 failed alloc, 17182475 free, 0 failed free
      vmalloc bytes  used: 10250500 expensive: yes
    
    Kernel memory (kmem) statistics:
      Total memory  bytes  used: 79864064   peak: 96999060
      Allocations: 4093198586 alloc, 0 failed alloc
                   4093000347 free, 0 failed free
      External Allocations: 0 for packets, 11334081 for SXL
    
    Cookies:
            2645637833 total, 0 alloc, 0 free,
            29977 dup, 3063202444 get, 4004515651 put,
            1820708845 len, 36532136 cached len, 0 chain alloc,
            0 chain free
    
    Connections:
            -1016938133 total, 1137248282 TCP, 1686904311 UDP, 453468596 ICMP,
            407974 other, 1593614 anticipated, 937393 recovered, 13096 concurrent,
            44960 peak concurrent
    
    Fragments:
            60340695 fragments, 25521762 packets, 19524 expired, 0 short,
            0 large, 0 duplicates, 0 failures
    
    NAT:
            -1957311752/0 forw, 95654455/0 bckw, 58335744 tcpudp,
            78334915 icmp, 43067913-54788358 alloc
    
    Sync:
            Version: new
            Status: Able to Send/Receive sync packets
            Sync packets sent:
             total : 930954602,  retransmitted : 1080276, retrans reqs : 20591,  acks : 87397
            Sync packets received:
             total : 1087297799,  were queued : 32688642, dropped by net : 94523
             retrans reqs : 178664, received 966358 acks
             retrans reqs for illegal seq : 0
             dropped updates as a result of sync overload: 706
    Code:
    # enabled_blades
    fw ips

  4. #4
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    Oh and yes, it's a static nat.

  5. #5
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: packets are always fragmented by firewall!

    I think something in checkpoint is acting like a proxy. I don't know if its checkpoint active streaming or a legit proxied connection. In either case as shadow peak pointed out this does not seem to be a IP fragmented packet. TCP has a option that is set on syn packets called MSS. This says how big of a payload the tcp packet can accept. If checkpoint wasn't building a new tcp connection then this connection should be dropped from an OS level and a icmp unreachable i'm thinking would have been sent.

    What is the problem you're seeing? By not enabling jump frames on both interfaces of the firewall you're asking for trouble unless you have something on the firewall (or something else) change the MSS value to the default for a 1500 byte frame (or is it 1514?). This of course wouldn't help you with udp based applications but there aren't may that use large frames.

  6. #6
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,229
    Rep Power
    13

    Default Re: MTU issues: packets are always fragmented by firewall!

    Quote Originally Posted by jeronimo View Post
    There you go.

    Code:
    > fw ctl pstat
    
    System Capacity Summary:
      Memory used: 6% (106 MB out of 1617 MB) - below watermark
      Concurrent Connections: 29% (13096 out of 44900) - below watermark
      Aggressive Aging is not active
    
    Hash kernel memory (hmem) statistics:
      Total memory allocated: 167772160 bytes in 40960 (4096 bytes) blocks using 40 pools
      Total memory bytes  used: 19492220   unused: 148279940 (88.38%)   peak: 40790380
      Total memory blocks used:     7024   unused:    33936 (82%)   peak:    10837
      Allocations: 4076060574 alloc, 0 failed alloc, 4075862621 free
    
    System kernel memory (smem) statistics:
      Total memory  bytes  used: 228316012   peak: 231904992
      Total memory bytes wasted:  1883011
        Blocking  memory  bytes   used:  2562888   peak:  5989608
        Non-Blocking memory bytes used: 225753124   peak: 225915384
      Allocations: 17183351 alloc, 0 failed alloc, 17182475 free, 0 failed free
      vmalloc bytes  used: 10250500 expensive: yes
    
    Kernel memory (kmem) statistics:
      Total memory  bytes  used: 79864064   peak: 96999060
      Allocations: 4093198586 alloc, 0 failed alloc
                   4093000347 free, 0 failed free
      External Allocations: 0 for packets, 11334081 for SXL
    
    Cookies:
            2645637833 total, 0 alloc, 0 free,
            29977 dup, 3063202444 get, 4004515651 put,
            1820708845 len, 36532136 cached len, 0 chain alloc,
            0 chain free
    
    Connections:
            -1016938133 total, 1137248282 TCP, 1686904311 UDP, 453468596 ICMP,
            407974 other, 1593614 anticipated, 937393 recovered, 13096 concurrent,
            44960 peak concurrent
    
    Fragments:
            60340695 fragments, 25521762 packets, 19524 expired, 0 short,
            0 large, 0 duplicates, 0 failures
    
    NAT:
            -1957311752/0 forw, 95654455/0 bckw, 58335744 tcpudp,
            78334915 icmp, 43067913-54788358 alloc
    
    Sync:
            Version: new
            Status: Able to Send/Receive sync packets
            Sync packets sent:
             total : 930954602,  retransmitted : 1080276, retrans reqs : 20591,  acks : 87397
            Sync packets received:
             total : 1087297799,  were queued : 32688642, dropped by net : 94523
             retrans reqs : 178664, received 966358 acks
             retrans reqs for illegal seq : 0
             dropped updates as a result of sync overload: 706
    Code:
    # enabled_blades
    fw ips
    Must be some function of IPS, try running ips off and retest to see if the reduction in packet size persists. Don't forget to turn IPS back on with ips on when you are done!
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  7. #7
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: our R7720 and R8810 behaves differently cocnerning fragmentation

    I'm so sorry, my approach at tackling this was entirely wrong and I pointed you in the wrong direction.

    The first capture was taken on the server, apparently before segmentation takes place. Great. *insert cynical applause here* What we see on the Checkpoint is simply processing what the server is sending all the time. So no mystery here.

    The server seems to segment the packets into MTU 1420 packets mostly. (a good question why this is so low)

    The ingress interface of the firewall is set to 1500 while the egress interface is set to 1432 (cause remote network needed this)

    On R7720 there was no problem sending stuff to the remote network at all. I'm trying to understand why. On R8810 the Checkpoint is sending ICMPs asking to please fragment. This seems more correct, but I'm trying to understand why there is such a drastic change in behavior.

    In any case, with the new facts, I will now need to collect more info what happens exactly. (what kind of packet goes through in each version and what does not)

  8. #8
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: our R7720 and R8810 behaves differently cocnerning fragmentation

    Check out sk61221 for how to do mss clamping.

  9. #9
    Join Date
    2006-09-26
    Posts
    3,164
    Rep Power
    16

    Default Re: MTU issues: our R7720 and R8810 behaves differently cocnerning fragmentation

    Quote Originally Posted by jflemingeds View Post
    Check out sk61221 for how to do mss clamping.
    How is that going to help with UDP traffics?

  10. #10
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    It won't help with UDP traffic, but that is not the point anyway.

    I am still looking for differences concerning MTU and fragmentation behavior between R7720 and R8010.

    I am investigating this on our infrastructure and I'll get back when I have more info. Meanwhile, feel free to let me know if something comes to mind :)

  11. #11
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    There are two things I don't understand for now:

    1) Look at the following packet capture which consists of two simultaneaous remote mirrors, one in front (1.2.3.1) and one behind (1.2.3.2) the Checkpoint. The incoming interface MTU is 1432 and the DF flag is set on the incoming packet (#1573, 1434 bytes). Yet, it looks like it's simply going through (#1575). Is that normal? Isn't the interface MTU = IP MTU?

    (I get an I/O error on uploading to the forum so here is the link to the packet capture screenshot: https://i.imgur.com/YIhVG2J.jpg)

    2) I see you can configure an MTU for a VLAN interface (on the CLI only). This has not changed from R7720 to R8810, but anyhow:
    a) What does it mean when the underlying physical interface has an MTU set, but the VLAN interface has not? I suppose it is being inherited?
    b) How would you remove the MTU from the VLAN interface (there is no "delete" command for that)?

  12. #12
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: our R7720 and R8810 behaves differently cocnerning fragmentation

    Quote Originally Posted by cciesec2006 View Post
    How is that going to help with UDP traffics?
    Udp is stateless there for anything like this would require app layer to handle it.
    Last edited by jflemingeds; 2018-02-04 at 15:24.

  13. #13
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: packets are always fragmented by firewall!

    Quote Originally Posted by jeronimo View Post
    There are two things I don't understand for now:

    1) Look at the following packet capture which consists of two simultaneaous remote mirrors, one in front (1.2.3.1) and one behind (1.2.3.2) the Checkpoint. The incoming interface MTU is 1432 and the DF flag is set on the incoming packet (#1573, 1434 bytes). Yet, it looks like it's simply going through (#1575). Is that normal? Isn't the interface MTU = IP MTU?

    (I get an I/O error on uploading to the forum so here is the link to the packet capture screenshot: https://i.imgur.com/YIhVG2J.jpg)

    2) I see you can configure an MTU for a VLAN interface (on the CLI only). This has not changed from R7720 to R8810, but anyhow:
    a) What does it mean when the underlying physical interface has an MTU set, but the VLAN interface has not? I suppose it is being inherited?
    b) How would you remove the MTU from the VLAN interface (there is no "delete" command for that)?
    Hmmm I canít seem to see the capture. Itís too blurry. Btw Ethernet has 18 bytes of overhead so max frame size will be 1518 with mtu of 1500 bytes.

    I would assume itís inherrited as well. I mean a vlan tag is just a few bytes on the Ethernet header so itís either going to be accepted into the nic based on size or not.

    So to circle around back to the original issue. Are you sure the old r77.30 box didnít have some fwkern.conf options set for this? I would be surprised if r77.30was doing mss clamping out of the box.

  14. #14
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    Here's a magnified screenshot... https://i.imgur.com/MURK95J.jpg

  15. #15
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    Ah, the basic uploader works.
    Click image for larger version. 

Name:	cap1.jpg 
Views:	44 
Size:	166.7 KB 
ID:	1364

  16. #16
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: packets are always fragmented by firewall!

    This site is really is a miserable experience on mobile. Iíll check it out from the airport if LA traffic doesnít curse me.

  17. #17
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    Quote Originally Posted by jflemingeds View Post
    Iíll check it out from the airport
    No worries :D

    We're currently running R77 again (it's 7720 not 30) and I see MSSs of 1460 all over the place, so I don't think there is any clamping there, except for the remote network sending some (I know they internally have some hosts with issues)

    I've looked at fwkern.conf, nothing related either.

    Tomorrow we will know more. I have prepared several scenarios to test with old and new software. This is going to be "interesting".

  18. #18
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,632
    Rep Power
    9

    Default Re: MTU issues: packets are always fragmented by firewall!

    Quote Originally Posted by jeronimo View Post
    No worries :D

    We're currently running R77 again (it's 7720 not 30) and I see MSSs of 1460 all over the place, so I don't think there is any clamping there, except for the remote network sending some (I know they internally have some hosts with issues)

    I've looked at fwkern.conf, nothing related either.

    Tomorrow we will know more. I have prepared several scenarios to test with old and new software. This is going to be "interesting".
    Ok well Iím checking out. Heading over to get settled for the game.

    If you can can you explain what thyou core problem is again?

  19. #19
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: MTU issues: packets are always fragmented by firewall!

    The core issue is that when we go from R7720 to R8010 (from old to new appliances with imported configuration), things break related to the interface with lesser MTU.

    Having rolled back to R7720 and starting to analyze the previous setup, I notice that packets coming in and using MTU 1434 seem to go through, although the interface MTU is 1432 on the incoming interface.

    Well, the VLAN MTU is not set specifically (that's why I asked that question) but "ip addr" seems to confirm that the MTU for the VLAN interface is indeed 1432.

    Code:
    set interface eth5 link-speed 1000M/full
    set interface eth5 state on
    set interface eth5 auto-negotiation on
    set interface eth5 mtu 1432
    set interface eth5.213 state on
    set interface eth5.213 ipv4-address xxx mask-length 29
    
    50: eth5.213@eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1432 qdisc noqueue
    In any case, I have confirmed that the remote side have their MTU on 1434, so there is a mismatch which is not good.

    So probably the issue is that R8810 behaves correctly and drops these (to be confirmed) and I'd like to understand why R7720 wasn't behaving that way.

  20. #20
    Join Date
    2006-09-26
    Posts
    3,164
    Rep Power
    16

    Default Re: MTU issues: packets are always fragmented by firewall!

    Quote Originally Posted by jeronimo View Post
    The core issue is that when we go from R7720 to R8010 (from old to new appliances with imported configuration), things break related to the interface with lesser MTU.

    Having rolled back to R7720 and starting to analyze the previous setup, I notice that packets coming in and using MTU 1434 seem to go through, although the interface MTU is 1432 on the incoming interface.

    Well, the VLAN MTU is not set specifically (that's why I asked that question) but "ip addr" seems to confirm that the MTU for the VLAN interface is indeed 1432.

    Code:
    set interface eth5 link-speed 1000M/full
    set interface eth5 state on
    set interface eth5 auto-negotiation on
    set interface eth5 mtu 1432
    set interface eth5.213 state on
    set interface eth5.213 ipv4-address xxx mask-length 29
    
    50: eth5.213@eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1432 qdisc noqueue
    In any case, I have confirmed that the remote side have their MTU on 1434, so there is a mismatch which is not good.

    So probably the issue is that R8810 behaves correctly and drops these (to be confirmed) and I'd like to understand why R7720 wasn't behaving that way.
    Did you install Jumbo hotfix 56? Did it make any differences?

Page 1 of 2 12 LastLast

Similar Threads

  1. H.323 Issues over VPN - Lots of out of state packets
    By DZelenak in forum Firewall Blade
    Replies: 3
    Last Post: 2014-09-15, 12:50
  2. Firewall dropping EDNS packets, smartdefense is turned off
    By B A Booracus in forum Content Security/Security Servers/CVP/UFP
    Replies: 1
    Last Post: 2010-07-22, 16:17
  3. Firewall Intermittently Drops packets
    By tdvit in forum SmartDashboard
    Replies: 4
    Last Post: 2008-09-11, 12:54
  4. Nokia Firewall performance on small packets
    By tohhwee72 in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 6
    Last Post: 2008-04-30, 07:39
  5. Video streaming packets through the firewall
    By hono222 in forum Versions Of Firewall-1/VPN-1
    Replies: 0
    Last Post: 2007-01-12, 19:40

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •