CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it yet again - That's right, the 3rd edition is here!
You can read his announcement post here.
It's a massive upgrade focusing on current versions, and well worth checking out. -E

 

Page 1 of 3 123 LastLast
Results 1 to 20 of 58

Thread: SecureXL does not work packet fragmentation

  1. #1
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default SecureXL does not work packet fragmentation

    I guess I don't have much experience with SecureXL until it bytes me in ass :-(

    I have a pair of GAIA R75.47 805 licenses ClusterXL H//A. The hardware is Dell R710 PowerEdge with 4 physical CPU with 2 cores and 32GB RAM with Intel NIC and latest BIOS firmware. I am not running anything other than FW blades on the gateway (enabled_blades output only shows fw blade) so no IPS and other stuffs on the gateway. SecureXL is enabled

    I have issue with both Microsoft 2008R2/2012R2 DFS file replication and Oracle RMAN replication traffics traversing the firewall. The rule for these two traffics are in rule #1 on the security policy.

    When traffics for Microsoft DFS (tcp high ports )traverses the firewall at the rate of 200Mbits/sec, one of the CPU cores reaches 100% utilization thus causing issues for other traffics traversing the firewall at the same time. When that happens, snmpd on the gateway becomes non-responsive.

    When traffics for Oracle RMAN (tcp 1521) traverses the firewall at the rate of 600Mbits/sec, one of the CPU cores reaches 100% utilization thus causing issues for other traffics traversing the firewall at the same time. When that happens, snmpd on the gateway becomes non-responsive.

    I've also disabled protocol inspection under both of the services (set to none) but it didn't do any good

    In both situations, when I perform "fwaccel stats -s" I could see that the traffics are not being accelerated by SecureXL even though "fwaccel stat" shows that SecureXL gets disabled at rule number #300 in the rulebase and yet this traffics are not being accelerated and they have to go through Slow Path, confirmed with "fwaccel stats -s" and "fwaccel conns | grep x.x.x.x"

    Further investigation via "fw monitor" and "tcpdump" revealed that both Microsoft DFS and Oracle sends out large packet size (in the range of 6000). Because of that, the packet gets fragmented to conform with Ethernet MTU 1500. Therefore, in checkpoint's world, SecureXL can NOT accelerated traffics. In other words, if the packet is higher than 1500, SecureXL will NOT work (according to the TAC engineer who works with me).

    In today's environment, there is NO application that I am aware of send data less than 1500 packet size.

    My microsoft guy tells me that it is NOT possible to set the packet size to less than 1500 in the application itself. I might be able to do this with Oracle but not with Microsoft DFS

    The point here is that checkpoint SecureXL is not as good as advertised and sometime the Checkpoint SE who promises you the moon has no clue about these things.

    In other words, I am completely F!!!! :-(

    Oh by the way, this is NOT an issue with Cisco ASA, go figure. For whatever reason, ASA does not seem to care about packet fragmentation.
    Last edited by cciesec2006; 2016-05-17 at 11:28.

  2. #2
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,658
    Rep Power
    10

    Default Re: SecureXL does not work packet fragmentation

    Are we talking tcp or udp? Also how are you making the determination the packet is fragmented?

  3. #3
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Unless you have a SAM card on a 21000 appliance, fragmented packets are ineligible for processing by SecureXL and will go Firewall Path. This is covered in my book in a section called "IP Fragmentation: Nemesis of SecureXL". You can confirm with these commands:

    fwaccel stats -p (provides statistics for SecureXL violations, one of which is going F2F due to fragmentation)

    tcpdump -c 100 -eni any "ip[6:2] & 0x3fff != 0x0000" (This shows all fragmented packets traversing the firewall)

    If the fragmented packets are showing up at the firewall due to inconsistent MTU settings in the internal network (or inconsistent deployment of Jumbo frames), there is not much you can do about it on the firewall other than forbidding fragments completely in the IPS settings (careful!). It is a limitation of SecureXL.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  4. #4
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    Unless you have a SAM card on a 21000 appliance, fragmented packets are ineligible for processing by SecureXL and will go Firewall Path. This is covered in my book in a section called "IP Fragmentation: Nemesis of SecureXL". You can confirm with these commands:

    fwaccel stats -p (provides statistics for SecureXL violations, one of which is going F2F due to fragmentation)

    tcpdump -c 100 -eni any "ip[6:2] & 0x3fff != 0x0000" (This shows all fragmented packets traversing the firewall)

    If the fragmented packets are showing up at the firewall due to inconsistent MTU settings in the internal network (or inconsistent deployment of Jumbo frames), there is not much you can do about it on the firewall other than forbidding fragments completely in the IPS settings (careful!). It is a limitation of SecureXL.
    both are TCP. I confirm packet fragmentation via tcpdump and confirmed by the TAC engineer.

    Jumbo frames are NOT running on the servers.

    Did I mention that IPS is NOT running on the Firewalls?

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    both are TCP. I confirm packet fragmentation via tcpdump and confirmed by the TAC engineer.

    Jumbo frames are NOT running on the servers.

    Did I mention that IPS is NOT running on the Firewalls?
    You will need to determine why the packets are being fragmented before they reach the firewall. Or are you saying firewall is the one fragmenting the packets? In that case you need to check the MTU settings on the firewall
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  6. #6
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    You will need to determine why the packets are being fragmented before they reach the firewall. Or are you saying firewall is the one fragmenting the packets? In that case you need to check the MTU settings on the firewall
    1- the server A, on on side of the firewall, the MTU is set to 1500,
    2- the firewals MTU is set to 1500
    3- the server B, on the other side of the firewall, the MTU is set to 1500,

    the Oracle RMAN application or Microsoft DFS application, server A is sending data with packet size of 5000. Now in order to comply with the MTU on the Ethernet, the packet will get fragmented to something less than 1460 so that with overhead add on, it can be in compliance with Ethernet MTU 1500.

    Therefore, a 5000 packet size will get chop up to 4 different packets to be re-assembled on the other side by Server B.

    This is where SecureXL is having issue with these packets.

    Both Oracle and Microsoft say that this is a checkpoint SecureXL issue because SecureXL can not inspect fragmented packet and because of that, it has to use the F2F path, thus causing high CPU issue on one of the cores.

    Microsoft and Oracle also says that it is very common for application to send packet size larger than 1500 ALL the times and they don't plan on changing it anytime soon.

    Does that make sense?

  7. #7
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,658
    Rep Power
    10

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    1- the server A, on on side of the firewall, the MTU is set to 1500,
    2- the firewals MTU is set to 1500
    3- the server B, on the other side of the firewall, the MTU is set to 1500,

    the Oracle RMAN application or Microsoft DFS application, server A is sending data with packet size of 5000. Now in order to comply with the MTU on the Ethernet, the packet will get fragmented to something less than 1460 so that with overhead add on, it can be in compliance with Ethernet MTU 1500.

    Therefore, a 5000 packet size will get chop up to 4 different packets to be re-assembled on the other side by Server B.

    This is where SecureXL is having issue with these packets.

    Both Oracle and Microsoft say that this is a checkpoint SecureXL issue because SecureXL can not inspect fragmented packet and because of that, it has to use the F2F path, thus causing high CPU issue on one of the cores.

    Microsoft and Oracle also says that it is very common for application to send packet size larger than 1500 ALL the times and they don't plan on changing it anytime soon.

    Does that make sense?
    What your describing isn't fragmentation but segmentation. The maximum amount of data tcp can accept in 1 packet is defined by the MSS value. This is transmisted on tcp syn packets. When 2 hosts start talking they'll both agree to use the lowest MSS value. Also note MSS is not the same as MTU. MSS will always be lower then MTU.

    This is completely different then IP fragmentation, which is what causes issues with securexl and really the firewall kernel as well since there is only a single table for keeping track of fragments.

  8. #8
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by jflemingeds View Post
    What your describing isn't fragmentation but segmentation. The maximum amount of data tcp can accept in 1 packet is defined by the MSS value. This is transmisted on tcp syn packets. When 2 hosts start talking they'll both agree to use the lowest MSS value. Also note MSS is not the same as MTU. MSS will always be lower then MTU.

    This is completely different then IP fragmentation, which is what causes issues with securexl and really the firewall kernel as well since there is only a single table for keeping track of fragments.
    If this is indeed a MSS issue check out the MSS Clamping function of the firewall which can induce both sides to reduce their segment size (and therefore packet size) to a size that can avoid fragmentation; this is typically used with IPSEC VPNs. Only issue is I don't know if MSS Clamping can be performed inside the accelerated path.
    Last edited by ShadowPeak.com; 2016-04-09 at 17:09. Reason: caps
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  9. #9
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,658
    Rep Power
    10

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    If this is indeed a mss issue check out the mss clamping function of the firewall which can induce both sides to reduce their segment size (and therefore packet size) to a size that can avoid fragmentation; this is typically used with IPSEC VPNs. Only issue is I don't know if mss clamping can be performed inside the accelerated path.
    I doubt this issue is anything more than a performance tuning issue. I was just trying to point out that he is saying he is seeing frags and then describing something else which makes me think the issue has been misdiagnosed and should just be treated as every other high cpu on old hardware/code case.
    Last edited by jflemingeds; 2016-04-09 at 15:08.

  10. #10
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by jflemingeds View Post
    I doubt this issue is anything more than a performance tuning issue. I was just trying to point out that he is saying he is seeing frags and then describing something else which makes me think the issue has been misdiagnosed and should just be treated as every other high cpu on old hardware/code case.
    sorry, I was under the influence of medication so wasn't thinking straight and I was getting the information from a colleague

    Jflemingeds is right. the packet was not fragmented and I was able to see this myself with wireshark with the following filters:

    ip.flags.mf == 1 or ip.frag_offset > 0

    icmp.type==3 and icmp.code==4

    ip[0,9,20:2]==4501:0304||ip[6:2]&3fff

    I looked at the ticket and the TAC engineer said that the packet was fragmented so I took it at what he said which wasn't the case with wireshark.

    going back to my original question, if the packet was not fragmented, how could it not get accelerated by SecureXL?

  11. #11
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    sorry, I was under the influence of medication so wasn't thinking straight and I was getting the information from a colleague

    Jflemingeds is right. the packet was not fragmented and I was able to see this myself with wireshark with the following filters:

    ip.flags.mf == 1 or ip.frag_offset > 0

    icmp.type==3 and icmp.code==4

    ip[0,9,20:2]==4501:0304||ip[6:2]&3fff

    I looked at the ticket and the TAC engineer said that the packet was fragmented so I took it at what he said which wasn't the case with wireshark.

    going back to my original question, if the packet was not fragmented, how could it not get accelerated by SecureXL?
    You could run "enabled_blades" on the firewall and we can try to guess why it didn't get accelerated, or you can find out for sure by following the procedure in the "Which Path is a Certain Connection Using, and Why?" section of my book, pages 221-225. Assume that you want to find out why traffic sourced from 192.168.1.1 going to 129.82.102.32 on port 80 is not being accelerated (use caution here, screwing up the filter and having a runaway debug in the SecureXL driver is very very bad and will almost certainly cause an outage and maybe even a crash):

    sim dbg -f 192.168.1.100,*,129.82.102.32,80,6
    sim dbg list (make sure the filter is correct!)
    Get the system to start passing traffic matching your filter and run this (note the "dead man's switch" nature of the command in the event you screwed up):
    sim dbg -m pkt + pxl + f2f ; sleep 15 ; sim dbg resetall
    sim dbg resetall (Yes run it again manually to be ultra-sure the debug is off!)

    Look in /var/log/messages for the output, as an example you might see:

    [fw4_0];get_conn_flags: ISP redundancy is set on for the connection -> F2F;

    ISP Redundancy was the culprit that caused F2F in this sample case.
    Last edited by ShadowPeak.com; 2016-04-09 at 22:08. Reason: fixed reversed arguments
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  12. #12
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    You could run "enabled_blades"
    [Expert@fw]# enabled_blades
    fw
    [Expert@fw]#

    I already run "fwaccel conns | grep 192.168.6.10" and confirms that it is using the slow path:

    192.168.77.206 12841 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.6.10 1522 192.168.77.207 20284 6 F..A..... 9/3 3/9 1 0
    192.168.77.206 12831 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.77.207 20264 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.6.10 1522 192.168.77.206 12817 6 F..A..... 9/3 3/9 0 0
    192.168.77.207 20289 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.6.10 1522 192.168.77.206 12809 6 F..A..... 9/3 3/9 0 0
    192.168.77.206 12851 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.6.10 1522 192.168.77.207 20292 6 F..A..... 9/3 3/9 1 0
    192.168.77.207 20284 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.77.206 12765 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.77.206 12755 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0

  13. #13
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    [Expert@fw]# enabled_blades
    fw
    [Expert@fw]#

    I already run "fwaccel conns | grep 192.168.6.10" and confirms that it is using the slow path:

    192.168.77.206 12841 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.6.10 1522 192.168.77.207 20284 6 F..A..... 9/3 3/9 1 0
    192.168.77.206 12831 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.77.207 20264 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.6.10 1522 192.168.77.206 12817 6 F..A..... 9/3 3/9 0 0
    192.168.77.207 20289 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.6.10 1522 192.168.77.206 12809 6 F..A..... 9/3 3/9 0 0
    192.168.77.206 12851 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.6.10 1522 192.168.77.207 20292 6 F..A..... 9/3 3/9 1 0
    192.168.77.207 20284 192.168.6.10 1522 6 F..A..... 9/3 3/9 1 0
    192.168.77.206 12765 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    192.168.77.206 12755 192.168.6.10 1522 6 F..A..... 9/3 3/9 0 0
    Yes, but if you want to see the reason why it is using the slow path (Firewall Path), execute the debug sequence in my last message. It will tell you why, not just what blade is causing it.

    Edit: If you provide the output of "fwaccel stats -p" as I requested earlier in the thread I may be able to hazard a guess without the debug results.
    Last edited by ShadowPeak.com; 2016-04-10 at 12:27.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  14. #14
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    Yes, but if you want to see the reason why it is using the slow path (Firewall Path), execute the debug sequence in my last message. It will tell you why, not just what blade is causing it.
    Edit: If you provide the output of "fwaccel stats -p" as I requested earlier in the thread I may be able to hazard a guess without the debug results.
    There is only one blade on the gateway and it is "fw" as confirmed with "enabled_blades".

    Let assume that we know the root cause for both DFS and oracle RMAN traffics not being accelerated, that nature of the traffics, how do I go about fixing this?

  15. #15
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,658
    Rep Power
    10

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    There is only one blade on the gateway and it is "fw" as confirmed with "enabled_blades".

    Let assume that we know the root cause for both DFS and oracle RMAN traffics not being accelerated, that nature of the traffics, how do I go about fixing this?
    That would depend on what caused it.

  16. #16
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,658
    Rep Power
    10

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by jflemingeds View Post
    That would depend on what caused it.
    If you're afraid to do this in production, then recreate your management server and firewall in a VM. Then put a server and client on both sides of the firewall and fire off a netcat on that 1521 port. I highly doubt the sql port has inspection on it. Make sure to make the correct src and dst ip also..

  17. #17
    Join Date
    2013-09-25
    Location
    Bucharest
    Posts
    649
    Rep Power
    7

    Default Re: SecureXL does not work packet fragmentation

    This has became like a mini TV series for me. When can we know the root cause? What episode :)?

  18. #18
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    There is only one blade on the gateway and it is "fw" as confirmed with "enabled_blades".

    Let assume that we know the root cause for both DFS and oracle RMAN traffics not being accelerated, that nature of the traffics, how do I go about fixing this?
    Just because the only enabled blade is "fw" doesn't automatically mean 100% of all traffic can be accelerated; little configuration items can make a big difference. Without the output of "fwaccel stats -p" (that I've asked for now twice) or the results of the sim debug we are just guessing.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  19. #19
    Join Date
    2006-09-26
    Posts
    3,194
    Rep Power
    17

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by ShadowPeak.com View Post
    Just because the only enabled blade is "fw" doesn't automatically mean 100% of all traffic can be accelerated; little configuration items can make a big difference. Without the output of "fwaccel stats -p" (that I've asked for now twice) or the results of the sim debug we are just guessing.
    F2F packets:
    --------------
    Violation Packets Violation Packets
    -------------------- --------------- -------------------- ---------------
    pkt is a fragment 712 pkt has IP options 4266
    ICMP miss conn 277769 TCP-SYN miss conn 1737069
    TCP-other miss conn 179707 UDP miss conn 16020186
    other miss conn 88543 VPN returned F2F 0
    ICMP conn is F2Fed 0 TCP conn is F2Fed 3572139
    UDP conn is F2Fed 681124 other conn is F2Fed 0
    uni-directional viol 0 possible spoof viol 0
    TCP state viol 167087 out if not def/accl 0
    bridge, src=dst 0 routing decision err 91
    sanity checks failed 0 temp conn expired 194
    fwd to non-pivot 0 broadcast/multicast 0
    cluster message 0 partial conn 0
    PXL returned F2F 5181 cluster forward 0
    chain forwarding 0 general reason 0

  20. #20
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    14

    Default Re: SecureXL does not work packet fragmentation

    Quote Originally Posted by cciesec2006 View Post
    F2F packets:
    --------------
    Violation Packets Violation Packets
    -------------------- --------------- -------------------- ---------------
    pkt is a fragment 712 pkt has IP options 4266
    ICMP miss conn 277769 TCP-SYN miss conn 1737069
    TCP-other miss conn 179707 UDP miss conn 16020186
    other miss conn 88543 VPN returned F2F 0
    ICMP conn is F2Fed 0 TCP conn is F2Fed 3572139
    UDP conn is F2Fed 681124 other conn is F2Fed 0
    uni-directional viol 0 possible spoof viol 0
    TCP state viol 167087 out if not def/accl 0
    bridge, src=dst 0 routing decision err 91
    sanity checks failed 0 temp conn expired 194
    fwd to non-pivot 0 broadcast/multicast 0
    cluster message 0 partial conn 0
    PXL returned F2F 5181 cluster forward 0
    chain forwarding 0 general reason 0
    Those violation numbers don't seem high enough (assuming you are passing lots of traffic) to correlate to a large amount of traffic not being accelerated. We will need the sim debug.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

Page 1 of 3 123 LastLast

Similar Threads

  1. SecureXL problem: TCP packet out of state: First packet isnīt SYN tcp_flags: PUSH-ACK
    By mhernandez in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 4
    Last Post: 2014-09-18, 05:37
  2. VPN Fragmentation and MTU anomalies
    By dp123 in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 5
    Last Post: 2011-04-26, 17:14
  3. Panic When SecureXL and NAT Are Used and a Malformed Packet Is Received
    By PhoneBoy in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 5
    Last Post: 2010-04-20, 03:11
  4. Replies: 1
    Last Post: 2007-10-02, 11:41
  5. TCP Packet out of state / unknown established TCP packet
    By roadrunner in forum Miscellaneous
    Replies: 0
    Last Post: 2005-08-13, 15:19

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •