CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 5 of 5

Thread: GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install ?

  1. #1
    Join Date
    2014-07-21
    Posts
    57
    Rep Power
    6

    Default GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install ?

    Hi,

    we are running different Gateways as ClusterXL and Standalone with GAiA R77.10 and HFA Take 131.

    ~8 weeks ago we upgrades from HFA Take 41 to HFA Take 131 on two 12200 Appliances with GAiA R77.10. The upgrade worked without any visible issues. Around 3 weeks after the upgrade we encountered problems and lost connections (tcp 1419 - specific application port) and ssh (tcp/22) when doing the Policy Install on both appliances. After rebooting both of the appliances all went OK. But after another ~3 weeks we had the same problem on both machines again.

    So we rebooted one of the appliances and this solved the problem again. The other appliance is still running and has the problem (we did not reboot it). Today we investigated the issue with checkpoint TAC.

    The first step was to disable SecureXL and the monitor the traffic when "rematch all connections" was on, "keep connections open" and with "keep connections open" for specif services. No matter what we configured it worked. So we assumed this was "black magic" from checkpoint TAC and finished the remote session. Unfortunately some minutes after the session I recognized that we still have SecureXL disabled.

    After I enabled SecureXL again the problem was there again, too. (Rematch all connections enabled and keep connections open unchecked for this specific service).
    So the next I did was checking "keep connections open" for the specific service and the problem was solved.

    So I assume that it has something to do with SecureXL. For all the different test - we did 6 different tests - we did packet captues with fw monitor and tcpdump so I hope that checkpoint TAC can find a solution.

    Is there someone else who encountered problems after upgrading to HFA Take 131 when using SecureXL ?
    Any ideas how to debug this ?

    Thanks for your help in advance!

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install

    A few questions:

    - When you say "lost connections" after a policy install, does the connection just hang (implying a drop) or are the connection(s) killed with a TCP RST similar to a Reject action? Do you see any "TCP out of state" log entries for 22/1419 connections?

    - Which version of the SSH service is being matched in the rule base (ssh vs ssh_version_2)? What is the Protocol Type setting (under the Advanced button) for the referenced version of SSH and the custom TCP/1419 service? None/Blank?

    - Do all currently open connections using these two services get killed when it happens or do only some of them die?

    - Are the security policy rules allowing this SSH/1419 traffic subject to SecureXL templating? Use "fwaccel stat" to check.

    - Do you have "Enable drop optimization" set on your firewall object?

    Upon policy load the SecureXL-level connections table is flushed and built back up as the pending traffic is rematched against the new security policy in the Firewall Path (where another copy of the connections table is kept), unless of course "Keep connections open after Policy has been installed" is set. Sounds like the connections using these services aren't making it back into the SecureXL connections table for some reason.
    Last edited by ShadowPeak.com; 2015-08-04 at 10:27.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2014-07-21
    Posts
    57
    Rep Power
    6

    Default Re: GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install

    Hi shadowpeak,

    thank your for your feedback. I appreciate it!

    - It looks more like hanging. The colleagues are connected vi SSH and they then tell me that it hangs and then after some time they need to reconnect. The other type of traffic tcp/1419 comes from an application. Not sure how it exactly behaves there but in the ende the application tells that the other server could not be reached and logs some application specific warnings. After that both ssh and the 1419 traffic can be established again very fast. So just a short "interrupt".

    - We did not focus on the ssh session - we more focused on the 1419/tcp application traffic which is not ssh (just to make that clear). The 1419 traffic is some other kind of traffic. When going to advanced the "protocol type " is empty for ssh/22 and tcp/1419.

    - 1419 traffic is only between two servers as far as I can see that so no other systems using the same service (on this gateway). ssh sessions will only be used for administration so we encountered this only when we did the different tests. Not sure if other ssh session to other servers over the same gateway will lose their connection.

    - Yes, these rules are covered by the accept templates.

    - Yes, drop templates are enabled. Same to NAT and of course accept templates.


    Your conclusion sounds good and I think describes what is happening. But the interesting thing is why this only occures only after some weeks after rebooting the gateway. So we have another gateway which covers the same kind of traffic but different servers and as I said in my opening posting - this gateway had the same problem, we rebooted this gateway and it was solved. Don't know if there are any mechanics, caches and so on which run out of space after some time and then the problem occurs.

    PS: CheckPoint support answered that CPU and RAM should not be the reason for that because it is quite low.

    Thank you for your help!

  4. #4
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install

    Quote Originally Posted by Nachtfalke View Post
    Hi shadowpeak,

    thank your for your feedback. I appreciate it!

    - It looks more like hanging. The colleagues are connected vi SSH and they then tell me that it hangs and then after some time they need to reconnect. The other type of traffic tcp/1419 comes from an application. Not sure how it exactly behaves there but in the ende the application tells that the other server could not be reached and logs some application specific warnings. After that both ssh and the 1419 traffic can be established again very fast. So just a short "interrupt".

    - We did not focus on the ssh session - we more focused on the 1419/tcp application traffic which is not ssh (just to make that clear). The 1419 traffic is some other kind of traffic. When going to advanced the "protocol type " is empty for ssh/22 and tcp/1419.
    Hmm, would be interesting to know if other connection types are affected as well. I don't see how just 22/1419 could be affected, perhaps these are the only ones you are noticing? It is also possible that your 1419 application is not closing or otherwise handling its TCP connections properly, is it a well-known application or some kind of custom, home-rolled app?

    - 1419 traffic is only between two servers as far as I can see that so no other systems using the same service (on this gateway). ssh sessions will only be used for administration so we encountered this only when we did the different tests. Not sure if other ssh session to other servers over the same gateway will lose their connection.

    - Yes, these rules are covered by the accept templates.
    Good to know but really shouldn't matter since this is an existing connection, I don't think templating comes into play during a policy installation/rematch but obviously templating is stopped as well when you disable SecureXL. If you are seeing templating being halted at let's say rule #408 in "fwaccel stats" output, it might be interesting to try moving the 22/1419 rules somewhere beneath rule #408 to poison the templating of these rules.

    - Yes, drop templates are enabled. Same to NAT and of course accept templates.
    Having NAT templates enabled is a bit unusual but unlikely to be responsible, are these problematic 22/1419 connections NATed? NAT Templates are implemented inside SecureXL after all and go away when SecureXL is disabled...

    However in regard to Drop Templates, allow me to copy/paste an ominous passage from page 252 of my book:

    There were numerous Drop Optimization bugfixes in the R77.20 release involving VPNs, DLP, DHCP, and a host of other features. If your firewall is running R77.10 or earlier, be ready to turn Drop Optimization back off quickly if problems ensue.
    You are running the jumbo hotfix for R77.10 which *should* have all the fixes for SecureXL Drop Templates that were added in R77.20. But this is a big red flag for me. My suggestions moving forward in order would be:

    1) Disable Optimized drops and install policy twice (first one to get drop templates disabled, second one to see if problems are still encountered).
    2) Poison the SecureXL templating of 22/1419 rules.
    3) Try disabling NAT Templates if the problematic 22/1419 traffic is NATed. (Especially if non-NATted 22/1419 traffic seems fine)
    4) Next step would be disabling SecureXL Throughput Acceleration on individual interfaces carrying the problematic 22/1419 traffic with the "sim nonaccel" command but that is starting to get off in the weeds.

    The fact that this only manifests itself after a period of time would seem to imply some kind of bug or resource issue. If you can rule out specific functions of SecureXL as the culprit that would be helpful, the individual SecureXL functions we need to rule out as a cause are:

    - Drop Templates
    - Accept Templates
    - NAT Templates
    - Throughput/Packet Acceleration

    All of these go away when SecureXL is disabled.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  5. #5
    Join Date
    2014-07-21
    Posts
    57
    Rep Power
    6

    Default Re: GAiA R77.10 + HFA Take 131 + SecureXL on - Losing connection while Policy Install

    Hi again,

    not really many news on this topic because I was involved in other things the last week. But some thing we found out:

    It is not only traffic on port 22/ssh and 1419 but it is for other traffic on higher ports like tcp/20800-20900
    And we found out that UDP sessions are affected, too. We have a router sending logs to a syslog server using default syslog (udp/514). We encountered problems - not every policy installation - that afer a policy installation there were no logs delivered to the syslog-server behind the firewall. So it looks like something was still blocking this traffic.

    So I can not say for sure it is "blocking" the traffic because I did not monitor it on the firewall but I can see that there is a gap on the syslog server for around 8 hours.

    And checkpoint did not find anything on the packet captures but I think this was to expect because if it is really related to SecureXL then packet captures will probably not be the best tool to debug. So checkpoint asked for another remote session to debug SecureXL more in detail.

    What I further did is placing service "traceroute" on top of my rulebase to disable all the templates. And I disabled drop templates but .... I did not any further tests to see if this helps ;-)

    Will post here if I have any news on that.

Similar Threads

  1. SecureXL: Connection templates are not possible for the installed policy.
    By arnolde in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 5
    Last Post: 2012-02-09, 05:02
  2. Gateway connection lost during policy install
    By quartino in forum SmartDashboard
    Replies: 3
    Last Post: 2010-08-02, 02:18
  3. Replies: 6
    Last Post: 2010-05-09, 12:05
  4. Connection to firewall drops on policy install
    By trifid1967 in forum Miscellaneous
    Replies: 0
    Last Post: 2006-03-09, 02:34
  5. Losing Connection with Management Console
    By Barry J. Stiefel in forum Miscellaneous
    Replies: 0
    Last Post: 2005-08-13, 23:28

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •