CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 18 of 18

Thread: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

  1. #1
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    We're setting up a new 1490 on R77.20.60 being managed by an R77.30 SmartCenter. The Add On is installed as well as the hotfix to manage the 1400 series. I just got off a three-hour three-person CP support call and they verified everything was setup correctly and to quote the Office Space movie: "Man, you are seriously screwed up."

    In no particular order:

    • When SIC is being configured, if you try to initialize it from the 1490 using the "Connect to SmartCenter button", it fails. The firewall object on the SmartCenter "Test SIC" button says "The peer sent the wrong DN" and it has the IP address of an internal interface on the 1490. But it's lying. All you have to do is Initialize SIC from the SmartCenter object using "Initialize now". Not a big deal once you know about it, though.


    • Logs are sent from the 1490 to the SmartCenter's NAT'd IP perfectly. Life is good. Until you bring up the site-to-site VPN using a Community. The VPN works perfectly but all logging stops.

      This took a while to isolate down and it was only when I decided to reboot the 1490 that I realized it sent logs until the VPN came up.


    • It is not possible to SSH or Web GUI (TCP 4434) to the 1490 across the Internet if the VPN is down unless you do an "fw unloadlocal". After the security policy is installed, both SSH and the Web GUI function but ONLY after the site-to-site VPN comes up. (It does work normally from the appliance itself. Yes, we have it IP-restricted and entered in the entire /24 we have as allowed IP addresses.)

      Sounds like an easy to diagnose policy problem, right? Sadly, because the VPN is down I do get logs and they show the traffic is accepted by the 1490 but it just does not work. CP ran packet captures that confirmed the 1490 is getting the traffic. No, adding the services to Excluded Services does precisely nothing.


    • When the VPN is up, Test SIC shows "Communicating" and SmartView Monitor shows normal.

      But take the VPN down and Test SIC now shows "Can't communicate on TCP 18191" (CPD) and SmartView Monitor shows the 1490 is down.

      But it's not. You can remotely install a policy on the 1490 which should not be possible if SIC is down. And when the VPN comes back up, all is normal now.

      So remove the 1490 from the VPN community, push the policy to both gateways and down it all goes including allegedly SIC. Can't SSH to it, can't web via TCP 4434 to it, nothing. (You do get logs, though!) But add the 1490 back in and install the policy on both gateways and Voila! It's all back.
      Except for the logs, of course.


    • We created a bridge between the DMZ port and the LAN Switch. Why? Because we need the fiber connection as well as the copper connections for this project. The bridge is assigned the IP used for the default gateway.

      From across the VPN you can ping the copper devices on LAN1 and the DMZ devices. But the LAN1 and DMZ devices cannot ping each other even though it's a bridge.

      We need to run DHCP on the LAN1 port and have it passed to the DMZ port but it doesn't get there.

      But do an "fw unloadlocal" and it all works.

      The policy has like a dozen rules; that's it. And they are all pretty much wide open for testing. If I could only get logs while the VPN was up so I could see what is going on...


    • When you try to install a QoS policy on the 1490, any policy, it errors out with several messages that a dynamically linked thingy is calling a statically linked library.


    It's the darndest thing. It's like the control connections that are supposed to run outside the VPN using implied rules are actually getting routed through the VPN. Yet you can establish SIC without the VPN, obviously.

    But logs are the opposite. When the VPN is up, there is zero TCP 257 traffic arriving at the gateway protecting the SmartCenter. A packet capture on the 1490 shows it is trying to connect to the SmartCenter using its private IP address. Yes, the SmartCenter IP is in the VPN Domain of the main gateway.

    I've done a lot of these over the years (but not with the small appliances) and have never had these issues. Even the CP people are baffled by what they are seeing.

    I'm about to reset the 1490 to the factory firmware and configure it up and see what happens. There are so many whacky seemingly unrelated things wrong it feels like a problem with the new firmware or how the SmartCenter is formatting the policy.

    Thanks for reading this. Any WAG's are appreciated.

    Ray
    Last edited by RayPesek; 2017-09-19 at 22:24.

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,226
    Rep Power
    13

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    How many gateways is the R77.30 SMS managing? Is this 1490 now the second gateway being managed? If so you may have triggered what I call the "NAT Bomb" if you have left the Install on Gateway set to Any on the NAT tab of all objects with Automatic NAT configured. If this situation is present the new firewall will try to start doing all the NATs configured for the original gateway upon policy installation and cause all kinds of seriously messed up routing problems.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2007-06-04
    Posts
    3,268
    Rep Power
    16

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    What you want to do is use the crypt.def to exclude encrypting traffic to the External IP of the 1490 so that the Local Gateway doesn't recognise that the External IP is part of the VPN.

    When you have a Management Server behind a Gateway then build a VPN between the Gateway that managing and the Gateway that the Management Server is located behind then due to the fact that Check Point automatically adds the external IP of a Gateway to the Enc Domain of the Gateways then ( presuming that the source is in the Gateways Enc Domain ) will attempt to pass the traffic over the VPN.

    Also ensure that the Management Server IP is NOT in the Enc Domain of the Local Gateway.

    Trick I have found is either

    A) use the crypt.def and say don't encrypt traffic too the Remote Gateway so that the local gateway doesn't try and encrypt traffic too or from Remote Gateway
    B) use a different gateway to connect through to Remote Gateway for the Management Connectivity then what you use for the VPN.

  4. #4
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by ShadowPeak.com View Post
    How many gateways is the R77.30 SMS managing? Is this 1490 now the second gateway being managed? If so you may have triggered what I call the "NAT Bomb" if you have left the Install on Gateway set to Any on the NAT tab of all objects with Automatic NAT configured. If this situation is present the new firewall will try to start doing all the NATs configured for the original gateway upon policy installation and cause all kinds of seriously messed up routing problems.

    It's the ninth firewall. I always explicitly specify the Install On target to avoid ugly surprises. Good thought, though.

    Ray

  5. #5
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    So we made some significant progress today. Yes, the SmartCenter is in the VPN domain of the gateway it's behind and needs to be because we use a site-to-site VPN to ship scripted "migrate export" backups off-site.

    Since R77.20.60 is fairly new, I decided to reset the 1490 to the factory defaults, which took it back to R77.20.31 from July 2016 or so. Yeah, you guessed it. Logging now works perfectly whether the VPN is up or down. The mess with SIC seemingly being affected by the VPN is also gone.

    My support case with CP essentially responded with "Great job. By the way, we recommend you only upgrade the 1400 series using the USB drive method and reconfigure them from scratch." Uh huh. Because that is so easy to do on a device hundreds of miles away... They did say that R77.20.51 is stable so we're going to try a non-USB in-place upgrade tomorrow.

    And now that I have logs, I see that the reason we cannot manage it over the Internet using the web GUI is because of the IPS SSL Tunneling protection. While we had an exclusion for TCP 4434 in place on the firewall I'm behind, the exclusions was written narrowly (by me) and did not get installed on to the 1490. Problem solved.

    No, I do not know why SSH used to act up before but it's OK now. It's all very odd because if the firmware was that bad it would have been yanked by now. The CP guy alluded to in-place upgrades going bad occasionally, which is what we did when we went to R77.20.60

    I'm going to try the QoS policy tomorrow and work on the LAN - DMZ bridge issue as well.

    One problem I'm seeing is that it appears that the centrally managed DMZ interface is being treated differently somehow than the LAN ports. By that I mean I look explicitly at either the DMZ interface in the logs or any other way and some traffic I generate to or from the DMZ simply does not show up in the logs.

    Or when I try to map a drive from the DMZ port to the LAN port, the logs show the traffic is accepted but the server I'm mapping to never responds. But do an "fw unloadlocal" and it all works. There literally are a dozen rules and the only blocking rules are the stealth rule and the cleanup rule.

    Another example is that it is not possible to manage the 1490 from the DMZ interface. You can do it from the LAN interfaces or the WAN interface but not from the DMZ interface. So much for a management network using the DMZ interface.

    Ray

  6. #6
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Update on the QoS issue. These are the errors I see when trying to install a QoS policy on the 1490 with either the R77.20.31 or R77.20.60 firmware. Yes, the Add On is installed and the hotfix to be able to select and manage the 1400 series is installed on the SmartCenter.

    Installation Targets Version Policy Type Details

    fw R77.20 QoS WARNING: SharedLibLoad(libverify_com.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(/opt/CPsuite-R77/fw1/lib/libCPInstMgrConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrIpsConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrConnectraR65ConvComps.so ): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrEdgeConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrSfwConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrEchoConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrIps1ConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrVsxConvComps.so): called from statically-linked code!
    fw R77.20 QoS Authentication error [ SIC error no. 147 ] check that peer SIC is configured properly and that system date and time on the Security Management Server and peer are synchronized (IP = xxx.xxx.xxx.xxx)(port = 18191).

    But SIC does work because I'm getting logs and can install policies.

    I remember reading that you could centrally manage a 1400 series by selecting the 1100 series hardware on the firewall object but you lost some capabilities. So I reset the hardware type from 1470/1490 Wired to 1100

    Yup, you guessed it. The QoS errors are gone and the QoS policy installed perfectly.

    Ray

  7. #7
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,625
    Rep Power
    9

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by RayPesek View Post
    Update on the QoS issue. These are the errors I see when trying to install a QoS policy on the 1490 with either the R77.20.31 or R77.20.60 firmware. Yes, the Add On is installed and the hotfix to be able to select and manage the 1400 series is installed on the SmartCenter.

    Installation Targets Version Policy Type Details

    fw R77.20 QoS WARNING: SharedLibLoad(libverify_com.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(/opt/CPsuite-R77/fw1/lib/libCPInstMgrConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrIpsConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrConnectraR65ConvComps.so ): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrEdgeConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrSfwConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrEchoConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrIps1ConvComps.so): called from statically-linked code!
    fw R77.20 QoS WARNING: SharedLibLoad(libCPInstMgrVsxConvComps.so): called from statically-linked code!
    fw R77.20 QoS Authentication error [ SIC error no. 147 ] check that peer SIC is configured properly and that system date and time on the Security Management Server and peer are synchronized (IP = xxx.xxx.xxx.xxx)(port = 18191).

    But SIC does work because I'm getting logs and can install policies.

    I remember reading that you could centrally manage a 1400 series by selecting the 1100 series hardware on the firewall object but you lost some capabilities. So I reset the hardware type from 1470/1490 Wired to 1100

    Yup, you guessed it. The QoS errors are gone and the QoS policy installed perfectly.

    Ray
    You need to talk to support about this. This is a programming issue.

    Windows has DLLs Unix has Shared Objects (.so).

    Basically the same thing.

    When you make a executable you can have the .so files included into the executable (this is static) or you can say use the OS to figure out where the shared objects are at execution time. This makes the exec smaller and means you can change the shared objects (in some cases) and not have to recompile the executable.

  8. #8
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Yup, I already added the error and the "resolution" to the ticket. The SmartCenter is on Gaia.

    Ray

  9. #9
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Today we successfully did an in-place upgrade to R77.20.51. Everything just worked immediately.

    We then did an in-place upgrade to R77.20.60 and it went bad. Upon the reboot the VPN failed to come up, the default policy was installed and clicking "Connect to SmartCenter" failed repeatedly. We did a Revert to Previous Firmware and everything started working again.

    Well, at least it's repeatable. :-)

    Ray

  10. #10
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    So I have a new one that is just kicking me to the curb.

    To recap, a main gateway which we'll call R77.30 Take 216 and a 1490 running R77.20.51, both managed by the same SmartCenter.

    There is just one /24 behind the 1490 and life was good until a new switch was added to it. The network team uses /32's on a loopback address to send logs and for access controls. In effect they are adding a new subnet that consists of just one IP address. I added a static route on the 1490 to send the traffic via the switch's IP address. That part actually works fine. The network team can SSH to the switch on the /24 and get to the /32 loopback address from the switch itself.

    I saw that the /32 loopback they are using could not communicate over the VPN to send logs to the R77.30 and it could not be pinged from behind the R77.30 gateway. Makes sense because that /32 was not part of the VPN domain of the 1490 or of the R77.30.

    So I added the /32 into the VPN Domain group of both firewalls and added access rules. When the /32 tries to send to the R77.30 it tries to bring up the VPN. That's good except it fails for No Valid SA, so it's not even a one-way VPN.

    But when we try to connect to the /32 from behind the R77.30, ping it, SSH to it, whatever, the traffic to the /32 just routes to the Internet and is never encrypted. I added the /32 destination into other rules that do work in the Community ( to the /24).

    Nothing I have tried fixes the problem that the R77.30 is not sending traffic to an IP address that is in the 1490's VPN Domain group. It's acting like the R77.30 is not seeing the added /32 IP address.

    When I do a Get Interfaces With Topology on the 1490, it sees the new /32. If I set the 1490 VPN Domain to use "based on topology table" instead of the group, it still fails.

    The /32 is totally unique to the company; it does not exist anywhere else in the SmartCenter.

    Any guesses are appreciated.

    Ray

  11. #11
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,226
    Rep Power
    13

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by RayPesek View Post
    So I have a new one that is just kicking me to the curb.

    To recap, a main gateway which we'll call R77.30 Take 216 and a 1490 running R77.20.51, both managed by the same SmartCenter.

    There is just one /24 behind the 1490 and life was good until a new switch was added to it. The network team uses /32's on a loopback address to send logs and for access controls. In effect they are adding a new subnet that consists of just one IP address. I added a static route on the 1490 to send the traffic via the switch's IP address. That part actually works fine. The network team can SSH to the switch on the /24 and get to the /32 loopback address from the switch itself.

    I saw that the /32 loopback they are using could not communicate over the VPN to send logs to the R77.30 and it could not be pinged from behind the R77.30 gateway. Makes sense because that /32 was not part of the VPN domain of the 1490 or of the R77.30.

    So I added the /32 into the VPN Domain group of both firewalls and added access rules. When the /32 tries to send to the R77.30 it tries to bring up the VPN. That's good except it fails for No Valid SA, so it's not even a one-way VPN.

    But when we try to connect to the /32 from behind the R77.30, ping it, SSH to it, whatever, the traffic to the /32 just routes to the Internet and is never encrypted. I added the /32 destination into other rules that do work in the Community ( to the /24).

    Nothing I have tried fixes the problem that the R77.30 is not sending traffic to an IP address that is in the 1490's VPN Domain group. It's acting like the R77.30 is not seeing the added /32 IP address.

    When I do a Get Interfaces With Topology on the 1490, it sees the new /32. If I set the 1490 VPN Domain to use "based on topology table" instead of the group, it still fails.

    The /32 is totally unique to the company; it does not exist anywhere else in the SmartCenter.

    Any guesses are appreciated.

    Ray
    I'm not sure how a /32 is going to be handled by the "one VPN tunnel per subnet pair" default VPN Tunnel Sharing setting in the community for IKE Phase2, since the /32 is a not technically a subnet. Does setting "one VPN tunnel per each pair of hosts" in the VPN Tunnel Sharing settings of the VPN Community help?
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  12. #12
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Thank you very much for following this thread. Yup, I already tried that with no change. I also defined an entire /24, since it would be unique to us, and put that in the VPN Domain with no change.

    I did find an SK about checking for duplicate objects and the community trying to build a VPN with the wrong one so I'm going to check for that. I literally hit one hundred database revisions in the last two weeks trying to fix the various problems so it's entirely possible.

    Symptom:
    VPN traffic is being sent in clear, when source, destination and community settings all match the community criteria.

    Cause:
    Duplicate objects with the same main IP address will cause the duplicate object to match for the Peer's encryption domain. The gateway will build the community based on the duplicate object and not the peer gateway object due to matching main IP addresses.


    Ray

  13. #13
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,625
    Rep Power
    9

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    So I added the /32 into the VPN Domain group of both firewalls and added access rules.
    Shouldn't this only be added to the encryption domain of the 1490?

  14. #14
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    !@#$%^&*

    Thank you for the "Duh" moment. And for being so polite when you pointed it out. Now I'm going to have to remote in today to see if that is actually how I screwed up. :-)

    Database revision 102 coming up soon...

    Ray

  15. #15
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,226
    Rep Power
    13

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by jflemingeds View Post
    Shouldn't this only be added to the encryption domain of the 1490?
    Haha missed that one, good catch.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  16. #16
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    And database revion #102 did the trick! Thank you VERY much!

    So I only have two issues left and I have tickets on both: The QoS issue and the .60 upgrade issue.

    Ray

  17. #17
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,625
    Rep Power
    9

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by RayPesek View Post
    And database revion #102 did the trick! Thank you VERY much!

    So I only have two issues left and I have tickets on both: The QoS issue and the .60 upgrade issue.

    Ray
    Hurray! its Miller time!

  18. #18
    Join Date
    2006-03-19
    Location
    Northern Ohio
    Posts
    1,386
    Rep Power
    14

    Default Re: Centrally managed 1490 - seriously screwed up control connections and VPN traffic

    Quote Originally Posted by ShadowPeak.com View Post
    Haha missed that one, good catch.
    Yeah, and you weren't the only one. It's been a long two weeks. :-)

    Ray

Similar Threads

  1. One-way VPN between a 1490 and an Open Server? And then no VPN traffic after topo chg
    By RayPesek in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 10
    Last Post: 2017-09-22, 17:35
  2. centrally managed 1100 with R75 from mgmt server with R80 ?
    By Irek_Romaniuk in forum Check Point Series 80/1100 Appliances
    Replies: 3
    Last Post: 2017-06-08, 11:24
  3. Replies: 4
    Last Post: 2013-01-04, 04:40
  4. Locally to Centrally managed deployment
    By netrix in forum Check Point UTM-1 Appliances
    Replies: 2
    Last Post: 2010-08-30, 07:14
  5. how to change from centrally managed?
    By whirrr in forum Check Point UTM-1 Appliances
    Replies: 6
    Last Post: 2008-11-14, 11:21

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •