ISP Redundancy - does Load Sharing really work?

[v33dubya]

Hi All,

I have a customer who has two Internet links and I am doing some testing in my lab to see if I can get Checkpoint to properly perform load sharing on two WAN links before I recommend doing it.

In this environment, the client has two Cisco routers terminating an Internet link each. The public IP address for each Internet link is on the WAN interface of each Cisco router. Then there is a /30 transit network from each Cisco router to a dedicated External/WAN interface on the Checkpoint gateway. This means there are two external interfaces on the gateway with each connected to a different router.

So the Checkpoint box has two External interfaces, though each of these are assigned private IP addresses and connect to a Cisco router. The Cisco routers perform Dynamic NAT translations so any outbound connections are NAT'd to the public IP of the WAN interface on each router. This makes sure that any returning traffic comes back via the same WAN link. The Checkpoint box will not perform any NAT, that will only be performed at the network edge by the routers.

I have set this up in the lab and have a couple of hosts inside the Checkpoint firewall. I have done some basic testing from these hosts using ICMP to the "Internet". What I have found though is that the Checkpoint box only ever uses one ISP interface, unless that interface goes down and then it fails over to the other interface which sends it via the other Cisco router even though I have configured the Gateway for Load Sharing.

Now I know that even if it could, the Checkpoint gateway would not for instance load share an FTP download over both WAN links as this would break the FTP protocol, but considering ICMP is connectionless, I thought it would be a very easy protocol to load share, sending every second ping out a different external interface, but this does not happen.

So my question is, does load sharing actually allow both WAN interfaces to be used simultaneously? If it does proper load sharing out both interfaces what is its methodology? Does it load share on a per-host, per-protocol, per-connection basis?

I know my setup is a little different as most would perform NAT on the CP gateway, not upstream on the routers as I do, but I don't see why removing this function from the gateway would cause any issues.

I would like to recommend Checkpoint if it is smart enough to properly load share these two WAN links but if it cannot, Cisco ASA's can do ISP redundancy in a Primary/Backup setup like what I am experiencing with CP at the moment for a lot less money.

Thanks

[Spawn]

hmm...lot of work in hand for you.
I don't like to use CP for loadsharing, that part is taken care by cisco routers ( our AS, multiple isp's,etc). Some like to use Radware...

check the following SK's : couldn't paste here cause of some SQL attack prevention feature... go site for more details.
==================
Controlling connections configured with ISP Redundancy in Load Sharing mode

Solution ID: sk42636

Symptoms

Connections from the same source pass only through one of the ISP channels and not through both ISP channels per Round-Robin mechanism when Security gateway is configured with ISP Redundancy in Load Sharing mode.

Cause - This behavior is the default design of ISP Redundancy in Load Sharing mode.

Solution Background:

By default, in ISP Redundancy in Load Sharing mode, connections from the same "Client" located behind the Gateway/Cluster are sent out the Gateway/Cluster every time over the same ISP channel.

This is a sort of "Client Stickiness" mode. This mode was chosen to be the default, because it is the best way to distribute connections between two ISP channels without losing communications that use dynamic ports or port redirection (e.g., FTP, VoIP, etc).

more sk's to read:

sk23630: Advanced configuration options for ISP redundancy

[v33dubya]

Thanks very much for pointing me towards that SK.

That explaination matched what I was seeing to a degree. One inside host was always going out the same interface. That being said, I also had tried a second inside host and it was sent out the same interface as the first host but I was willing to accept that maybe the load balancing wasn't perfect and maybe the next two inside hosts would go out the second interface.

I made the DBedit modifications as per the SK to have the same source/destination flows go out the same interface instead of the default to have just the same source. However nothing has changed. All connections go out the same interface regardless of the destination address of the connection. I tested in the following ways from the same inside host and considering the gateway was now configured to send source/destination pairs out the same interface I would have expected connections from the same source to different destinations would utilise both connections:

- ping outside host 1
- ping outside host 2
- ping outside host 3
- telnet outside host 4
- telnet outside host 5

All of these connections went out the same interface on the gateway. I checked SVMonitor and saw that the ISP Redundancy on the gateway was operating in LS mode and both WAN interface were regsitered as up.

So I think I am starting to come to the conclusion that the answer to my original question - does load sharing reall work? - is "no". Is there anything else I might be missing?

Thanks

[mcnallym]

ISP Redundancy from Check Points perspective is about INBOUND Traffic, not outbound traffic.

For inbound Load Balancing then is nothing more advanced the DNS based, wether you use DNS Proxy and then host a DNS internally on the DMZ for the MX Records and other DNS entries that the Check Point DNS Proxy cannot handle, or use an externally hosted DNS.

Once a client does a DNS lookup and gets the name resolved then will use that IP address. The DNS is effectively doing the Load Sharing for you not Check Point.

For Outbound Traffic then works in the following way

1.) Static NAT traffic

This ALWAYS uses the DG of the box to route the traffic out and then the manual nat rules will NAT the traffic accordingly. The Check Point only ever has one Default Gateway and you can see that the cpisp_update script basically changes the DG when the ISP link fails over.

You have to use manual NAT rules for this as if you use the Automatic NAT option then the traffic is always routed out over the ISP link that the NAT IP is part of in terms of IP range. ie if have two line

40.40.40.x/24
50.50.50.x/24

The if the active DG is 40.40.40.x but the Automatic NAT you configure uses a 50.50.50.x IP address then you will see the traffic routed out over the 50.50.50.x link even though that is effectively the secondary ISP link and teh 40.40.40.x is the primary link and has the active DG. If the line fails then continues to NAT with the 50.50.50.x IP and unless the other ISP will route the 50.50.50.x traffic back to you then won't work.

From this then determined that NAT plays a big part in how Check Point handles Outbound Traffic when ISP Redundancy is configured. As you aren't intending to NAT on the Check Point then I don't think that the Check Point is really suitable for you in this scenario.


2.) Hide NAT traffic

This is shared across the two lines and you have to set the traffic to NAT behind the Gateway, it doesn't work properly if specifying an IP address instead to Hide behind. The same client will always use the same ISP line for a session so that it doesn't break connections such as FTP etc. A different client may depending upon the algorithm determine that the second ISP link is used, however a single host machine will always use a single line, it doesn't share the traffic across both links from a host.

Is Load Balancing in the same way that Check Point consider there Zero Downtime Upgrade, whereby failover with no session synch between two gateways is Zero Downtime. As such Load Sharing in ISP Redundancy does work as Check Point intended, however not necessarily in the way that other people understand Load Sharing.

Personally I never design to use the ISP redundancy feature, unless the customer is adamant about it. It ticks a box for the 1 page reviews however when you dig deeper into the feature then whilst maybe OK for a Small Business, that just wants a bit of resilience, however once you get to the point where require Load Sharing then I always advise to get a 'proper' solution for the requirement. From talking with other consultants at Events then that seems to be the general feeling of the product. Even our past Check Point SE's have advised to get a proper solution. They don't work there anymore so not dropping any current Check Point employee in it. The feature is there and if can live with the limitations and how Check Point intend it to be used then it works. If you expect it to compete with the likes of an F5 ISP Load Balancing solution / Cisco Routers then you will be disappointed.

[v33dubya]

Hi,

thanks for your response.

So, from what I understand it is the DNS responses which perform inbound load balancing on the CP GW. Well that is pretty straight forward, as setting up two host records for the same DNS name and having them used in a Round Robin fashion, each directing the client to an IP address reachable via a different ISP link, would provide inbound load balancing. That process would be totally independent of the firewall though.

That being understood though, it is one thing to have DNS queries resolved as above so each inbound connection comes in via an alternating ISP link but it is a second thing for the firewall to support this and make sure that response traffic goes back out the same interface, especially when NAT is involved. There must be something more the CP GW is doing to track these inbound connections to make sure their return traffic goes back out the same interface it arrived on. Can you provide any further details on this? Is it just the same algorithm discussed previously which sticks each source address to the same WAN link by default though this can be modified bind source-destination pairs to the same WAN link.

I did try what you suggested for outbound connections and created a Network object which represented an Inside subnet within which I have 3 hosts. On that network object I applied automatic hide NAT and hid the network behind behind the gateway. I, for the sake of testing, removed the Dynamic PAT/overloading from the upstream Cisco routers which performed this NAT previously and then tested ping an "internet" IP from each of the three hosts. I can see from these tests that two hosts are hidden behind the IP of ISP link 1 and one behind the IP of ISP link 2.

So it seems that what is required for this to work is the CP GW to perform NAT of some sort. I did not test with static NAT as this isn't required in this design but the automatic hide NAT did the trick to support outbound load balancing.

Additionally, I was also able to confirm that not only were the outbound connections (ICMP) being distributed across the two WAN links on a per source IP basis, that a single source would have its ICMP packets load balanced across the two WAN links when pinging different destination IP addresses. This is as a result of the previous suggestion to modify the algorithm using DBEdit. Without this, the same inside host would always take the same WAN link regardless of destination which would not have worked for this client as they use Windows Remote Desktop services so all outbound connections would only appear from a small number of source IPs.

In the end, as performing NAT on the CP GW would require it to either having public addressing itself (only the Cisco routers have this at the moment) or for outbound connections to be source NAT'd twice (on the CP GW for LS to work and again on the Cisco routers for public address translation) I am not sure this is going to be an ideal solution.

Thanks very much to both of you for your input.

VD

[mcnallym]

I suspect that it is able to route the traffic back out of the same interface that it came in on, as the traffic would be natted. Looking at the outbound traffic then the NAT seems to make a difference with the static outbound so as is an existing session then the traffic is natted back out the same as normal traffic and thus uses the same interface for the reply as came back in.

Thats my educated guess as to how it does it.

However without the NAT on the Check Point I don't see this as suitable solution.