Urgent: ClusterXL Active/Active Unicast mode and icmp issue

[cciesec2006]

Can some help me with this issue:

1- I have a pair of Secureplatform NG with AI R55 gateways with HFA_17
being managed by a Provider-1 NGx R65 CMA with HFA_02 and hf_249
Secureplatform. Everything is working fine. From gw1, I can ping
host Linux_1 (IP 10.109.114.9). From gw2, I can also ping host
Linux_1 as well. Everything is fine. By the way, I am running Active/Active
ClusterXL Unicast mode.

2- Yesterday, I migrated the gateways to NGx R65 with HFA_02 and hf_249.
ALL IP addresses on the gateways stay the same. on the CMA, I modified
the gateway cluster object general properties from NG w/ AI R55
to NGx R65. Other than that, everything stays the same. I then re-SIC
the gateways, applied the new NGx license and then push the policy to
the gateway cluster. I am also running Active/Active ClusterXL
Unicast mode.

Everything is working fine, as far as I know; however, from gw1, I CAN
ping Linux_1 but I can NOT ping Linux_2. From gw2, I CAN ping Linux_2
but I can NOT ping Linux_1. When I run tcpdump on Linux_2 and ping
hot Linux_2 from gw1, I can see that icmp echo get to Linux_2 with the
source is the ClusterXL IP, 192.168.15.254, get to Linux_2 and that
host Linux_2 replied with echo-reply to clusterXL ip 192.168.15.254.
When I am running tcpdump on gw2, I see "echo-reply" hit gw2.
Therefore, from gw1, I can NOT ping host Linux_2.

I get the same issue with gw2 when I tried to ping host Linux_1.
Linux_1 see "echo" request from clusterXL ip address when gw2 pings
and that Linux_1 replies with "echo-reply"; However, the "echo-reply"
gets to gw1. Therefore, from gw2, I can NOT ping Linux_1.

It seems to effect only icmp. I am not seeing issues with ssh, telnet, wget
when I initiated traffics from either gw1 or gw2, just icmp.

Can anyone explain this me and how I can fix this icmp issue?

Thanks everyone.

[Thorpuse]

See Solution ID: sk26874. icmp requests are managed a little differently than TCP sessions in an active/active cluster. If TCP sessions are working, then I wonder what the issue is really - is pinging a host from the cluster member something you really need to do?

I realise that the sk is for the opposite condition than what you're testing, but I suspect the solution will work both ways.

[cciesec2006]

Originally Posted by Thorpuse

See Solution ID: sk26874. icmp requests are managed a little differently than TCP sessions in an active/active cluster. If TCP sessions are working, then I wonder what the issue is really - is pinging a host from the cluster member something you really need to do?

I realise that the sk is for the opposite condition than what you're testing, but I suspect the solution will work both ways.

"is pinging a host from the cluster member something you really need to do?"

Issue is that it worked NG with AI R55 gateways but not in NGx R65. Why?

I am very aware with sk26874.

I can NOT reproduce this in my lab environment with the same CMA
that I migrated over from my production CMA. I am at a lost at the moment.

[melipla]

How about trying something like sk32058?

Also, are you using SDF?

[cciesec2006]

Originally Posted by melipla

How about trying something like sk32058?

Also, are you using SDF?

Am I using SmartDefense? No. Everything is turned OFF.

sk32058 applies to NG with AI R55. I am using NGx R65. Furthermore,
it talks about NAT traffics going through the gateways. My traffics initiate
from the gateways members themselves.

sk32122 sort of applies to my situation but it applies for NGx R60 managing
FP3 gateways. Both of my CMA and gateways are NGx R65.

I am at a lost at the moment

[melipla]

Originally Posted by cciesec2006

Am I using SmartDefense? No. Everything is turned OFF.

I was referring to Sticky Decision Function.

Originally Posted by cciesec2006

sk32058 applies to NG with AI R55. I am using NGx R65. Furthermore, it talks about NAT traffics going through the gateways. My traffics initiate from the gateways members themselves.

And you've never seen an NG issue exist in an NGX installation? ;p sk32058 refers to NAT rules which were specific for the gateways themselves. I think your problem has to do with the fact that traffic leaving your gateway is Hide NAT'ed to the cluster VIP. I'm betting you have SDF enabled which means the icmp reply [going to the VIP] is traversing the wrong [stickied] gateway which is why you're seeing replies from Host X and not Host Y. GW2 would only be able to ping hosts which are stickied to it...

[cciesec2006]

Originally Posted by melipla

I was referring to Sticky Decision Function.



And you've never seen an NG issue exist in an NGX installation? ;p sk32058 refers to NAT rules which were specific for the gateways themselves. I think your problem has to do with the fact that traffic leaving your gateway is Hide NAT'ed to the cluster VIP. I'm betting you have SDF enabled which means the icmp reply [going to the VIP] is traversing the wrong [stickied] gateway which is why you're seeing replies from Host X and not Host Y. GW2 would only be able to ping hosts which are stickied to it...

I took the production CMA and migrated it into my lab Provider-1. EVERYTHING
IS EXACTLY THE SAME INCLUDING THE IP ADDRESSES OF THE GATEWAYS.
My lab gateways gw1 and gw2 can ping both linux_1 and linux_2. If what
you're saying is correct, then should I see the same thing in my lab
environment since this is a CMA migration and everything is identical. Why
am I getting different results? I am at a lost here.

Anymore ideas my friend? I am getting desparate.

[melipla]

Originally Posted by cciesec2006

If what you're saying is correct

That's the rub, you already know you can't reproduce the behavior in the lab. You'd have to test my theory on the production cluster. You know traffic between GW2 and Linux1 doesn't work, put in a specific NAT rule to tell it not to hide NAT GW2 behind the cluster vip & see what happens.

[cciesec2006]

Originally Posted by melipla

That's the rub, you already know you can't reproduce the behavior in the lab. You'd have to test my theory on the production cluster. You know traffic between GW2 and Linux1 doesn't work, put in a specific NAT rule to tell it not to hide NAT GW2 behind the cluster vip & see what happens.

"You'd have to test my theory on the production cluster."

I think I would need to up my resume before I test this on the production
cluster. I would get my ass kicked if something happen to it.

[northlandboy]

I never really understood that default setting for hide-natting outbound traffic from the clusters behind the cluster VIP. I couldn't quite work out why you'd want to do that. I know you can go and change it to not hide-nat certain services, by editing the *.def files, but still, it seemed pointless. I guess someone somewhere has a reason for why they wanted it.

Which makes me wonder a bit about why you can't reproduce it in the lab - you've done a CMA migrate, which, if I remember correctly (it's been over 18 months since I last did a migration, OK ;-), won't carry across custom *.def file changes you may have made. I suspect that if you did a full MDS backup from prod, and a restore into the lab, you might be able to reproduce it there. Sound like a possible cause for the lab/prod discrepancy?

[cciesec2006]

Originally Posted by northlandboy

I never really understood that default setting for hide-natting outbound traffic from the clusters behind the cluster VIP. I couldn't quite work out why you'd want to do that. I know you can go and change it to not hide-nat certain services, by editing the *.def files, but still, it seemed pointless. I guess someone somewhere has a reason for why they wanted it.

Which makes me wonder a bit about why you can't reproduce it in the lab - you've done a CMA migrate, which, if I remember correctly (it's been over 18 months since I last did a migration, OK ;-), won't carry across custom *.def file changes you may have made. I suspect that if you did a full MDS backup from prod, and a restore into the lab, you might be able to reproduce it there. Sound like a possible cause for the lab/prod discrepancy?

1- There is a way to change it without editing the *.def files. Un-check
ClusterXL, that will take you to 3 party cluster configuration. There, you can
change how inbound and outbound traffics should be hidden behind the
Cluster VIP. I checked both the production CMA and lab CMA and the
settings are identical.

2- You're correct that the *.def files will not carry over with CMA migration;
Both the production and lab Provider-1 are Secureplatform NGx R65 running
with identical HFA and hot fixes. Furthermore, I also did a diff on ALL *.def
files between production and lab CMA and they are IDENTICAL.

I've opened a TAC with Checkpoint and the TAC guy has been useless so
far. It is a sad day when the customers know more about Checkpoint
products than Checkpoint TAC themselves. Hopefully, the case will get to
a more knowledgable checkpoint TAC engineer.

[melipla]

Originally Posted by cciesec2006

I think I would need to up my resume before I test this on the production cluster. I would get my ass kicked if something happen to it.

You have no faith your NAT writting capabilities ;) To throw some other ideas:

I've seen varying / inconsistent response from Hide-NAT'ed cluster member IP(s) before, so it doesn't surprise me that the lab behaves differently than the production. Are you using identical switching equipment in your lab setup or is the routing virtualized too?

From the 3rd Party config options, do you have "use state sync" and "forward cluster's incoming traffic to cluster members' IP addresses" checked? Is icmp a sync'ed service?

[cciesec2006]

Originally Posted by melipla

You have no faith your NAT writting capabilities ;) To throw some other ideas:

I've seen varying / inconsistent response from Hide-NAT'ed cluster member IP(s) before, so it doesn't surprise me that the lab behaves differently than the production. Are you using identical switching equipment in your lab setup or is the routing virtualized too?

From the 3rd Party config options, do you have "use state sync" and "forward cluster's incoming traffic to cluster members' IP addresses" checked? Is icmp a sync'ed service?

"Are you using identical switching equipment in your lab setup or is the routing virtualized too?"

Yes, I am using Catalyst 3750 in both production and lab equipments. Furthermore, I am running identical version of IOS on both production and
lab equipments.


Routers are runing identical IOS in both production and lab equipments.

"From the 3rd Party config options, do you have "use state sync" and "forward cluster's incoming traffic to cluster members' IP addresses" checked?"

it is NOT checked. However, in my lab environment, it works regardless if
this option is checked or un-checked,

"Is icmp a sync'ed service?"

Yes, I have the icmp service synchronized in the cluster.

[melipla]

How do the fw monitors compare production vs lab?

[msjouw]

for your lab environment do you have the capability to generate at least the same amount of traffic as in the production?