URGENT HELP!!!!! CluserXL Unicast mode

[cciesec2006]

I have a pair of NGx R65 running in ClusterXL Active/Active Unicast mode.

Eth0 of FW1 is connect to Catalyst switch SW1 6513 port 7/7 and Eth0 of FW2
is connected to Catalyst switch SW2 6513 port 7/8. There is an EtherChannel
trunk between these two switches.

When I connect to SW1 and run "show cam dynamic 7/7" I see this:

CAT6513-1> sh cam dynamic 7/7
* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.
X = Port Security Entry $ = Dot1x Security Entry M = Mac-Auth-Bypass Entry

Destination Ports or
VLAN Dest MAC/Route Des [CoS] Age VCs / [Protocol Type]
---- ------------------ ----- ---------- ---------------------
199 00-15-17-79-12-c6 0 7/7 [ALL]
199 00-d0-fe-8e-40-03 0 7/7 [ALL]
199 00-00-00-00-fe-00 0 7/7 [ALL]
199 00-d0-fe-8e-64-03 0 7/7 [ALL]
Total Matching CAM Entries Displayed = 4
CAT6513>

00-15-17-79-12-c6 = Firewall #1 physical MAC address
00-d0-fe-8e-40-03 = Cisco MAC address (no idea where it comes from)
00-00-00-00-fe-00 = Firewall #1 ClusterXL MAC address
00-d0-fe-8e-64-03 = Cisco MAC address (no idea where it comes from)


When I connect to SW2, I see this, which is normal:

CAT6513-2> sh cam dynamic 7/8
* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.
X = Port Security Entry $ = Dot1x Security Entry M = Mac-Auth-Bypass Entry

Destination Ports or
VLAN Dest MAC/Route Des [CoS] Age VCs / [Protocol Type]
---- ------------------ ----- ---------- ---------------------
199 00-15-17-79-16-dc 0 7/8 [ALL]
199 00-00-00-00-fe-01 0 7/8 [ALL]
Total Matching CAM Entries Displayed = 2
CAT6513-2>

00-15-17-79-16-dc = Firewall #2 physical MAC address
00-00-00-00-fe-01 = CluserXL #2 MAC address


Can anyone explain to me why I am seeing 4 MAC addresses on port 7/7
of SW1? The other Cisco MAC addresses I could not find anywhere except
port 7/7.

These two firewalls were upgraded from NG R55 to NGx R65. If I rebuild
both firewalls with NG R55, then on both port 7/7 on SW1 and 7/8 on SW2
I only see two MAC addresses, Firewall physical IP and ClusterXL MAC, which
is expected.

Can someone explain this to me? Thanks.

[th0i3]

Have you tried using Vendor/Ethernet/Bluetooth MAC Address Lookup and Search to find out what these mac addresses are. Most important, what impact is this causing to your environmnet?

[cciesec2006]

it causes some impact in the production environment that the firewalls can
ping some hosts but not others and vice versa. The important question is
why it behaves this way. It wasn't an issue in NG R55

[northlandboy]

Originally Posted by cciesec2006

it causes some impact in the production environment that the firewalls can
ping some hosts but not others and vice versa. The important question is
why it behaves this way. It wasn't an issue in NG R55

A quick lookup indicated those mac addresses belong to "Convision Technology Gmbh" not Cisco.

Do those mac addresses actually belong to other devices on your network? Is that what is causing problems with pinging some hosts?

Normally a switch will update its cam tables based on the source mac addresses that it sees. If you run tcpdump -e on those firewalls, are you seeing frames leaving the firewall with that source MAC? If you're not, then your switch is doing something it shouldn't. If you are seeing them, well what are those frames?

As an aside, you might want to be a little more circumspect before declaring every issue as being "URGENT HELP!!!!!" Boy who cried wolf and all that. Most environments would not consider it an urgent issue if the firewall could not ping some hosts. Most places would probably not notice for a while. If some hosts could not route through the firewall, then yes that would be an issue. But if you'd just done an upgrade, you would troubleshoot for a while, then roll back, and pursue it in a test environment, and through normal support channels. It would not be a priority 1 fault.

[cciesec2006]

Originally Posted by northlandboy

A quick lookup indicated those mac addresses belong to "Convision Technology Gmbh" not Cisco.

Do those mac addresses actually belong to other devices on your network? Is that what is causing problems with pinging some hosts?

Normally a switch will update its cam tables based on the source mac addresses that it sees. If you run tcpdump -e on those firewalls, are you seeing frames leaving the firewall with that source MAC? If you're not, then your switch is doing something it shouldn't. If you are seeing them, well what are those frames?

As an aside, you might want to be a little more circumspect before declaring every issue as being "URGENT HELP!!!!!" Boy who cried wolf and all that. Most environments would not consider it an urgent issue if the firewall could not ping some hosts. Most places would probably not notice for a while. If some hosts could not route through the firewall, then yes that would be an issue. But if you'd just done an upgrade, you would troubleshoot for a while, then roll back, and pursue it in a test environment, and through normal support channels. It would not be a priority 1 fault.

"Do those mac addresses actually belong to other devices on your network? Is that what is causing problems with pinging some hosts?"

THERE ARE NO OTHER DEVICES ON THE NETWORK WITH THOSE MAC ADDRESSES. I think I know my cisco stuffs pretty well. I am not a Cisco
expert but I think I am quite good at it. I do my homework quite well before
asking the forum.

The fact of the matter is that the switchport which the firewall is connected
to should only show TWO MAC addresses, NOT FOUR. That's how life works.
It is telling me that the firewall are doing some crazy things.

"Boy who cried wolf and all that. Most environments would not consider it an urgent issue if the firewall could not ping some hosts. Most places would probably not notice for a while. If some hosts could not route through the firewall, then yes that would be an issue."

I upgraded this firewall two weeks ago and about 95% of the traffics make
it through. There are several weird issues after the upgrade but I did not
notice them after the time. My customer does not know about it yet.

I took the whole thing and brought it back into my lab environment but I
could NOT re-produce this either. That's what this thing so frustrating.

As a side notes, in my lab environment, the switchport sometimes showed
up with 2 MAC Addresses, sometimes it showed up with 1 MAC addresses:

C3550-lab>sh mac-address-table dynamic interface g0/1
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
100 0000.0000.fe00 DYNAMIC Gi0/1 (clusterXL MAC)
100 0015.1775.d74a DYNAMIC Gi0/1 (firewall physical MAC)
Total Mac Addresses for this criterion: 2
C3550-lab>sh mac-address-table dynamic interface g0/1
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
100 0015.1775.d74a DYNAMIC Gi0/1
Total Mac Addresses for this criterion: 1
C3550-lab>
C3550-lab>sh mac-address-table dynamic interface g0/1
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
100 0000.0000.fe00 DYNAMIC Gi0/1
100 0015.1775.d74a DYNAMIC Gi0/1
Total Mac Addresses for this criterion: 2
C3550-lab>

I HAVE CONSTANT TRAFFICS GOING ACROSS THIS FIREWALL. ANYONE KNOW WHY?

[th0i3]

Is it possible if you put static ARP entries for the firewall and multicast MACs. If this is case, are there any impact to your other mysterious devices?

To be honest, I have never seen this issue before in my support career.

[cciesec2006]

Originally Posted by th0i3

Is it possible if you put static ARP entries for the firewall and multicast MACs. If this is case, are there any impact to your other mysterious devices?

To be honest, I have never seen this issue before in my support career.

Yes, I would agree with you that static ARP entries are needed BUT only if
you use Active/Active Multicast mode. I am using Active/Active Unicast mode.
Therefore, it is not needed. Furthermore, it works about 95% of the time.

I notice that since I swapped out two cluster firewalls from NG AI R55
to NGx R65, I am seeing the same thing on both of them. Weird part is
that I can not produce these things in my lab environment.

[msjouw]

and as a sidenote, I see you were using the sh cam command which is CATOS? You cannot compare the way CATOS and the 3550 IOS switches work. Could it be you have a IGMP configured?

Regards, Maarten.

[cciesec2006]

Originally Posted by msjouw

and as a sidenote, I see you were using the sh cam command which is CATOS? You cannot compare the way CATOS and the 3550 IOS switches work. Could it be you have a IGMP configured?

Regards, Maarten.

I see this issue in both CatOS and IOS in the production environment as well.
I have IGMP configured for both CatOS and IOS in both production and lab
environment.

[northlandboy]

I thought Cisco had finally taken CatOS out the back and put a bullet in it...guess it's still out there though.

Are you able to capture any packets leaving the firewall with the wrong source MAC? If you could capture that traffic with tcpdump/fw monitor, you should then be able to present it to Check Point and ask them WTF is going on.

The sometimes seeing one mac, sometimes two, should just be due to cam table aging. Virtual macs can be a pain with some protocols, since the devices don't send out packets with a source virtual mac. VRRP uses gratuitous arp to get around this, and update cam tables.

Regards IGMP, I have run into problems with a few specific IOS versions, where they did some nasty things with multicast packets - like adding static CAM entries, even though it wasn't configured to - but it's not related to this issue.

[msjouw]

fw monitor will not do for getting MAC info as fw monitor replaces the mac info with the interface and direction fields. so you will need to use tcpdump for that.
Regards, Maarten.