High CPU between 2811 and ClusterXL

[Brittin_C]

Hello!

I am having a problem with my clusters connection to the internet.

My set up is as follows:


When connecting out to the Internet from hosts on the 3750s out the ClusterXL firewall and pulling down even just 1 download (say 600MB at 300Kbps), the active HSRP 2811 redlines at 99% utilization.

Laptops directly connected to the 2811 do not generate this kind of traffic. Anyone have some ideas as to why?

Additive facts:
- IGMP Snooping turned off on all switches and routers
- 2811's utilize a NME16 etherswitch card to connect to the firewall
- 2811 NME16 have a trunk between them
- 3750's only show about 7% utilization

I'd appreciate any help I could get. Thank you!

[chillyjim]

Try to switch to unicast mode (aka pivot mode).
I assume you are in load share mode.

[cciesec2006]

Hi,

I have the exact same setup you have for my home lab with a couple of exceptions:

- I use HSRP on a pair of Cisco 2851 using IOS 12.4(16) IP Advanced
Service code,

- I run Checkpoint NGx R65 with HFA_02 and HFA_249 on a pair of
Dell Optiplex G270 2.8GHz with 1GB RAM,

- I use ClusterXL Active/Active in Unicast Mode. However, when
I switch over to multicast mode, everything still works fine, ofcourse,
I have to use arp command on the router in muticast mode. But
other than that, everything is the same. It works and no high cpu
on the router,

- I have 5 interfaces on the Firewalls,

- Everything is connected to a pair of 3750s with trunk between them,

- I get about 40mbps download through the 2851 when CPU hits 99% CPU.

CCIE Security

[Brittin_C]

I am thinking that the 2811 doesnt handle multicast well. And that a broadcast storm is being generated on the NME16 on the back side of the router.

I tried to statically assign MAC addresses to ports but it doesnt seem to like this.

I have not tried it in Unicast mode, but I suspect it would work fine in that case.

[cciesec2006]

I've worked with ClusterXL Active/Active for the past 8 months and I can
say that I do NOT like it.

Unless you have a very special reason to run Active/Active, Active/Standby
will work well in about 99% of the time. Running Active/Active is pain in the
ass especially when you have run tcpdump. In Active/Active mode, you have
to run two tcpdump for each sessions because traffics pass through both
firewalls then you have to merge the files. It's a real pain.

That's the law of un-intended consequence when you run Active/Active.

Back to your question, Active/Active Unicast mode works well 99% of the
time. You will NOT get a true 50-50 load sharing. Instead you will get
30-70 load sharing.

[chillyjim]

Originally Posted by cciesec2006

I've worked with ClusterXL Active/Active for the past 8 months and I can
say that I do NOT like it.

Unless you have a very special reason to run Active/Active, Active/Standby
will work well in about 99% of the time. Running Active/Active is pain in the
ass especially when you have run tcpdump. In Active/Active mode, you have
to run two tcpdump for each sessions because traffics pass through both
firewalls then you have to merge the files. It's a real pain.

The two main reasons to run in active/active mode:

1. Your gateways are running with high CPU load (normally due to security servers or logging requirements).

2. You are tired of having the boss say "So that extra gateway is just sitting there doing nothing?"

I have to say I run into #2 a lot more than #1.

Active/Active, no matter how well it works for you, will always make debugging a problem harder.

My personal recommendation to just make life easier is to run active/standby in unicast (pivot) mode, it just plain works and unless you are really pushing your hardware, the other modes won't really buy you anything you'll notice in the real-world.

[Brittin_C]

How does that work from an IP address point of view? Do they still have a shared IP address?

And do i have to take the APR statements out of my routers for it to work?

[cciesec2006]

In ClusterXL, either Active/Active or Active/Standby, Unicast mode, you do
NOT need ARP statements in the router.

[chillyjim]

Yes you do get the VIP (Virtual IP)/shared IP address
No ARP required as ccsesec said.

[Brittin_C]

Update...

So, on the 2811... turning on IGMP snooping is a GOOOOOODDDD thing. Despite how bad it is on switches.

Using igmp statics... VERY BAD.

I took out the static IGMPs (not the static ARPs) and performance and reliability shot thru the roof.

I think the key issue is the 2811 is unable to statically assign a multicast address to more then one port using the "ip igmp snooping vlan 100 static ..." command. Putting these in kills 1/2 of connectivity. Running with out IGMP snooping = packet storm and dead router. I ordered a pair of 3750's to put between the routers and firewalls anyhow as the routers still run at about 80% with 1.2 Mbps going thru them. But still HUGE improvement.