Secureplatform performance !!!!!!

[cciesec2006]

Hi all,

After spending about six weeks of testing both NGx R65 2.4/2.6 kernel on both
Sun X4200-M2 and IBM 3650, I would like to offer some of the results and issues
that I've come acrossed with both SPLAT 2.4/2.6 kernel and hardware vendors.
Keep in mind that the constraint that I have is that my management server
is Provider-1 NGx R65 with HFA_02 2.4 kernel Secureplatform. Before I begin,
I would like to express my gratitude to ChillyJim for providing me with
eval licenses. Without that, my test would not have been possible. Here we go:

Scenario #1:
Sun X4200-M2 dualcore Opetaron Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.6 kernel. In this scenario, cpd takes up
100% of the CPU in a particular core. Under moderate and heavy traffics, the firewalls,
at random, keeps losing SIC with the CMA. Firewalls sometime stopped passing
traffics and required a mannual reboot. Because the X4200-M2 comes with Nividia
on-boar NIC, this must be disabled to ensure system stability. Without SecureXL
installed, CPU always bound to a single core thus causing system becoming unstable.
If you also use QoS, SecureXL will be disabled. With SecureXL enable, all workload
are evenly distributed across all 4 AMD cores. In term of hard drive RAID-1 mirror,
sometimes it works, sometime it does not work.

Scenario #2:
IBM x3650 dual quad-core Intel Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.6 kernel. Same result as Scenario #1

Scenario #3:
IBM x3650 dual quad-core Intel Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.4 kernel. Result is that this kernel
version is very stable. Everything is working very smoothly. WITHOUT SecureXL,
CPU workload is evenly distributed on all cores (I know SecureXL is disabled
because I have QoS enable). Throughput is excellent. I am getting about 990Mbps
throughput on Copper Gig interface. RAID-1 mirror works extremely well.

Recommendations:

#1: STAY AWAY FROM NGX R65 2.6 KERNEL. THIS PLATFORM IS EXTREMELY UNSTABLE.

#2: Do NOT buy Sun X4200-M2 and/or Dell 2950-III because these boxes do not
support SPLAT 2.4 kernel.

#3: IBM x3650 is the best platform, IMHO for NGx R65.

#4: Stay with the SPLAT NGx R65 2.4 kernel. IBM Firefly uses 2.4 kernel.

#5: Checkpoint Secureplatform PRO is rather unstable, IMHO, especially with
multicast

Contact me off-line if you have questions. Thanks.

[chillyjim]

Please elaborate on #5.

[cciesec2006]

Please refer to this post for multicast:

HELP!!!!! localhost.localdomain#EU0 999 Error reading message from AMI server

The problem with Secureplatfom Pro is that I keep getting this error:

localhost-localdomain-eu0-999-error-reading-message-ami-server

Sometimes things work. Sometimes it does not. I am at a lost as to why.
When you are in router mode, if you do a "control-Z" by mistake, since
I come from cisco background, it will break the configuration and generate
all kind of errors. The only way to fix this is to reboot the firewall.

[BarryStiefel]

Originally Posted by cciesec2006

Hi all,

After spending about six weeks of testing both NGx R65 2.4/2.6 kernel on both
Sun X4200-M2 and IBM 3650, I would like to offer some of the results and issues
that I've come acrossed with both SPLAT 2.4/2.6 kernel and hardware vendors.
Keep in mind that the constraint that I have is that my management server
is Provider-1 NGx R65 with HFA_02 2.4 kernel Secureplatform. Before I begin,
I would like to express my gratitude to ChillyJim for providing me with
eval licenses. Without that, my test would not have been possible. Here we go:

Scenario #1:
Sun X4200-M2 dualcore Opetaron Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.6 kernel. In this scenario, cpd takes up
100% of the CPU in a particular core. Under moderate and heavy traffics, the firewalls,
at random, keeps losing SIC with the CMA. Firewalls sometime stopped passing
traffics and required a mannual reboot. Because the X4200-M2 comes with Nividia
on-boar NIC, this must be disabled to ensure system stability. Without SecureXL
installed, CPU always bound to a single core thus causing system becoming unstable.
If you also use QoS, SecureXL will be disabled. With SecureXL enable, all workload
are evenly distributed across all 4 AMD cores. In term of hard drive RAID-1 mirror,
sometimes it works, sometime it does not work.

Scenario #2:
IBM x3650 dual quad-core Intel Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.6 kernel. Same result as Scenario #1

Scenario #3:
IBM x3650 dual quad-core Intel Processors with 4GB RAM in Active/Active
ClusterXL unicast mode with NGx R65 2.4 kernel. Result is that this kernel
version is very stable. Everything is working very smoothly. WITHOUT SecureXL,
CPU workload is evenly distributed on all cores (I know SecureXL is disabled
because I have QoS enable). Throughput is excellent. I am getting about 990Mbps
throughput on Copper Gig interface. RAID-1 mirror works extremely well.

Recommendations:

#1: STAY AWAY FROM NGX R65 2.6 KERNEL. THIS PLATFORM IS EXTREMELY UNSTABLE.

#2: Do NOT buy Sun X4200-M2 and/or Dell 2950-III because these boxes do not
support SPLAT 2.4 kernel.

#3: IBM x3650 is the best platform, IMHO for NGx R65.

#4: Stay with the SPLAT NGx R65 2.4 kernel. IBM Firefly uses 2.4 kernel.

#5: Checkpoint Secureplatform PRO is rather unstable, IMHO, especially with
multicast

Contact me off-line if you have questions. Thanks.

This is really outstanding! This is precisely the type of useful information that you'd never get from Check Point that we should be sharing with each other here on CPUG. Thanks for all your hard work on this!

[patrick]

Hello,

Can you explain the procedure to disable the nvidia card on the Netra X4200 M2. Because the only way I find to disable these interfaces is either we force Splat 2.6 to not load the forcedeth driver or in the BIOS where we have to disable all the onboard interfaces.

Please share the procedure you use to disable the nvidia interfaces?

Other question are you aware of any problem with the Quad Card X4445A PCI-X and Splat 2.5 ( cassini drivers)?

Because we installed SecurePlatform 2.6 (2.6.18-22cp SMP) on a Netra X4200 M2. We are using the built-in Intel Gigabit Ethernet ports and ports from a Sun X4445A (Sun Quad GigaSwift Ethernet Card).

The problem is that both enforcement modules report all of the configured X4445A ports as being DOWN. "cphaprob -a if" also reports only two interfaces required when it should be six interfaces required.

The OS is not reporting any problems with the network cards. When I do a tcpdump on one of the DOWN interfaces, "cphaprob -a if" will report Inbound UP for a couple of seconds and then state the interface is down again.

[lammbo]

Thanks for this!

I do have 1 question though... You say 2.6 is unstable and yet it appears you only tested in Active/Active. Would you say this statement remains true in an Active/Passive HA cluster?

[cciesec2006]

Follow this instruction and you will be able to disable both Nvidia
NICs on the Sun X4200-M2:


*The following 2 switches are located in the G12F BIOS setup
utility under the Chipset\Southbridge Configuration menu.

* MAC Interface [Enable/Disable]
* I04 MAC Interface [Enable/Disable]*

Setting the "MAC Interface" switch to [Disabled] will disable
all functionality (PXE and OS level) of the Nvidia CK8-04 NIC.

Setting the "I04 MAC Interface" switch to [Disabled] will
disable all functionality (PXE and OS level) of the Nvidia I04 NIC.

[fireverse]

Originally Posted by cciesec2006

#1: STAY AWAY FROM NGX R65 2.6 KERNEL. THIS PLATFORM IS EXTREMELY UNSTABLE.

I have a very very large customer (one of the largest privately held companies in the world) running P1 on 2.6 and their gateways on 2.6 for four months, with no serious issues. The only issue they encountered so far was a RADIUS authentication problem with Provider-1. This was corrected with a Hot Fix. So I guess I have to say that my experience has been dramatically different what was posted. The customer is using the Dell 2950-III and could not be happier with 2.6.

Originally Posted by cciesec2006

#2: Do NOT buy Sun X4200-M2 and/or Dell 2950-III because these boxes do not
support SPLAT 2.4 kernel.

This is to be expected as the HCL clearly states that only R65 2.6 will run on these platforms.

Originally Posted by cciesec2006

#3: IBM x3650 is the best platform, IMHO for NGx R65.

I guess I would differ with this statement. There are plenty of great platforms for R65. Dell, IBM, and HP all have excellent offerings. The workhorses that I see routinely are the Dell 1950/2960, IBM 3650/2550, and the HP DL 380. All are great platforms.

Originally Posted by cciesec2006

#4: Stay with the SPLAT NGx R65 2.4 kernel. IBM Firefly uses 2.4 kernel.

In my experience, if you need to use the 2.6 kernel because it will support a newer platform, then use it. I do not consider the 2.6 kernel to be that risky based on my customer's experience.

Originally Posted by cciesec2006

#5: Checkpoint Secureplatform PRO is rather unstable, IMHO, especially with
multicast

Have not had a chance to test #5.

Am hoping that one person's experience is not going to keep other people from persuing the 2.6 kernel if they need it for hardware support. If you current hardware is supported by the 2.4 kernel, then stay with 2.4. However, SPlat will be moving more and more towards that 2.6 kernel so expect to see a shift from 2.4 with the new versions. Always make sure you consult the HCL before purchasing a new platform. If you need to test 2,6, then get in contact with your reseller or Check Point SE and they should be able to get you 30 day evals.

Regards.

[cciesec2006]

Originally Posted by fireverse

I have a very very large customer (one of the largest privately held companies in the world) running P1 on 2.6 and their gateways on 2.6 for four months, with no serious issues. The only issue they encountered so far was a RADIUS authentication problem with Provider-1. This was corrected with a Hot Fix. So I guess I have to say that my experience has been dramatically different what was posted. The customer is using the Dell 2950-III and could not be happier with 2.6.



This is to be expected as the HCL clearly states that only R65 2.6 will run on these platforms.




I guess I would differ with this statement. There are plenty of great platforms for R65. Dell, IBM, and HP all have excellent offerings. The workhorses that I see routinely are the Dell 1950/2960, IBM 3650/2550, and the HP DL 380. All are great platforms.



In my experience, if you need to use the 2.6 kernel because it will support a newer platform, then use it. I do not consider the 2.6 kernel to be that risky based on my customer's experience.



Have not had a chance to test #5.

Am hoping that one person's experience is not going to keep other people from persuing the 2.6 kernel if they need it for hardware support. If you current hardware is supported by the 2.4 kernel, then stay with 2.4. However, SPlat will be moving more and more towards that 2.6 kernel so expect to see a shift from 2.4 with the new versions. Always make sure you consult the HCL before purchasing a new platform. If you need to test 2,6, then get in contact with your reseller or Check Point SE and they should be able to get you 30 day evals.

Regards.

Item #1: do not take my world for it. Meplia is having similar issues with
2.6 kernel as well. Funny thing is that Checkpoint did not believe him.
I have to contact Checkpoint TAC to tell her that I have the same issue
with 2.6 kernel as well. It is a confirmed issue with Checkpoint TAC now.

Item #2: If Dell, HP and Sun are such great products, then how come
Checkpoint decided to use IBM x3650 for the Firefly product?

Item #3: If 2.6 kernel is such a great thing, then how come checkpoint
Firefly is still on 2.4 kernel? Remember that x3650 supports both 2.4 and
2.6 kernel. Why is CP Firefly still on 2.4 kernel if 2.6 kernel is such a great
thing ?

Item #4: Do you push a LOT of traffics through the firewall with 2.6 kernel?
When you do that, you will see Checkpoint cpd process go to 100% utilization
on one core while the other cores sit idle. After that, SIC is lost between
the gateways and CMA. This is confirmed. I never had such issue with
2.4 kernel.

Hopefully someone working for CP can explain item #3 and settle the 2.4
vs. 2.6 kernel.

[Pascal]

Really interesting threat about SPLAT with kernel 2.4 or 2.6. Thank you cciesec2006!

I have updated our P1 management three months ago and we haven't had any issues yet.

However we try to migrate one of our Nokia cluster to SPLAT 2.6 and we still have issues with these systems (2x HP DL585 G2) since march!

We had different issues and when one can be closed another issue appears and so on... At the moment we have had problems with CCP-multicast-mode (which is Default CCP mode). When I change CCP protocol to broadcast all seems to be fine. I don't know why the problem occurs with multicast-mode and I hope that Check Point can solve this issue this week. It's really strange for me!

Unfortunately we have to use SPLAT with kernel 2.6 because our system is only supported with this kernel :-/

Regards
Pascal

[ccie16798]

hello

I just think SPLAT 2.6 is not mature yet; we did performance tests on firefly M8 (one coreXL image and one SPLAT 2.6 image) and under load splat 2.6 uses 100% of one CPU and 0% on three others....
at the same time, load was nicely distributed on corexl 2.4 kernel.

that said, 2.6 is clearly the future; 2.4/RHEL3 is supported till 2010 so they have to make the move; I just wish they give us 64bit SPLAT kernels, as all CPU are, and checkpoint kernel memory is limited to 2G

by
etienne

[chillyjim]

NGX R65 will likely be the last major release of the 2.4 kernel.

As for a 64 bit build of the FW kernel, there just hasn't been enough call for it. Policies that are large enough to be an issue tend to have other problems (namely being poorly written).

[fireverse]

Originally Posted by cciesec2006

Item #1: do not take my world for it. Meplia is having similar issues with 2.6 kernel as well. Funny thing is that Checkpoint did not believe him. I have to contact Checkpoint TAC to tell her that I have the same issue with 2.6 kernel as well. It is a confirmed issue with Checkpoint TAC now.

Not knowing the exact configuration from the BIOS, to the switches, to the OS and Check Point application itself, I guess it would be hard to say what the exact problem is you are experiencing. I posted on this thread to share my experience with 2.6, with one of the largest companies in the world running it on both P1 and their gateways. Their experience has been mainly positive. My customer is using different hardware (Dell 2950-III), and a different configuration with regards to clustering.

Originally Posted by cciesec2006

Item #2: If Dell, HP and Sun are such great products, then how come Checkpoint decided to use IBM x3650 for the Firefly product?

I suppose Check Point had to choose one so they chose IBM for the IAS. It was probably more of a business decision, but I can only guess at this point. In the past Check Point used Dell for the Interspect, and Connectra platforms. HP (until a year ago) had a bundled Check Point firewall product. The UTMs (450, 1050, 2050) are based on Crossbeam. There are a lot of good choices for platforms.

My customer list is mainly US Fortune 500, 100, and 25 in high performance environments. They all use either HP, IBM or Dell for their firewall platforms. It doesn't seem to matter what vendor is chosen, they tend to perform about the same. There are very detailed performance guides for most of the platforms on the HCL. These are internal documents, but if you know someone within Check Point they may be able to share that info with you. I have seen them, and the performance numbers are pretty close across the different Intel platforms.

As I indicated earlier the most common platforms I see are the 1950/2950, 2550/2650, and DL380. The fact that Check Point chose the IBM for IAS doesn't say to me that that is the now uber-platform dujour. I think you are getting hung up on the fact that Check Point uses IBM for the IAS, and mis-interpreting that as meaning that the 3550 (M6) and 3650 (M8) are now the best platforms out there.

Originally Posted by cciesec2006

Item #3: If 2.6 kernel is such a great thing, then how come checkpoint Firefly is still on 2.4 kernel? Remember that x3650 supports both 2.4 and 2.6 kernel. Why is CP Firefly still on 2.4 kernel if 2.6 kernel is such a great thing ?

If you read the release notes for R65 2.6 it states:

"SecurePlatform 2.6 for NGX R65 expands hardware support."

Since SPlat 2.6 doesn't expand anything for the IAS (as it is already supported with 2.4) that is probably the biggest reason right there. Why install it if it is not needed?

Also keep in mind that the IAS is built per the customer specifications. You can order up an IAS with R60, R61, R62, or R65. So unless a customer specifically requests R65 2.6, they are going to get a 2.4 build. I am guessing there are not a lot of people (if any) that would need R65 2.6 on IAS as it doesn't give them anything additional.

Originally Posted by cciesec2006

Item #4: Do you push a LOT of traffics through the firewall with 2.6 kernel? When you do that, you will see Checkpoint cpd process go to 100% utilization on one core while the other cores sit idle. After that, SIC is lost between the gateways and CMA. This is confirmed. I never had such issue with 2.4 kernel.

I guess YMMV, as I am not seeing this issue with my customer and their 40+ firewalls. I do appreciate what you have reported, and as I have another major customer moving to 2.6 I will be watching to see if they have any of the issues you have reported. I am not here to turn this into a flame war, but clearly your experience is not the definitive answer with regards to the IAS (3650) platform and 2.6.

I hope this thread doesn't give people the impression that:

1. The IAS is the new defacto platform for SPlat.
2. The IAS is the best platform when it comes to performance and support.
3. That the 2.6 kernel is unstable and unusable in production.

From what I personally am seeing in the field, the three items above are not true.

[chillyjim]

I will attempt to clarify this as best I can. Please do not take this as an official Check Point statement, it is just what I have been able to gather.

Originally Posted by fireverse

Not knowing the exact configuration from the BIOS, to the switches, to the OS and Check Point application itself, I guess it would be hard to say what the exact problem is you are experiencing. I posted on this thread to share my experience with 2.6, with one of the largest companies in the world running it on both P1 and their gateways. Their experience has been mainly positive. My customer is using different hardware (Dell 2950-III), and a different configuration with regards to clustering.

R65 SPLAT 2.6 is functioning at many sites without issue. There are some sites/hardware combos that are having issues with it.
This version should be used if required for environment, but if you don't need it, don't use it.

SPLAT 2.6 will be what the next full release of the Check Point Suite is based on.

I suppose Check Point had to choose one so they chose IBM for the IAS. It was probably more of a business decision, but I can only guess at this point. In the past Check Point used Dell for the Interspect, and Connectra platforms. HP (until a year ago) had a bundled Check Point firewall product. The UTMs (450, 1050, 2050) are based on Crossbeam. There are a lot of good choices for platforms.

The choice of IBM was in part a business choice (Political) and in part technical. The technical part had to do with early support for IRQ swizling.
Dell is still being used for some of the Connectra appliances because they have a very easy OEM program to work with.

The UTM-x50's are not based on the Crossbeams, but they do come from the same OEM. Crossbeam was also hired to perform logistics until Check Point was up to speed on it.

My customer list is mainly US Fortune 500, 100, and 25 in high performance environments. They all use either HP, IBM or Dell for their firewall platforms. It doesn't seem to matter what vendor is chosen, they tend to perform about the same. There are very detailed performance guides for most of the platforms on the HCL. These are internal documents, but if you know someone within Check Point they may be able to share that info with you. I have seen them, and the performance numbers are pretty close across the different Intel platforms.

True, the largest differences are in the PCI buses.

As I indicated earlier the most common platforms I see are the 1950/2950, 2550/2650, and DL380. The fact that Check Point chose the IBM for IAS doesn't say to me that that is the now uber-platform dujour. I think you are getting hung up on the fact that Check Point uses IBM for the IAS, and mis-interpreting that as meaning that the 3550 (M6) and 3650 (M8) are now the best platforms out there.

The IBM IAS (M2, M6, M8) are the platform of choice for Check Point now. This is the standard development platform, and what is being sold the most.
You best price/performance/support will differ depending on your circumstances, but 9 out 10 times I'm telling people to use the M6/M8 and not just because I get more money.

I guess YMMV, as I am not seeing this issue with my customer and their 40+ firewalls. I do appreciate what you have reported, and as I have another major customer moving to 2.6 I will be watching to see if they have any of the issues you have reported. I am not here to turn this into a flame war, but clearly your experience is not the definitive answer with regards to the IAS (3650) platform and 2.6.

I think this holds true for any platform. Personally I've had a lot of good luck with SUN. I really wish that they had not dropped out of the IAS program, but that was a SUN decision and not a Check Point.

[fireverse]

Originally Posted by chillyjim

The UTM-x50's are not based on the Crossbeams, but they do come from the same OEM. Crossbeam was also hired to perform logistics until Check Point was up to speed on it.

Same Crossbeam OEM, and Crossbeam is handling the RMAs. Guess I should have been more clear with this statement. The key words here are "based on." I probably should have written:

"The UTMs (450, 1050, 2050) are based on Crossbeam's program."

Walks like a duck, quacks like a duck...it's based on a duck.

Originally Posted by chillyjim

The IBM IAS (M2, M6, M8) are the platform of choice for Check Point now. This is the standard development platform, and what is being sold the most.

True, it has been said that IAS is the platform for R+D. At the same time Check Point is continuing to update the HCL for other Intel platforms. Not saying this is a contradiction, just showing there are options. AFA "being sold the most" I guess it depends....

Disclaimer: This is not an official Check Point response either and is based solely on information that is made available to the public.

[cciesec2006]

I am here to flame anyone either. We're all here to learn new things.

I've been told by my Managed Security Service Providers, MSSP, who managed
fortune 100, 50, 25. The MSSP has multiple P-1 systems across the globe and
this is what they told me:

"Unless you have a specific needs to use SPLAT 2.6 kernel, we strongly
recommend that you stay with 2.4 kernel"

When I asked them: but my hardware supports both 2.4 and 2.6 kernel, why
should I NOT go with 2.6 kernel?

MSSP response: because 2.6 kernel is not in the mainstream release yet.
Therefore, we can not guarantee the stability of 2.6 kernel.

When MSSP makes statements like that, I will prefer 2.4 kernel.

[ccie16798]

Originally Posted by chillyjim

NGX R65 will likely be the last major release of the 2.4 kernel.

As for a 64 bit build of the FW kernel, there just hasn't been enough call for it. Policies that are large enough to be an issue tend to have other problems (namely being poorly written).

The problem is not with large policies, but with large connexion table;
secureXL + NAT = 1Ko/conn, +4Ko if HTTP (http buffers).
with 32bit kernel, you cannot go far beyond 300k HTTP connexions, 1,2M TCP connexions

Etienne

[Pascal]

On first july it seems that Check Point has "fixed" my issue with CCP-multicast (Default CCP protocol), but for me it's just a workaround - why?

Machine is: HP DL585G2 with SPLAT Pro (kernel 2.6)

After I have had problems with HA transfers from member A to B and reverse I opened another case at CP support (#1 out of #3).

Problem occurs when I stop member A via SmartView Monitor, B becomes active and after that I started member A within my cluster again. Member A is active but status is "active attention" and "cphaprob -a if" shows that 18 interfaces, all NICs of that machine, are required. Before that only 5 interfaces are required by cphaprob.

So CP refers me to SK30060, which will be enhanced to cover our problem, that I should use $FWDIR/conf/discntd.if to disable all unused interfaces from monitoring - fantastic!

Now my decision is to switch to CCP-broadcast mode and all works fine! No issue and no problems at all.

Regards
Pascal

[lbraid]

I'm setting up R65 multicore clusterXL on to x3650 machines, in primary/secondary setup.

I'm having issues when testing failover, 2 out of 6 interfaces will cause failover to secondary. The other 4 interfaces do not. When doing cphaprob state it show primary as up with attention required and the secondary as down.

does anyone know a fix for this issue?

Regards

Lee

[rubber_chicken]

Originally Posted by fireverse

Walks like a duck, quacks like a duck...it's based on a duck.

Thats the 2nd funniest thing I've ever read in these forums. Quality....