Experiences with Crossbeam C-Series

[tomsite]

About time that somebody puts some experiences down with Crossbeams C-Series, and especially the C10s.

Having evaluated the C10s last year we decided that it was usefull enough. Everything we setup and tried worked. However after the decision was made to use these appliances the problems started. Based on 4 C10 devices we had 6 replacements of the complete Hardware already and the 7th replacement is waiting now. Main issues were here a) the wrong IDE cable caused the HDDs to become corrupted (cause of 4 replacements so far, but read on), b) we had two devices with failing switching engines. The intial problem with the wrong IDE cables was rectified by Crossbeam by replacing our entire base of Crossbeam C10s including customer systems, even though Crossbeam made an entire mess of this replacement (but also partially caused by our warehouse sending new devices back and keeping the old ones). Now we have another device again failing with an unrecoverable HDD error despite that this is already new HW with the fixed IDE Cable. All in all one can count on the fact that every 6 months at least one device has to be replaced. Our environment is not cause of the problems (everything is controlled and fully climatized)

We also asked for support for BGP and other dynamic routing protocols, got a so called Beta Code of a routing daemon, which i frankly would not even have released as Alpha Code. it was the most buggiest piece of SW i have seen since a long time (even more buggy than Windows ;o)). Generally Crossbeam Support is useless. The guys don't even know what a computer is, let alone that they know anything about their own products.

All in all i can only recommend to not touch these devices with a barge pole. It is really not worth the hassle. I would not even recommend these appliances for test labs or as 'play-around' kit and rather implement a Watchguard SOHO FW (sorry for the use of the swear word Watchguard and the mentioning of Watchguard and Firewall in the same sentence) and poke my own eyes out with hot needles.

C30s: This is a different fish alltogether. So far the C30s we have run relatively stable, even if not very smooth. You always hit a slight snag with Crossbeams. All of a sudden active/active fails for no apparent reason, however a reboot helps in most cases. For no apparent reason the trust between module and Mngmt Server is lost and one has to re-establish SIC. No indication in any logs as to why. Best one on this is that i was asked beginning of last year by Crossbeam support what SIC is.

X-Series: After the problems we had already with the C-Series we did not go any further down the Crossbeam track and are throwing all Crossbeam devices out. It will be a huge pleassure for me to take a sledge hammer and fix these boxes into something arty and send the bits and pieces back to Crossbeam as one huge pile of rubbish. For me it will be a 'Crossbeam? Never, ever again', even if somebody offers me shed loads of money to use them.

If you plan to use any Crossbeam HW i suggest that you do a very thorough evaluation (minimum of 8 weeks in average to heavy load environment) if the device is fit for purpose in your specific scenario, and if possible use Loadtesters to blow the box apart. You will very quickly notice that the more traffic the box has to handle the more likely it is that it falls apart. Then get Crossbeam to send an Engineer our to verify the design and setup and get it in writing that the appliances are fit for purpose. If anything goes wrong you can kick Crossbeam up the backside and ask for your money back. If you don't do it Crossbeam just turns around and shows you the middle finger. Important!!!!: Get it in writing and signed by Crossbeam! I can't stress it enough. Crossbeam is interested to get your money, but they are by far not interested in customer satisfaction, all you get of them is a scottish greeting (kick in the privates and a headbutt) and afterwards a moonie (for our foreign Friends: A Moonie. Scottish thing where you show your bare bones backside, preferably whilst wearing a kilt with nothing underneath).

Crossbeam as alternative to Nokia: If you want sleepless nights, lots of support calls and if you like to gamble the Crossbeam plattform is certainly a good choice. The chances that you get a fully working device is not any better than the chance to win the lottery.

Tom

[RingBuffer]

Tom,

I find it interesting that yours is the only post in the Forum regarding Crossbeam as a platform. It speaks volumes I think.

Let me set the background to my following comments:-

I worked at Check Point for over 6 years before going to Crossbeam. Boy, did I make a mistake! This company has no ambition and no security heritage. The heritage if you are interested is Bay and Nortel.

Most of the designs of both C and certainly X are Bay based engineering. Ask yourself one question, where are they right now?

Crossbeam seem to be hanging one for one of two things to happen, 1) Check Point buy them; 2) They IPO.

Lets take the last point first, they don't have the products in dev to make an IPO viable. They have been talking 10Gig since June last year. The current X-series would require forklift upgrade to make the most of any 10Gig offering due to backplane changes.

On the first point, Nokia customers are disaffected, Check Point don't have a hardware offering and so in their blinkered ignorance, Crossbeam think that Check Point will buy them to finally seal Noki's fate. When hell freezes over! Check Point will continue to be a software company with wide hardware coverage. They can't afford to buy Crossbeam because it injures forever their precarious relationship with Nokia, and thats 45% of their installed base.

Lets take some C-series experiences. At least one ISP will never buy C-series again based on their reliability record. The disk errors you mention are not the end, nor the start of the issues. Another academic customer who experienced another more serious problem was GIVEN a second C30 for free to hush things up in the academic community.

Firstly, Crossbeam (in common with many manufacturers) don't make their own kit, so they are beholding to the contract manufacturer to ensure quality. Problem with that is the manufacturer is in Taiwan and only one person at Crossbeam HQ speaks Taiwanese, and they are in Marketing! This has given rise to many quality challenges.

The C30 uses a Network Processor to offload the CHKP SecureXL processing, except it doesn't really work. If you cluster the c30 then you must disable hardware SecureXL. This takes a 4G box down to 1G immediately. I am not going to suggest you lift the lid on a C30 to check this out, but if you do you will see that the C30 NP has a 1GB ethernet connection to the Pentium host board. So you have 18 1Gb copper and 2 1Gb fiber ports all talking over a 1Gb ethernet connection. Even Cisco doesnt oversubscribe like this.

You would hope Crossbeam could fix this? Unfortunately the engineer who wrote the code doesnt work there anymore and no-one understands his code. Look for the C30 to be EOL before long.

The C30i is the baby to go for. At least it's NIC cards are PCI based and so have a faster connection to the host processor. Oh and thats the real point. You are paying $17k for what is essentially a PC. The C30i contains absolutely nothing you couldn't build yourself using SPLAT or RHEL.

Crossbeam shipped a SMP kernel for its dual cpu C30 variants. It didn't work, kept crashing. Fix from Support? Use the Uni kernel, we'll fix it eventually. Hence the free C30 to the academic customer above!

Now onto the falsehood of UTM appliances. Why is the Crossbeam solution special in this regard? It is simply a PC, with a desktop, not server, chipset. It runs a variant of RH Linux. The reason it's UTM? Well you can run more than CHKP on it (this is its only differentiator from Nokia), but you still have to buy the license and install the product.

Haven't security professionals been saying for years (including everyone at CHKP), DON'T run other programmes on a firewall???

Onto the disk errors. These devices, unlike the latest Nokia IP range do have disks, disks break. To make matters worse, early revisions used no vibration supression on the guard disk mounts. All the range use a laptop IDE disk drive which is neither quick, nor designed for server operation. Later versions still have disk issues because 40 pin and not 80 pin cables were used for the drive. This lead to ECC errors which meant swap out of the appliance. Crossbeam claim to have fixed these issues, but unless you have a very late model (Jan this year onwards) then you will eventually have a failure. Their first attempt to fix it was to disable the IDE mode that required 80 pin connectors in SOFTWARE. This slowed the disk even more.

All in all, a customer support nightmare. The C-series should be dismissed and Secureplatform used to build your own server out of a known reliable platform such as HP or IBM.

The X is a different beast with its own problems.

[Lackie]

Very good reviews and comments from both parties.

Guess the questions would be...

Tom, what platform did you move toward when you tossed Crossbeam?
RingBuffer, do you still work for Crossbeam? :)

[RingBuffer]

>>RingBuffer, do you still work for Crossbeam?

Fortunately, no.

I should take up the point about BGP. Crossbeam charge you for advanced routing support that is flaky at best, not internally developed and poorly supported by the vendor. If you want BGP/OSPF, go for Nokia (or SPLAT)

On the support issue - the support guys are very good people, just stuck in the world of Bay and Nortel switching and routing... Their upper management are blinkered to the issues and so they do the best they can in that environment.

I will post a snippet of a memo which shows you the understanding...

8<-------------------------------

The difference between the C10 / C30 generation and the new generation is tremendous.

C2 has been designed under quality requirements. Nexcom, the ODM delivers a product which has been hardly tested. Burning tests, Software Tests, Hardware tests, Vibration test, cold tests (-40), Heat test (+80). That baby is solid. The disc is strong. Well fixed. Crossbeam Engineering and Manufacturing learned from the past and did a great job.

SQA did a great job as well. 161 bugs have been found and fixed. A lot of testing has been done in customer set up.

C2 is 3 time faster than IP260 on small packets. C2 can be a FW box, a VPN box, an IPS box. C2 has fail open capability. It is a very impressive baby with a strong personality.

Today we looked inside the box. There are many hardware details which have been built to make it robust. The disc is protected against vibration. It is designed for industry environment. It is strong. The CPU got a special fixation to get better protected against heat. The cabling has been optimized. The connectors have been designed for solidity. The chassis can be rack mounted and is solid as well.

The marketing launch program has been successfully started. The world is just waiting for that baby.

The WW SE team will see C2 over the next 2 weeks in Concord. Please look at his power, take it and make it strong in your local market

Today Nokia announced a big lay off in their security department. 130 people have been made redundant. A large part of their engineering is gone. Their VPN group is gone.

Look at it and convince yourself how nice it is and talk about it everyday to all your channels and customers. C2 will make you happy and will give you a lot of fun.


8<-------------------------------------------

make sense?? This is a US company remember. I copy this verbatim, with no changes.

I should say in their defense; if Crossbeam get SPLAT on the C2 at least, then this will be a credible platform.

I should also correct the mistake I made in my first post, C30 is 16 10/100 and not 18 1Gb as I said.

RB

[Christofer Hoff]

Hi all.

My name is Christofer Hoff and I am Crossbeam's Chief
Security Strategist. I was also a Crossbeam customer
for 3 years before joining the company, so I think that I
have an interesting perspective on the things written above.

This isn't going to be a fancy email with all sorts of squirrelly
slick marketing responses bent on damage control. It is a
reasonable and rational response to a post which contains
several inaccurate statements.

Like any solutions provider, we've had our hiccups and had
issues we've worked hard to resolve. Do we have issues?
You bet. So does every other company on the planet.
We work very hard to make sure they are mitigated quickly.

We certainly do everything we can to take care of our customers.
Doing what we need to do to make sure they are happy with our
products is what companies ought to do, no?

Conspiracy theories are fun to read, but reality is often much more
boring ;) The points raised above fail to communicate the reality of
our company, our employees, our products and our customers.

We have companies that have selected to deploy hundreds
of our boxes globally as part of their overall security strategy after
live trials in their production environments.

These customers are some of the largest companies in the
world; they don't make decisions lightly and they don't buy
products that don't work or work as poorly as is described
above.

In terms of the other accusations relayed above regarding our company,
they are not worth dignifying since they are neither constructive,
appropriate, or accurate. Grinding axes on the Internet is a waste
of time. Nobody wins.

If anyone would like to discuss our products openly in
a rational and fair manner, I look forward to doing so.

Regards,

Christofer Hoff, CISSP CISM CISA
Chief Security Strategist
Crossbeam Systems, Inc.

[RingBuffer]

Hello Christopher,

I should firstly congratulate you on your appointment at Crossbeam. I hope your first few weeks has been interesting and rewarding.

Your comments, whilst interesting in themselves do nothing to dispel the original poster's issues, who as a Crossbeam Customer *IS* disaffected by Crossbeam.

As a Crossbeam Customer you were exposed to X-series which I said was a different beast than C, and largely unaffected by the C quality issues.

Perhaps you could address the real problems with the C-series that the original poster had and that I merely added to with truthful comment and NOT anecdotal comment. Can you deny the C10 and C30 had disk problems? Can you deny that the Dual-CPU C-series had a crippled kernel that did not support SMP? Can you deny that the support fix was to boot the UNI kernel? Can you deny that the NP on the C30 cannot be used with CHKP ClusterXL?

If you can refute all these comments then I take my hat off that the issues have finally been fixed, however, merely pointing to the new C2/6/12/25 do not alleviate existing Customer issues.

RB

RB

[abboud]

Unlike many above, we have had a problem-free 1 year period with our C10 since we bought it. Based on that experience, we just recently decided to replace 6 Nokia 130s with C2 platforms.

We were always extremely happy with Nokia's support, especially the fact that we could get a single point of support for hardware and checkpoint software. However, Nokia's platforms always seemed limited in memory expendibility, which would always force us to replace them with newer hardware because of increasing checkpoint memory demands.

We had a long discussion with the Crossbeam sales team, indicating plainly that any technical advantages that a C2 might have over an IP260 would not justify a switch unless crossbeam matched and exceeded Nokia's helpful and knowledgeable support. Crossbeam assured us that they will always deliver. We now have six C2 boxes sitting in our IT server room and placed the first phone call to Crossbeam. crossbeam tech support was adamant that even under the most expensive level of support, they will not answer any question pertaining to the checkpoint management station (something that Nokia routinely supports).

So it would appear that if one wants to be a crossbeam client, one has to give up on having a single point of support for checkpoint. We are now seriously considering returning the six C2s we bought and instead buying IP260s.

I would be interested in other's perspective on crossbeam's support.

[RingBuffer]

Interesting.

I am glad you have had no issues with your C10, and I am sure the C2 will be pain free.

Support is always a thorny issue, even for Nokia. In the early days they wouldn't take calls on CHKP issues unless it was an integration issue. Perhaps this is now Crossbeam stance?

RB

[Lackie]

Nokia is far from that now. They will support the Check Point software as long as it's pertaining to the Nokia somehow. This includes on your Windows management station if it's pushing a policy to a Nokia somewhere in there. If needed they will also escalate any problem up to Check Point directly as they are a partner with them.

So if Crossbeam doesn't do anything support related when it comes to the Check Point software, that would mean that you would need to additionally purchase support from Check Point as well.

Not sure about anyone else but I like the single point of contact, that way you don't get into the argument that one vender accuses the other of having the problem.

[Christofer Hoff]

Hello again. I will try to answer these questions from
multiple posts in somewhat of a relevant order:

1. Thanks for the kind words, Ringbuffer. I've actually
been at Crossbeam for almost 7 months, so my first few 28+
weeks have been very interesting and rewarding ;) By the
way, I purchased both X and C Series products at my prior
place of employment.

2. BGP has not been offered or supported on the C-Series. I don't know from
whom Tomsite obtained the routing code he refers to, but I show no
EA releases for any sort of 'beta routing code' for the C-Series
in out ticketing system. I will personally continue to query our field engineers
as well as support staff and I would be more than glad to research
this further to clarify/resolve, however. It just seems odd to me.

Onto the more pertinent issues:

3. Can you deny the C10 and C30 had disk problems?
No, as the C10 did have disk problems, but the C30/C30i did not.
We have remedied the faulty equipment by replacing any identified or affected
drives with a more robust drive unit from a different manufacturer. Further, we also
cleared all distribution stock to make sure no customers received a
unit with the faulty drive. As to the software issue you referred to
this was fixed in COS 3.6.

4. Can you deny that the Dual-CPU C-series had a crippled kernel that did not support SMP?
Yes, as your statement is inaccurate. This issue is not an issue of a "crippled kernel"
but rather an issue where the Dual-CPU C-Series did not support SMP with Check Point FP3
due to an application incompatibility. The newest versions of Check Point's products (R55, R55W, R60
(NGX) and R61) are fully supported and working with SMP kernels. It was a bug. It was
fixed. The workaround (see #5) is no longer required.

5. Can you deny that the support fix was to boot the UNI kernel?
No, as this was the most prudent response until we found the root
cause and fixed it. See above. At this point, any C-Series that includes dual
processor support is fully qualified for dual CPU and Check Point firewall.
Just to be clear, dual-CPU is beneficial only for security servers, logging
and VPN acceleration.

6. Can you deny that the NP on the C30 cannot be used with CHKP ClusterXL?
Yes, as your statement is inaccurate. To make sure everyone is on the
same page, there are three basic acceleration options available:

a) CheckPoint Performance Pack – software based acceleration coming from
CKP, available on all C-Series for an extra license cost.

b) Crossbeam NIM, hardware accelerator on C30 – hardware based acceleration
from CBS, available on C30 for no extra cost

c) Crossbeam FAM, software based acceleration on NIM less C-Series -
available from CBS for no extra cost

You cannot utilize Crossbeam’s NIM, SecureXL and ClusterXL simultaneously
on the C30 in “High Availability” and “Load Sharing Multicast” mode with R55 and
NGX. VPN-1 and FloodGate-1 are not supported together with any SecureXL enabled
– this is a limitation from CKP.

The C30 and NIM acceleration can be used with CHKP ClusterXL in “High Availability”
and “Load Sharing Multicast” modes.

Crossbeam’s SecureXL implementation does not include the ClusterXL module
for acceleration. Therefore SecureXL would have to be disabled to run ClusterXL
on the C30. ClusterXL works on all other C platforms.

7. As to the issue regarding support of Check Point products, I believe there
may have been a misunderstanding depending upon your level of support. If you can
PM/eMail me with your case number, I will research this issue.

To be clear, our support engineers will actively work with a customer and their Check
Point installations inasmuch as the Check Point issue at hand can be related to an issue
with their supported Crossbeam hardware (such as in the Nokia examples above.)

I trust I have answered all of the outstanding questions and clarified those issues
raised above. Should you have any issues, please feel free to contact me at your convenience
and I -- or anyone on the Crossbeam team -- will make sure to get anyone the answers they
need, quickly, accurately and truthfully.


Regards,

Chris

[RingBuffer]

Christopher,

Good answers and ones I certainly agree with, to an extent. I don't have an axe to grind, far from it, just wanted a truthful response from Crossbeam to the frustrations of a Customer. That's all I ever wanted in fact.

Anyway, hopefully the original poster will now contact you to have his issues looked at.

Just to concur with a statement you made right at the end, the Support people at Crossbeam are helpful, truthful and honest. I enjoyed working with them and would never disrespect anything they do.

I am encouraged that a more professional approach seems to be instilled in Crossbeam with your arrival. If it is 28 weeks, then the Press release got lost somewhere :o) http://www.crossbeamsys.com/press_040306.asp

RB

RB

[Christofer Hoff]

Originally Posted by RingBuffer

Christopher,

Good answers and ones I certainly agree with, to an extent. I don't have an axe to grind, far from it, just wanted a truthful response from Crossbeam to the frustrations of a Customer. That's all I ever wanted in fact.

Anyway, hopefully the original poster will now contact you to have his issues looked at.

Just to concur with a statement you made right at the end, the Support people at Crossbeam are helpful, truthful and honest. I enjoyed working with them and would never disrespect anything they do.

I am encouraged that a more professional approach seems to be instilled in Crossbeam with your arrival. If it is 28 weeks, then the Press release got lost somewhere :o) http://www.crossbeamsys.com/press_040306.asp

RB

RB

Thanks. I very much hope and encourage that should anyone ever require anything that you simply call or email me. The team here at Crossbeam really is focused on creating the best products we can and supporting our customers in the best way possible.

We aren't perfect, but we really do strive to be.

In terms of the late press release...well, I never said we were an efficient PR machine. I would rather spend my time focusing on important issues such as this rather than seeing my ugly mug in a press release ;)

Again, if anyone has issues with any of our products that they feel requires
escalation or increased focus, please contact me and I will make sure it gets
the attention it deserves.

Regards,

Christofer Hoff, CISSP CISA CISM
Chief Security Strategist
Crossbeam Systems, Inc.

[abboud]

To update my previous post concerning Crossbeam's support of the Checkpoint Smart Console - Management Station (as opposed to the enforcement module):

I am glad to say that Crossbeam has just informed me that they will support the Checkpoint Management Station under any support contract that would have previously just supported the enforcement module on the C10 and C2 platforms. As far as I am concerned, they now match Nokia's support coverage, and the choice between the 2 is now reduced to technical specs, reliability and price, the way it should be.

[joelmoses]

I thought I might weigh in here as a consumer of both Nokia and Crossbeam hardware. I've had experience with the low, middle, and high-end solutions from both companies, and can say that the Linux-based Crossbeam platform is typically easier to deal with from a support perspective than the Nokia is.

As far as quality goes, we've run into one or two snags on our Crossbeam gear, but nothing as serious as the performance problems on high-rate interfaces we've experienced in later IPSO builds. The latest C-series gear -- specifically the C25 -- looks very good. Dual power supplies, slotted interface cards, and dual mirrored hard-drives in vibration-mount sliding trays. The only real disadvantage at this point is the somewhat anemic dynamic routing support and the lack of a "pretty" configuration interface.

I've always viewed Nokia's refusal to adopt a Linux-based OS as an issue with the platform. Check Point does all their development on Linux first (SecurePlatform saw to that). This forces Check Point to, by necessity, treat the Nokia platform as a "target build" and not a reference platform. Because of this, I've seen situations where Nokia and Check Point, when faced with a particular issue, point the finger at each other and refuse to solve the issue. ("It's an OS issue... It's a Check Point issue...")

[tomsite]

>>Tom, what platform did you move toward when you tossed Crossbeam?<<

Went back to trusted Nokias and funny enough, not a single problem since then. One can say about Nokias what they want, but i found them always stable and never really had any problems. We still have internally some old ip650s running even still with CP4.1. We had to swap one couple of months ago for a new IP380 as i didn't want to support 4.1 anymore and we had a problem. The problem was a nice reason for me to purchase new HW and upgrade to NGX. However the old HW only needs a new IPSO and an update of CP and it will be as new. Don't ask me how old that 650 is, but it is ancient and still doing a fine job. During all my experience (7 -8 years now) with Nokias i came only once across a failing NIC, but due to the design it was fixed very quickly. At an investment bank we did a test how long it takes us to replace an entire Nokia (out of rack, into rack, restore config and policy push from management) with two people. The time was less than 5 minutes even without sweating. And that Nokias run a BSD, well who cares..... it's a *nix in the end, almost the same like Unix, Linux, same like AIX or HP-UX. Sorry, but i worked with all sorts of *nix OSs so nothing scares me anymore and i find my way around on all systems. You simply have to consider that a Nokia is a purpose build appliance, the OS does exactly what is necessary to run CP on it, not more, not less. It does not claim to be an appliance for all sorts of different software. I rather have a box that runs stable than a box that supports all sorts of applications but runs like a dog. And a firewall is a firewall, not an AV Server, not an IDS or IPS or whatever else. Tell anybody to get lost if they want to sell you a firewall appliance and offer to also install ISS or other software on it.

And where a Nokia is overkill simply take SPLAT. Is good enough for most applications.

Tom

[mogwai]

An old 530 I used to work with had a dead hard disk and power supply in the space of 12 months. It was about 3 years old mind you.

Nothings perfect