Crazy recurring Problem

[benhabing]

I am fairly new to the checkpoint line (cisco background) We are having a VERY strange issue happen randomly. Hoping someone might be able to tell me what could possibly cause this.

We are on NGX R65 HFA02, new installation on CP UTM-1 2050. We're running BGP on our external interface, have multiple subnets behind internal interface, web dmz interface, client network, etc...

First occurred about 2 weeks ago... Get a call about our customer support department can't connect into our client hosting network (different subnet routed through CP). I try to access vpn and terminal server from outside (of course it's a saturday) can't connect to either, they are on different subnets behind the internal interface. grab my laptop, jump in the car, race downtown, get into the office start running some diagnostics and figure out all of our VPN tunnels are working, but we can't get to any subnet routed via the CP box. After about 2.5 hours everything returns to normal on it's own. ??? I call Checkpoint support, run cpinfo, fw monitor, etc... the think it's related to a memory leak BTW there's a fix for it. We go ahead and install hotfixes CP tech recommends. So far so good, with the exception of losing our BGP config during one of the reboots. Everything runs fine for about 2 weeks, then about 2:30am the box decides to do the same thing again for about 3 hours this time. About 40 hours later it does it again, this time for 4.5 hours. Each time the box comes back up just as quickly as it goes down. (Note: during these "outages" can't connect to WebUI or SmartDashboard, etc...) I was out of the office, and started to do some ping tests to our two internal hosts, notice that I'm getting about 19% packet loss and avg latency of about 800ms.

Last time it occured was about on Monday and it was down for 6 hours!!! They crazy thing about all of this is that NONE, I repeat, NONE of our VPN tunnels are affected.

We've checked cables, ISPs, traffic on network, done a dozen CPINFOs, top, vmstat, df, etc..., talked to CP support multiple times, they are scratching their heads as well. The only thing we've found is ksoftirqd has about 6x the CPU time as anything else.
We're about ready to take the box up to the 24th floor............

For any of you checkpoint experts, what could cause the network to go to crap, but the vpn tunnels, and traffic to our web dmz interface be unaffected???

Any ideas are greatly appreaciated.

Thanks,
Bald in Canada