Crazy Recurring Problem

[benhabing]

:::Also posted under Miscellaneous:::

I am fairly new to the checkpoint line (cisco background) We are having a VERY strange issue happen randomly. Hoping someone might be able to tell me what could possibly cause this.

We are on NGX R65 HFA02, new installation on CP UTM-1 2050. We're running BGP on our external interface, have multiple subnets behind internal interface, web dmz interface, client network, etc...

First occurred about 2 weeks ago... Get a call about our customer support department can't connect into our client hosting network (different subnet routed through CP). I try to access vpn and terminal server from outside (of course it's a saturday) can't connect to either, they are on different subnets behind the internal interface. grab my laptop, jump in the car, race downtown, get into the office start running some diagnostics and figure out all of our VPN tunnels are working, but we can't get to any subnet routed via the CP box. After about 2.5 hours everything returns to normal on it's own. ??? I call Checkpoint support, run cpinfo, fw monitor, etc... the think it's related to a memory leak BTW there's a fix for it. We go ahead and install hotfixes CP tech recommends. So far so good, with the exception of losing our BGP config during one of the reboots. Everything runs fine for about 2 weeks, then about 2:30am the box decides to do the same thing again for about 3 hours this time. About 40 hours later it does it again, this time for 4.5 hours. Each time the box comes back up just as quickly as it goes down. (Note: during these "outages" can't connect to WebUI or SmartDashboard, etc...) I was out of the office, and started to do some ping tests to our two internal hosts, notice that I'm getting about 19% packet loss and avg latency of about 800ms.

Last time it occured was about on Monday and it was down for 6 hours!!! They crazy thing about all of this is that NONE, I repeat, NONE of our VPN tunnels are affected.

We've checked cables, ISPs, traffic on network, done a dozen CPINFOs, top, vmstat, df, etc..., talked to CP support multiple times, they are scratching their heads as well. The only thing we've found is ksoftirqd has about 6x the CPU time as anything else.
We're about ready to take the box up to the 24th floor............

For any of you checkpoint experts, what could cause the network to go to crap, but the vpn tunnels, and traffic to our web dmz interface be unaffected???

Any ideas are greatly appreaciated.

Thanks,
Bald in Canada

[RayPesek]

Get a call about our customer support department can't connect into our client hosting network (different subnet routed through CP).

Does this mean they are behind the internal interface and trying to get to another subnet off a different FW-1 interface? One that does not go across the Internet?

...but we can't get to any subnet routed via the CP box

From where? From the firewall itself via a console cable? From the internal interface?

Would it be like the next-hop inbound router cannot reach the internal interface of the firewall?

Ray

[benhabing]

Originally Posted by RayPesek

Does this mean they are behind the internal interface and trying to get to another subnet off a different FW-1 interface? One that does not go across the Internet?



From where? From the firewall itself via a console cable? From the internal interface?

Would it be like the next-hop inbound router cannot reach the internal interface of the firewall?

Ray

|
External
|
Checkpoint UTM-1
| | |
Internal Client WebDMZ
| | |


So when the issue occurs the following behaviour is noted

1. Can't connect to WebUI or smartCenter on Internal Interface
2. Can't connect from Internal Interface to Client Interface
3. VPN traffic goes from external to Client (no latency)
4. External traffic to WebDMZ is fine (port 80;443;21
5. Internal traffic to any other interface is crap, pings over 32 bytes times out
6. Pings to Internal interface are very high in latency avg. 800ms

Hopefully that gives you a clearer picture. Thanks for the reply.

[RayPesek]

So only traffic to or from the "internal" interface is affected? When this occurs, can you get on the firewall somehow and ping or traceroute to and through the next-hop inbound router?

In fact, is there a router between the internal interface and the LAN? If not, what's there?

This sounds more like a networking problem than a firewall problem.

Ray