Smart Centre Server fails to connect to Enforcement points

[mdp_uk]

Hi

We have a pair of Nokia 380's running in clustering mode. The SCS recently died with no hope of restore. I built a new server and installed the relevent software then ran import _upgrade from the most recent export_upgrade I had, luckily yesterdays! It all ran fine and the rule base looks complete however when I went to test SIC I got the following message:

SIC Status for FW1.madeup.madeup.uk: Unknown

Could not establish TCP connection with 10.0.0.1

** Check that CPD is running on FW1.madeup.madeup.uk and that TCP connectivity is allowed from SmartCenter server to IP 10.0.0.1, Port 18191 **

(Obviously I have omitted the actual IP and hostname)

In a test enviroment I would happily run an unload local and try again and if that failed re-establish SIC but the FWs are happily passing traffic at the moment so i really don't want to break them without being sure the re-establishment will work.

Has anyone come across this problem before? As i understood it as long as the hostname and IP are the same as the original SCS all should be well !!

Any help would be very much appreciated !!!!!!!

[jethrowaya]

Hi,

I would follow the following process.

1) On the Enforcement Module, re-initialize a new SIC key by going to cpconfig and selecting the SIC option. The re-initialization process should be followed by a cprestart.
2) On the production dashboard, open the Firewall object of the enforcement module and reset SIC, then initialize again using the SIC key above.


Thanks
John

[mdp_uk]

Thanks, I have now reset SIC on both nodes. When I reset SIC it tests ok, then when the policy is installed SIC is broken again. I have spoken with checkpoint support and they say that the rulebase is compiled from new every time a change is made to it, ie just adding a node to a rule so it can't be corrupt. So at the moment the only way to install apolicy is to take one of the cluster members down, reset SIC, install the new policy! a big pain!

Has any one ever experience this before?

[RayPesek]

Is SIC saying it's not communicating or is it testing OK after a policy push?

How are you re-establishing SIC? Are you doing a "fw unloadlocal" first? If so, it may mean you do not have a rule allowing the SmartCenter to talk to the firewall.

Ray

[chuachongchee]

Originally Posted by mdp_uk

Thanks, I have now reset SIC on both nodes. When I reset SIC it tests ok, then when the policy is installed SIC is broken again. I have spoken with checkpoint support and they say that the rulebase is compiled from new every time a change is made to it, ie just adding a node to a rule so it can't be corrupt. So at the moment the only way to install apolicy is to take one of the cluster members down, reset SIC, install the new policy! a big pain!

Has any one ever experience this before?

I would assume after you install policy, sic doesn't work? Have you tired to wait for a few mins or so??

Can post your checkpoint versions, hjfa, and hardware specs??

[mdp_uk]

IP380 X 2 running clustering,IPSO 4 load sharing
SCS on Windows 2003 Server
Version R60 HFA05 on all modules

To reset SIC i perform the following:

1. cpstop
2. cpconfig
3. Reset SIC
4. Reset SIC on SCS FW node.
5. Test SIC (All ok)
6. Push policy (successful)
7. Test SIC (Unable to communicate, SIC unknown)

There are implied rules to allow CPD to comminucate with the FW nodes and Vice verser. I have also added a test rule as rule 1 to allow all traffic between SCS and FW.

The point is that it worked before, Checkpoint say that the rule can't be corrupt as it is recompilied after every change to the rulebase, could it be the SCS node? Is it possible to either install a secondary SCS or delete the SCS node?

Any ideas?

[mdp_uk]

I have today rebuilt the Smart Centre Server from new on a new box, new install of operating system with correct hostname and IP address. import_upgrade successful. Reset SIC on both nodes pushed policy again successful. Tested SIC "SIC status Unknown - Could not establish TCP connection with *.*.*.*"

Annoying!!!! Exactly the same error message.

Thanks to anyone who has replied to this thread your assistance is very much appreciated!

[chuachongchee]

Originally Posted by mdp_uk

I have today rebuilt the Smart Centre Server from new on a new box, new install of operating system with correct hostname and IP address. import_upgrade successful. Reset SIC on both nodes pushed policy again successful. Tested SIC "SIC status Unknown - Could not establish TCP connection with *.*.*.*"

Annoying!!!! Exactly the same error message.

Thanks to anyone who has replied to this thread your assistance is very much appreciated!

Have you check cpu load on the firewalls??? After policy install, does cpu spike?? What if you wait like 15mins or so b4 checking sic again... any issues??

Next.. on implied rules... did you allow checkpoint mgmt connections??

[mdp_uk]

CPU is fine, I did add a test rule to allow all traffic between SCS and FWs and Vice verser, same issues. I have now disabled that rule.

SIC sometimes tests ok maybe an hour or two hours after the policy is installed but is soon lost again.

[gladiatorkev]

Hi,

Any luck on resolution of this issue for your environment.?!


Regards

[chuachongchee]

Hmm.... oh yah... can you do a top? Not soo sure on IPSO since it is a freebsd... i once had this issue too.... the cpd process will take up 100% cpu can cause the sic, checkpoint app etc to fail, after i reinstalled the checkpoint app, the issue went away magically...

[cciesec2006]

There is one last hope that you can do. Probably something on the
Nokia is probably corrupted. I would do the following:

1- perform cpstop on both the SCS and Nokia,
2- break the SIC on the Nokia and SCS,
3- perform csstop again on all devices, just to be sure,
4- cd $FWDIR/state on both nokia and SCS;
5- "rm -rf *" on SCS and Nokia,
6- reboot ALL devices,
7- fw unloadlocal on the Nokia,
8- perform SIC,
9- push the policy,

I think it will work after that. Now test your SIC again. I've run into
this issue in the past but on NG Feature Pack 3 on Nokia VRRP, not
IPSO clustering and that's what I did to fix it.

Good luck!!!

[mdp_uk]

Hi thanks for your advice, on the advise of checkpoint I did the following debug on the enforcement points:

fw ctl zdebug -m fw + drop | grep 18191

Basically translated, debug cluster droped packets with filter for CPD packets.

It returned IP address dropped. which when checked turned out to be the NAT address (It manages other VPN tunnel enforement points) of the SCS.

Added the NAT address to the rule allowing communication with the FW and bingo!

Very strange as there has never been a rule allowing the NAT address to manage the FWs.

Checkpoint say it may be down to a possible file overwrite (corrupted) when the FW had an unclean shutdown and it's changed the way it is managed - Doesn't make any sense to me but i'll have to put it in the jar on the side with all the other "how the hell did that happen" faults i've had.

Thanks again for all your advice!!

[mdp_uk]

Just to let you know the fault has reappeared, the Firewalls are now seeing SCS traffic from the original non-NAT address and are rejecting with error: address spoofing.

I have reopened the call with checkpoint and will let you know the outcome. It had previously been working for 2-3 days.

[rickyboone]

Any updates on this issue?

We are noticing very similar behavior on a standby node in an HA cluster. Luckily the active node is not showing the same problem, but the standby is useless in it's current state. :|

[Routerkid1]

rebuild the standby:

[rickyboone]

Originally Posted by Routerkid1

rebuild the standby:

Yep, thought so. -.-;; Chances are we'll have to do the primary as well, given my luck.