ClusterXL trouble. Help!!!!!!

[cciesec2006]

I need help with ClusterXL question.

Single SmartCenter running SPLAT NGx R65 with HFA_02.
I have ClusterXL license on the SmartCenter. The
license on the SmartCenter is as follows:

CPMP-CXL-HA-1-NGX CPVP-CPLS-1-NGX
CPMP-SCPRO-U-NGX

Basically, I have SmartCenter Pro and ClusterXL
license installed on the SmartCenter.

I built a NGx SPLAT R65 with HFA_02 Enforcement
Module. The SPLAT Enforcement module has 3 interfaces:
External, Internal and DMZ.

I created a checkpoint gateway object in the Dashboard,
called fw, set the topology to "undefined". In other
words, I turned off antispoofing. I then created a
gateway cluster, called fw-cluster, set the ip
addresses. I checked firewall and clusterXL.
I then set clusterXL to load-sharing unicast mode.
When I pushed policy, my clusterXL works as it
should:

However, if I reboot the firewall or do "cpstop;cpstart",
the cluster will show as "down". To fix it,
I have to uncheck "clusterXL" in the gateway cluster
properties, push the policy, check "clusterXL" again,
push the policy again, then I get my clusterXL back.

Anyone run into this issue before? Please help.

This is very straight forward SPLAT installed
with "any any accept". Thanks.

[melipla]

Originally Posted by cciesec2006

However, if I reboot the firewall or do "cpstop;cpstart", the cluster will show as "down". To fix it, I have to uncheck "clusterXL" in the gateway cluster properties, push the policy, check "clusterXL" again, push the policy again, then I get my clusterXL back.

I'm guessing that the "down" you see is in the Smartview Monitor? I'd be more interested in what the cluster members themselves said their state was via "cphaprob state" and "cpahprob -a if" for each member. It may give you more information with which to work on.

Also, not defining the topology seems like a bad idea to me, especially since many cluster features depend on whats defined in the topo. If you don't want to do antispoofing then simply uncheck it once the interface topo is defined.

[cciesec2006]

I am seeing "down" on the box itself through "cphaprob state".

"Also, not defining the topology seems like a bad idea to me, especially since many cluster features depend on whats defined in the topo. If you don't want to do antispoofing then simply uncheck it once the interface topo is defined."

Where do you base this information from? I have eth4 "defined" as the
sync interface, NOT cluster interface.

I have an identical setup on NG with AI R55 hfa_20 with interfaces
undefined and it works just fine.

[melipla]

Originally Posted by cciesec2006

I am seeing "down" on the box itself through "cphaprob state".

Are you saying that cpha isn't even running?

If cpha is running then cphaprob state gives more information than "down", I guess I was looking for the complete output, with the cphaprob -a if you should be able to identify [loosely] why its down. The logs also come in handy here.

Originally Posted by cciesec2006

Where do you base this information from? I have eth4 "defined" as the sync interface, NOT cluster interface.

So I'm confused you said you didn't set up any topology, are you now saying that you only set up half of your topology?

[cciesec2006]

hi,

- cpha is running but it is showing as "down". To fix it, I have to
do what I stated earlier.

- I set EXTERNAL, INTERNAL and DMZ topology to "undefined".
I set eth4 interface to "this network" because this is the sync
interface. That's exactly the same way I setup in AI R55
as well.

[Thorpuse]

Are you running SPLAT Pro? If so, you need to create a rule for the FIBMGR service to communicate between the Cluster members. I don't remember the port for this, but it's in SecureKnowledge.

[Routerkid1]

we have a service called fibmgr tcp 2010 as a predefined service. Use this with a src and dst of each cluster member.

[laril]

Could you please paste the output of the following commands here:
cphaprob state
cphaprob list
cphaprob syncstat?

Also the sync part of fw ctl pstat would be helpful.

Then it would be easier for us to help with your issue. :-)

[cciesec2006]

Check Point SecurePlatform NGX (R65)
For Web User Interface access connect to https://192.168.1.201

login: admin
Password:

? for list of commands
sysconfig for system and products configuration

[gw1]# cphaprob state

Cluster Mode: Load Sharing (Unicast)

Number Unique Address Assigned Load State

1 (local) 192.168.1.201 100% Active (pivot)

[gw1]# cphaprob -a if

Required interfaces: 4
Required secured interfaces: 0

eth0 UP non sync(non secured), broadcast
eth1 UP non sync(non secured), broadcast
eth2 UP non sync(non secured), broadcast
eth3 UP non sync(non secured), broadcast
eth4 DOWN (148760 secs)non sync(non secured), broadcast
eth5 DOWN (148760 secs)sync(secured), broadcast

Virtual cluster interfaces: 4

eth0 192.168.1.200
eth1 10.100.10.1
eth2 192.168.1.1
eth3 192.168.1.17

[gw1]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: OK

Registered Devices:

Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 148788 sec

Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 148780 sec

Device Name: cphad
Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0.1 sec

Device Name: fwd
Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.1 sec

[gw1]# cphaprob syncstat

Sync Statistics (IDs of F&A Peers - None):

Other Member Updates:
Sent retransmission requests................... 0
Avg missing updates per request................ 0
Old or too-new arriving updates................ 0
Unsynced missing updates....................... 0
Lost sync connection (num of events)........... 0
Timed out sync connection ..................... 0

Local Updates:
Total generated updates ....................... 101437
Recv Retransmission requests................... 0
Recv Duplicate Retrans request................. 0

Blocking Events................................ 0
Blocked packets................................ 0
Max length of sending queue.................... 0
Avg length of sending queue.................... 0
Hold Pkts events............................... 0
Unhold Pkt events.............................. 0
Not held due to no members..................... 4
Max held duration (sync ticks)................. 0
Avg held duration (sync ticks)................. 0

Timers:
Sync tick (ms)................................. 100
CPHA tick (ms)................................. 100

Queues:
Sending queue size............................. 512
Receiving queue size........................... 256

[gw1]# fw ctl pstat

Machine Capacity Summary:
Memory used: 2% (28MB out of 1202MB) - below low watermark
Concurrent Connections: 0% (5 out of 24900) - below low watermark
Aggressive Aging is not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 20971520 bytes in 5115 4KB blocks using 5 pools
Total memory bytes used: 1015776 unused: 19955744 (95.16%) peak: 1515084
Total memory blocks used: 345 unused: 4770 (93%) peak: 424
Allocations: 751677 alloc, 0 failed alloc, 735512 free

System kernel memory (smem) statistics:
Total memory bytes used: 46306696 peak: 46310384
Blocking memory bytes used: 1668500 peak: 1670124
Non-Blocking memory bytes used: 44638196 peak: 44640260
Allocations: 50694157 alloc, 0 failed alloc, 50693661 free, 0 failed free

Kernel memory (kmem) statistics:
Total memory bytes used: 26321316 peak: 26343420
Allocations: 752298 alloc, 0 failed alloc, 735879 free, 0 failed free
External Allocations: 0 for packets, 7000 for SXL

Kernel stacks:
0 bytes total, 0 bytes stack size, 0 stacks,
0 peak used, 0 max stack bytes used, 0 min stack bytes used,
0 failed stack calls

INSPECT:
8200 packets, 1936066 operations, 17892 lookups,
0 record, 545814 extract

Cookies:
1138277 total, 0 alloc, 0 free,
4 dup, 780826 get, 41 put,
1138327 len, 12 cached len, 0 chain alloc,
0 chain free

Connections:
2963 total, 517 TCP, 1488 UDP, 953 ICMP,
5 other, 4 anticipated, 11 recovered, 5 concurrent,
14 peak concurrent

Fragments:
0 fragments, 0 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
11742/0 forw, 26795/0 bckw, 1 tcpudp,
0 icmp, 131-135 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 89043, retransmitted : 0, retrans reqs : 0, acks : 0
Sync packets received:
total : 0, were queued : 0, dropped by net : 0
retrans reqs : 0, received 0 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
[gw1]#
[gw1]#



---------------

After I performed "cpstop;cpstart" I get this, as
you can see clusterXL is shown as down:

[Expert@gw1]# cpstop;cpstart
Stopping SmartView Monitor daemon ...
SmartView Monitor daemon stopped
Stopping SmartView Monitor kernel ...
SmartView Monitor kernel stopped
FloodGate-1 is already stopped.
SecureXL device disabled.
FW: stopping VPN-1 module -- OK
FireWall-1: disabling IPv4 forwarding and bridge forwarding
SVN Foundation: cpd stopped
SVN Foundation: cpWatchDog stopped
SVN Foundation stopped
cpstart: Power-Up self tests passed successfully

cpstart: Starting product - SVN Foundation

SVN Foundation: Starting cpWatchDog
SVN Foundation: Starting cpd
SVN Foundation started

cpstart: Starting product - VPN-1

FireWall-1: Starting external VPN module -- OK
FireWall-1: Starting fwd

SecureXL will be started after a policy is loaded.

Installing Security Policy NGx on all.all@gw1
Fetching Security Policy from localhost succeeded

Fetching Security Policy From: 172.25.2.254

Local Policy is Up-To-Date.
The Policy was not installed because it is the same as the Policy already on the Module.
FireWall-1: enabling bridge forwarding
FireWall-1 started

cpstart: Starting product - FloodGate-1

FloodGate-1 is disabled. If you wish to start the service, please run 'etmstart enable'.

cpstart: Starting product - SmartView Monitor

SmartView Monitor: Loading kernel ...
rtmstart: Loading SmartView Monitor kernel module
Starting SmartView Monitor kernel ...
SmartView Monitor kernel started
Starting SmartView Monitor daemon
[Expert@gw1]# netstat -an | grep 257
tcp 0 0 0.0.0.0:257 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.201:32894 172.25.2.254:257 ESTABLISHED
tcp 0 0 192.168.1.201:32882 172.25.2.254:257 TIME_WAIT
tcp 0 0 127.0.0.1:257 127.0.0.1:32836 TIME_WAIT
tcp 0 0 127.0.0.1:32911 127.0.0.1:257 ESTABLISHED
tcp 0 0 127.0.0.1:257 127.0.0.1:32911 ESTABLISHED
[Expert@gw1]#
[Expert@gw1]#
[Expert@gw1]#
[Expert@gw1]#
[Expert@gw1]# cphaprob state

Cluster Mode: Load Sharing (Unicast)

Number Unique Address Assigned Load State

1 (local) 192.168.1.201 100% Down

[Expert@gw1]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: OK

Registered Devices:

Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: problem
Time since last report: 58.7 sec

Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 42.8 sec

Device Name: cphad
Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0.7 sec

Device Name: fwd
Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.6 sec

[Expert@gw1]# cphaprob -a if

Required interfaces: 4
Required secured interfaces: 0

eth0 UP non sync(non secured), broadcast
eth1 UP non sync(non secured), broadcast
eth2 UP non sync(non secured), broadcast
eth3 UP non sync(non secured), broadcast
eth4 DOWN (54.1 secs)non sync(non secured), broadcast
eth5 DOWN (54.1 secs)sync(secured), broadcast

Virtual cluster interfaces: 4

eth0 192.168.1.200
eth1 10.100.10.1
eth2 192.168.1.1
eth3 192.168.1.17

[Expert@gw1]# cphaprob syncstat

Sync Statistics (IDs of F&A Peers - None):

Other Member Updates:
Sent retransmission requests................... 0
Avg missing updates per request................ 0
Old or too-new arriving updates................ 0
Unsynced missing updates....................... 0
Lost sync connection (num of events)........... 0
Timed out sync connection ..................... 0

Local Updates:
Total generated updates ....................... 105586
Recv Retransmission requests................... 0
Recv Duplicate Retrans request................. 0

Blocking Events................................ 0
Blocked packets................................ 0
Max length of sending queue.................... 0
Avg length of sending queue.................... 0
Hold Pkts events............................... 0
Unhold Pkt events.............................. 0
Not held due to no members..................... 4
Max held duration (sync ticks)................. 0
Avg held duration (sync ticks)................. 0

Timers:
Sync tick (ms)................................. 100
CPHA tick (ms)................................. 100

Queues:
Sending queue size............................. 512
Receiving queue size........................... 256

[Expert@gw1]# fwctl pstat
bash: fwctl: command not found
[Expert@gw1]# fw ctl pstat

Machine Capacity Summary:
Memory used: 2% (27MB out of 1202MB) - below low watermark
Concurrent Connections: 0% (11 out of 24900) - below low watermark
Aggressive Aging is not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 20971520 bytes in 5115 4KB blocks using 5 pools
Total memory bytes used: 951456 unused: 20020064 (95.46%) peak: 1515084
Total memory blocks used: 299 unused: 4816 (94%) peak: 424
Allocations: 766019 alloc, 0 failed alloc, 750499 free

System kernel memory (smem) statistics:
Total memory bytes used: 45744184 peak: 46310384
Blocking memory bytes used: 1397444 peak: 1670124
Non-Blocking memory bytes used: 44346740 peak: 44640260
Allocations: 50745501 alloc, 0 failed alloc, 50745027 free, 0 failed free

Kernel memory (kmem) statistics:
Total memory bytes used: 25695580 peak: 26343420
Allocations: 766810 alloc, 0 failed alloc, 751058 free, 0 failed free
External Allocations: 0 for packets, 7700 for SXL

Kernel stacks:
0 bytes total, 0 bytes stack size, 0 stacks,
0 peak used, 0 max stack bytes used, 0 min stack bytes used,
0 failed stack calls

INSPECT:
14 packets, 7236 operations, 81 lookups,
0 record, 1975 extract

Cookies:
892 total, 0 alloc, 0 free,
0 dup, 700 get, 12 put,
916 len, 6 cached len, 0 chain alloc,
0 chain free

Connections:
13 total, 2 TCP, 2 UDP, 4 ICMP,
5 other, 0 anticipated, 0 recovered, 11 concurrent,
13 peak concurrent

Fragments:
0 fragments, 0 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
0/0 forw, 0/0 bckw, 0 tcpudp,
0 icmp, 3-0 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 10, retransmitted : 0, retrans reqs : 0, acks : 0
Sync packets received:
total : 0, were queued : 0, dropped by net : 0
retrans reqs : 0, received 0 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
[Expert@gw1]#

[Thorpuse]

Your sync interface is down.... did you use a crossover cable to connect the sync networks together, or are they plugged into a switch/hub?

Another silly mistake that I have made before is you give the sync interface the same IP address on both machines.... ooops....

get the systems so that they can communicate on the sync interface (i.e. ping the other machine's sync interface) and things should work.

[cciesec2006]

I connect the sync interface into a Cisco Catalyst 2960. I do not
use x-over cable.

"Another silly mistake that I have made before is you give the sync interface the same IP address on both machines.... ooops.."

Well, there is only ONE machine in this case.

"get the systems so that they can communicate on the sync interface (i.e. ping the other machine's sync interface) and things should work."

Well, I only have 1 system here. clusterXL is independent of how many
systems you have.

Any ideas? Thanks.