Newbie - How to check clustering on R65 and perform 'graceful' failover in cluster

[longlong]

Hi everyone,

Not sure if this is a really simple thing to do, but I've just taken over the administration of some R65 checkpoints running on crossbeam, looks like its running cluster XL and I'm not sure how to check of the following:

1) How to check whether the pair of firewalls are running Active/Standby or Active/Active ( both from the firewall CLI and via the smart monitor / cluster XL section ), and how to set up which load balance specifics when running Active/Active

2a) How to failover 'gracefully from one fw to another in both Active/Standby or Active/Active mode and check connections table etc ( both in CLI and smartmonitor GUI )

2b) Will graceful failover in both clustering setups retain current connections, and how to verify this in both CLI and Smartmonitor GUI?

Also when I do a fw ver this is what we are running:

This is Check Point VPN-1(TM) & FireWall-1(R) NGX (R65) - Build 427

thanks heaps in advance!

[cciesec2006]

Originally Posted by longlong

Hi everyone,

Not sure if this is a really simple thing to do, but I've just taken over the administration of some R65 checkpoints running on crossbeam, looks like its running cluster XL and I'm not sure how to check of the following:

1) How to check whether the pair of firewalls are running Active/Standby or Active/Active ( both from the firewall CLI and via the smart monitor / cluster XL section ), and how to set up which load balance specifics when running Active/Active

2a) How to failover 'gracefully from one fw to another in both Active/Standby or Active/Active mode and check connections table etc ( both in CLI and smartmonitor GUI )

2b) Will graceful failover in both clustering setups retain current connections, and how to verify this in both CLI and Smartmonitor GUI?

Also when I do a fw ver this is what we are running:

This is Check Point VPN-1(TM) & FireWall-1(R) NGX (R65) - Build 427

thanks heaps in advance!

1- cphaprob state, cphaprob -a if

2a - shutdown or reboot one of the firewalls and you will see failover or shutdown one of the interfaces on the Active firewalls on the switch and watch the failover to occur. "fw tab -t -s connections".

2b- telnet, ssh or sqlnet through the firewalls and when you reboot the Active firewalls, you will not lose connections. If you do, then failover is not working correctly.

[ShadowPeak.com]

While rebooting or unplugging a network interface on the active firewall will indeed cause a failover, it will not be completely graceful since it will take the cluster 2-3 seconds to figure out what happened. If running a continuous ping through the cluster, you will see 2-3 ping packets get lost while the non-graceful failover happens.

For a truly graceful failover, either run cphastop on the active member or select the active member's firewall object in SmartView Monitor, right click and select Cluster Member..Stop Member. You should lose zero packets on your continuous ping in this scenario.

[dantro]

Oh well. Graceful as defined by Check Point is this:

To switch cluster members gracefully by creating a faildevice on one member enter:

    echo cphaprob -d faildevice -t 0 -s ok register
    echo cphaprob -d faildevice -s problem report

Remove the faildevice like this:

    cphaprob -d faildevice -s ok report
    cphaprob -d faildevice unregister

Check the cluster status via:

    cphaprob stat

[ShadowPeak.com]

Originally Posted by dantro

Oh well. Graceful as defined by Check Point is this:

To switch cluster members gracefully by creating a faildevice on one member enter:

    echo cphaprob -d faildevice -t 0 -s ok register
    echo cphaprob -d faildevice -s problem report

Remove the faildevice like this:

    cphaprob -d faildevice -s ok report
    cphaprob -d faildevice unregister

Check the cluster status via:

    cphaprob stat

Right, that will work too with zero continuous ping losses. My method involves much less typing. :-)

[longlong]

Thanks heaps guys for the quick response!

I've just done the cphaprob commands as suggested, and I'm a bit confused by the output ( ie I dont think the clustering is working properly? ), is there some other commands to verify if things or ok or not?

In Smartmonitor the clustering to both fw's look ok, but the CLI output is a bit worrying?

[root@FW01 root]# cphaprob state

Cluster Mode: Sync only (OPSEC))

Number Unique Address Firewall State (*)

1 (local) 172.30.31.13 Active

(*) FW-1 monitors only the sync operation and the security policy
Use OPSEC's monitoring tool to get the cluster status

[root@FW01 root]# cphaprob -a if

eth5 non sync(non secured)
eth6 sync(secured), multicast
eth7 non sync(non secured)
eth8 non sync(non secured)

Virtual cluster interfaces: 3

eth5 172.30.20.253
eth7 x.x.x.x
eth8 172.30.21.254

[root@FW01 root]#
! --------------------------------------
[root@FW02 os]# cphaprob state

Cluster Mode: Sync only (OPSEC))

Number Unique Address Firewall State (*)

1 172.30.31.13 Down
2 (local) 172.30.31.14 Active

(*) FW-1 monitors only the sync operation and the security policy
Use OPSEC's monitoring tool to get the cluster status

[root@FW02 os]# cphaprob -a if

eth5 non sync(non secured)
eth6 sync(secured), multicast
eth8 non sync(non secured)
eth7 non sync(non secured)

Virtual cluster interfaces: 3

eth5 172.30.20.253
eth8 172.30.21.254
eth7 x.x.x.x

[root@FW02 os]#

do you guys see any issues with the above?

[belvdr]

Do you have this cluster defined as ClusterXL in SmartDashboard? I'm thinking this was originally a Nokia VRRP cluster and was converted to ClusterXL.

A properly configured ClusterXL object shows the following from 'cphaprob state':

# cphaprob state

Cluster Mode:   New High Availability (Primary Up)

You might also find it useful to switch to broadcast, instead of multicast:

cphaconf set_ccp broadcast

[bmolnar]

What type/model of firewalls are you running for FW01 and FW02?

[longlong]

Originally Posted by belvdr

Do you have this cluster defined as ClusterXL in SmartDashboard? I'm thinking this was originally a Nokia VRRP cluster and was converted to ClusterXL.

A properly configured ClusterXL object shows the following from 'cphaprob state':

# cphaprob state

Cluster Mode:   New High Availability (Primary Up)

You might also find it useful to switch to broadcast, instead of multicast:

cphaconf set_ccp broadcast

Yes the cluster is setup in Smartdashboard, well from what I can see...

* Actually this morning FW01 failed but still passing traffice via FW02 so I'm guessing the clustering is working to some degree!

we are running crossbeam's as the hardware platform FYI...

[belvdr]

Originally Posted by longlong

Yes the cluster is setup in Smartdashboard, well from what I can see...

* Actually this morning FW01 failed but still passing traffice via FW02 so I'm guessing the clustering is working to some degree!

we are running crossbeam's as the hardware platform FYI...

How is the cluster defined, though? From your output, it is not defined as ClusterXL.

[longlong]

In Smartmonitor it looks like it is clustering, as now one of the boxes is down, the remaining FW is getting connections and says 'OK' in the cluster.

Is there any other commands or smartmonitor places where I can check how this clustering is working? its a bit black magic-ish at the moment!

thanks..

[belvdr]

Open SmartDashboard, edit the cluster object.

ClusterXL should be checked under General Properties.

You should also have a ClusterXL section on the left.

[longlong]

Originally Posted by belvdr

Open SmartDashboard, edit the cluster object.

ClusterXL should be checked under General Properties.

You should also have a ClusterXL section on the left.

Thanks belvdr, I've checked this section, it looks like we are clustering using VRRP in HA mode (in 3rd party Configuration) instead of using cluster XL...

btw, what is the difference between using VRRP over Cluster XL?

FYI, found out some VRRP settings on the crossbeam below..

Also does anyone know how to do the CLI instead of going via the menu's?

Main Menu (for user admin)
=========

1) System Information
2) Network Information
3) Configuration
4) Interview
5) Application Install/Uninstall
-) Routing Protocols (N/A)
-) Licenses Manager (N/A)
8) System Shell
X) Exit

Enter choice? <1 - 8, X> []: 2

Network Information
===================

1) Interface Status
2) Interface Counters
-) Network Processor (N/A)
-) FW Accelerator (N/A)
5) IP Counters
6) ICMP Counters
7) TCP Counters
8) TCP Connections
9) Udp Counters
10) Udp Listeners
11) Arp Table
12) Routing Table
13) VRRP Status
-) Link Aggregation (N/A)
-) Rate Limiter Statistics (N/A)
-) ByPass Segment Status (N/A)
X) Exit

Enter choice? <1 - 16, X> [X]: 13

VRRP Status Information
===================

1) Summary
2) Group
3) Adjusted Groups
X) Exit

Enter choice? <1 - 3, X> [X]: 1

VRRP Status
===========
ID VRRP-Interface Group State If-State Master-Address Preempt C-Prio A-Prio M-Prio Description
1 eth5 DCSEXTFW master up x.x.x.x enabled 195 195 195
3 eth8 DCSEXTFW master up x.x.x.x enabled 195 195 195
7 eth7 DCSEXTFW master up x.x.x.x enabled 195 195 195


Would You Like To Relinquish VRRP Master Interface? <Y or N> [N]:

VRRP Status Information
===================

1) Summary
2) Group
3) Adjusted Groups
X) Exit

Enter choice? <1 - 3, X> [X]: 2
Failover Groups

1) All
2) DCSEXTFW
X) Exit

Enter choice? <1, 2, X> [X]: 2

VRRP Global Status
===========
Version NextHop-Check Apps-Stats-Check Apps-Status Stats-Prio-Delta
4 OFF ON UP 10

VRRP Group Status
===========
VRRP-Group Prior Act-Prior Adver-Int Preemp
DCSEXTFW 195 195 1 ON

VRRP Group Virtual Router Status
===========
VR-ID Interface Prior-Delta If-State
7 eth7 10 UP
1 eth5 10 UP
3 eth8 10 UP
Total: 3, UP: 3, DOWN: 0, Priority Altered: 0

VRRP Group NextHop Status
===========
NextHop Interface Prior-Delta Reachability

VRRP Group Monitored Interface Status
===========
Interface Priority-Delta State

[melipla]

Originally Posted by longlong

[root@FW01 root]# cphaprob state
Cluster Mode: Sync only (OPSEC))
Number Unique Address Firewall State (*)
1 (local) 172.30.31.13 Active


[root@FW02 os]# cphaprob state
Cluster Mode: Sync only (OPSEC))
Number Unique Address Firewall State (*)
1 172.30.31.13 Down
2 (local) 172.30.31.14 Active

You can see states like this when one cluster member (FW01) has been upgraded with a hotfix & rebooted, but the other cluster member (FW02) has not. Try a "fw ver" if they're the same, try rebooting FW02. Additionally, there's been *a lot* of bug fixes since R65 GA, please patch your system.

[cloehn]

Originally Posted by longlong

btw, what is the difference between using VRRP over Cluster XL?

With VRRP the Operating System is responsible for the decision which device is the active member and which is backup. Based on ARP mechanic it attracts the whole traffic to the active device beeing therefore responsible for the packet flow. VRRP is active-standby only but load balancing could be simulated with adding more then one VRRP group. With ClusterXL active/standby decision and packet flow is done by the CheckPoint Software. There are different modes how that will work excactly including active-active configurations. Simple High Availibilty Mode in ClusterXL is similiar to VRRP, but with VRRP you could work more granular by assigning priorities to the monitored circuits (with ClusterXL it is only working or not working). In both Scenarios the CheckPoint Software is responsible for State Synchronization.

[longlong]

Originally Posted by cloehn

........ In both Scenarios the CheckPoint Software is responsible for State Synchronization.

Is there some good CLI commands/Via smartcentre GUI to check State Syncronisation/how state Syncronisation works ( in terms of connects that get kept pre/during/post failover between members in the cluster )?

[cloehn]

There is no statistic for surviving sessions. Either there is a session entry or not. If not, you should see "out of state" packets in the SmartView tracker.

"fw ctl pstat" gives you information about dropped sync packets (if I remember corrcetly could also be seen in SmartView Monitor,when clicking on "sync"). If the firewall is under heavy load and too many sync packets are lost, maybe you could try to put HTTP (internet surving) out of the sync process.

If done simultaniously on both gateways "fw tab -t connections -s" should show more or less equal values in the VALs field (number of total sessions) if sync is working fine.

[siegrist]

Originally Posted by dantro

Oh well. Graceful as defined by Check Point is this:

To switch cluster members gracefully by creating a faildevice on one member enter:

    echo cphaprob -d faildevice -t 0 -s ok register
    echo cphaprob -d faildevice -s problem report

Remove the faildevice like this:

    cphaprob -d faildevice -s ok report
    cphaprob -d faildevice unregister

Check the cluster status via:

    cphaprob stat

This can also be easily done with

clusterXL_admin up/down