Zero Down Time Upgrade (from NGR55 to NGX61) on a Cluster (VRRP)

[mirage131]

I have 2 IP380s (VRRP) running NGR55 & will be upgrading this weekend to
NGX 61.. I have read in the upgrade guide (NGX6R1) that it's possible to do an upgrade with Zero downtime..I am curious has anyone tried to do this..& succeeded?..Also any tips or suggestions would be greatly appreciated.

Thanks in advance,

Larry Milliken
lmilliken@uslec.com

[northlandboy]

Since you're running VRRP, things should be much easier - you shouldn't have to go through all the steps detailed in the upgrade guide, and you shouldn't have the problems that others have experienced.

Here's the way I would do it if I was you:

1/ On your backup firewall, upgrade the IPSO and Check Point.

2/ Within SmartDashboard, change the cluster version to NGX. Install policy on the cluster, but uncheck the box that causes it to back out the install if it can't install on all cluster members. Policy should only be installed on the upgraded module - you'll get a version error on the other one.

3/ Once policy is installed, and you're happy with that, bump up the VRRP priorities on your upgraded module. At this point, it should go into VRRP master, and start taking all traffic. Note that this is a zero-downtime upgrade, not a full connectivity upgrade - so any sessions that were running across the R55 module will get broken.

4/ Sit and watch that for a while. Do your service tests. Make sure you're happy with the way everything is working. If you've got any problems, you can just drop the VRRP priorities back down, and traffic will go back across the R55 box.

5/ Once you're happy with everything, upgrade the IPSO and CP versions on the R55 box. It will come up, you can then push policy to the cluster again, which should complete with no problems. Sync should start working between the R61 nodes, but you may find that only delta sync works, and no full sync has been done - in that case you'll see a consistently higher number of connections on the master node than the one you upgraded second - but there will still be a few connections on the backup node. In that case, do a cpstop;cpstart on the module you upgraded second, so that a full sync is forced.

6/ If everything is OK, then switch your VRRP priorities back to make sure everything is still cool running on your normal master.

7/ There is no step 7

8/ Profit!

I haven't done exactly what you're doing with those particular versions, but I have done many, many VRRP upgrades in the past, and this is how I'd handle it.

Things are a little different if you're using ClusterXL, or IP Clustering.

[Yasushi Kono]

Hi northlandboy,

my apologies for my poor English skills. Hopefully, you're able to understand of what I am trying to tell.

You stated that any session running across the R55 module will get broken. I am not really sure if it is intentionally by design. You should try to get a really Zero downtime upgrade by checking the "Support non-sticky connections" in the 3rd Party Configuration options of your Gateway Cluster Properties which is unchecked by default!

Regards,
Yasushi

[northlandboy]

I'm not sure that changing that option will do what you want - it's more for active/active clusters, not VRRP.

If you're running VRRP, you should have that option turned off, as it can slow down connection setup rate. My understanding is that turning it on will make it hold up initial SYNs until all members have acknowledged receiving the sync update.

It's still not going to help with version upgrades, as prior to NGX, you couldn't sync between versions. You can sync between R60 and R61, allowing a full connectivity upgrade. The closest you can get to it with a pre-NGX system is to allow out of state TCP traffic for a while.

[fwjockey]

When putting the upgraded firewall module into production, in a version-mismatched cluster, during the 0 downtime upgrade, I uncheck the new "firewall monitor" option in the VRRP config on the standby node to get it to take master in the VRRP cluster after bumping it's VRRP priority. I think it's because sync isn't healthy (mismatched cp versions.) Anyone else see this in IPSO 4.1?

fwj

[avilT]

Can I apply the above procedure from NG FP3 to NGX upgrade?
What happens to VRRP cluster when the IPSO/FW version is different on each member? What about connection table?
Thank You