Upgrading from R55 to R60 - a few notes

[andrew]

I hope this post is still useful even though I do not have the exact error language that would make it turn up in searches. In the haste to correct problems I didn't make notes.

We decided to upgrade from R55 to R60 because we were told that R60 supported Dynamic DNS in objects. We're hoping this allows us to configure site-to-site VPNs with telecommuters that possess dynamic IPs, such as cable modem customers.

The upgrade documentation suggested that the first step would be to verify eligibility for and to upgrade the FW licensing for NGX with the license_upgrade utility. The utility is OS dependent and was to be located on the R60 CD within the corresponding OS directory, however for Secure Platform, the util wasn't there. I assumed the license_upgrade found in the Linux folder would work, but instead we received a missing library error that prevented license_upgrade from starting. SmartUpdate, (a method we'd used in the past) after downloading the license from the CP user center, worked. The documentation included with the CDs doesn't mention SmartUpdate and instead gives explicit instruction on license_upgrade.

I spoke to CP support prior to the upgrade and left with the impression that R55 would function with R60 licenses installed. However, after applying the R60 licenses, VPNs ceased to function, and multiple 'No license for VPN' errors appeared in the Tracker. Rather than revert, I applied the OS and program upgrade with the patch add cd command. The installation quit with Error Code 2 after verifying upgrade readiness and then again when backing up the configuration. I think this may have had to do with SmartUpdate still being open on an admin machine, because after closing it patch add ran perfectly.

After the reboot, I could not connect to SmartCenter with the R60 utilities, though it ended up not having anything to do with the FW module but the local network interface. The FW hardware links up to a GBIC on a Cisco 3550, which requires Gigabit negotiation, but the 10/100/1000 NIC on server did not autonegotiate. This of course seems more like a hardware issue, but I mention it because R55 ran for more than a year with multiple reboots on the older version of SPlat and always brought the interface up without issue. Forcing the interface to gigabit with ifconfig resolved the problem.

Finally running I noticed no traffic in the SmartView tracker, and the SmartView Monitor reported that the policy server wasn't up. Installing the policy from the dashboard failed with a mismatch width error in the static table or static_table on a line in our .pf file. CP support (thanks Joe!) helped us determine that the problem was in our static NAT table, and by disabling NAT rules and process of elimination, we discovered an offending rule. This was a NAT rule that functioned fine under R55.
After removing the NAT the policy installed appropriately.

Unresolved issues are lack of ISP redundancy and errors with IP Telephony over VPN. When these are taken care of I'll add to this thread.

To this point the upgrade has required about 6 hours. None of the issues have made us regret the decision to upgrade, however without CP the static_table problem may have been a complete stumper.

Thanks

Andrew

[Sergej]

Thanks for feedback. Like we all mentioned before license upgrade is one of the main issues. The common problem is auto-upgraded license not works correctly. You need to connect to the usercenter by yourself and regenerate the license. That's why it is always better to have several 30 days demo licenses at the time of upgrade.
The bad news are - in a week or two we will see NGX_R61. And you need to upgrade one more time again :)

P.S. Do not forget to install last HFA asap (02 or 03 for NGX)

[chillyjim]

No licenses changes with R61, so at least that head ache will be gone

[andrew]

Update:

Came in this morning to talk to CP support. It seemed like my problem with VoIP over VPN was my biggest issue, but as it turns out the ISP Redundancy deal is much larger.

Here is the situation. With R55, we had two external links configured and setup for ISP redundancy in load balancing mode. After upgrading the load balancing stop working. Even worse is the way it stopped working --- Load balancing still wants to use that link but the FW doesn't actually finish sending the traffic. A web user behind our firewall might get their page right away on one click, and then the next click wait 5 or 10 seconds while the FW waits for the send out the backup to time out. The obvious solution would be to disable load balancing while we investigate, except that now there is no ISP Redundancy page on my FW object. So it's broken and can't be disabled or reconfigured.

I called CP and at first they thought it was a licensing issue, maybe NGX didn't give you ISP redundancy and an extra license was required? So we added a full-blown CP demo license to the box and still no ISP Redundancy under topology.

Then they started talking about revert and even ... gasp ... rebuild! That my policy made be corrupt. upgrade_export doesn't run from the shell, fails, however I haven't tried to do it after an fwstop because we're in production. I'll be in to make the donuts again early tomorrow.

Suggestions appreciated.

Andrew

[andrew]

FYI - if I log in under demo mode, I get the ISP Redundancy area under Topology, so it doesn't appear to be the GUI.

[andrew]

More from the front:

I've decided to build a new FW, R55, on another system. I'll import the backup from the old system. That way I'll have a backup FW ready when I try to revert the production server ... if the PF really is corrupt I'm afraid it may not come online again.

After moving the traffic to the new FW maybe it will be a good time to build a brand new policy. It's about 5 or more years old and is been part of a few upgrades.

Anyone else had a possibly corrupt policy before? Any one else start over from scratch? And are we the only ones still using static manual nats and traditional mode VPN config?

Andrew

[chillyjim]

Have you installed HFA2? I vaguely remember something about ISP redundancy problems being fixed in 1 or 2. HFA 2 should also fix you VoIP problems.

[andrew]

Well, actually, no.

I had planned on it after you first suggested doing so in the VoiP thread, but this morning I was afraid doing so would make me unable to use revert. Then, when I spoke to CP, they convinced me the PF was corrupt and I forgot about HFA2.

I'll get this backup FW running and then try HFA2. Maybe that will save me a whole bunch of work. Thanks.

Andrew

[chillyjim]

FYI if you are using SPLAT for your operating system you can take a Snapshot which will allow you to revert from a patch or upgrade.

Have you been able to run a CPINFO?

[andrew]

I did take a snapshot of my setup before the ugprade. It was part of the upgrade process and I was prompted to do so. I did a dry run of revert and the file is there.

Haven't tried cpinfo. What should I look for?

[andrew]

Also, there apply HFA2 won't impact my saved snapshot, correct? I can't imagine it would.

[Sergej]

Applying HFA is a first priority task before start any troubleshooting. Why to have pain if you can skip it (the pain)?

Download the HFA readmes first and search by the keyword redundancy also a good troubleshooting start point.

[chillyjim]

Originally Posted by andrew

Haven't tried cpinfo. What should I look for?

If cpinfo doesn't explode I doubt your ruleset is really corrupted.

[andrew]

My backup FW was installed yesterday morning and we moved to it. So far so good, now can I can troubleshoot the upgrade with less pressure and am free to reboot or push policy to it as I please. I was impressed with how well the restore worked once I did it correctly. It retrieved from a TFTP server (the same it backs up to) and ran without needing any input from me after starting. I left that evening with the file transfering and returned the next morning with OS and FW module completely configured. I moved a few cables over and voila, it was working.

One thing I should mention is that if you try to use upgrade_import with a backup file, you'll get this error: Failed to read the configuration of the production machine. I tried it twice before it dawned on me there is a restore command and I should use it instead.


I was able to run cpinfo but did get some weird statements. At least I thought they were weird. I'll try it again and paste the oddities to see if they're really that odd.

[andrew]

What's the best way to go about installing HFA2? I downloaded and extracted the files but not clear what to do afterwards. patch add?

[Sergej]

HFA_2 is to complex for patch add. Read the release notes and follow the instructions for you OS.

[andrew]

Patch applied, but still no ISP Redundancy under Topology in the Check Point Gateway object.

Baffling.

[Sergej]

Originally Posted by andrew

Patch applied, but still no ISP Redundancy under Topology in the Check Point Gateway object.

Baffling.

I have no ISP Redundancy page

[andrew]

Thanks for the link.

Our FW is running on SPlat. In fact, coincidently, we went with SecurePlatform when we moved to NG because we specifically wanted ISP redun...

It was there in R55. Upgrade to R60 and it is gone.

Available in demo mode.

[Sergej]

:)

Ok, I'm not 100% sure ISP redundancy is available in Express edition.