Upgrade: NG FP3 to NGX R62 (win2k)

[seizeadonai]

All,

***PROLOGUE***

First I want to state that I appreciate anyone who has taken the time to respond to this thread in any way shape or form. Now I can beg for help!

Our organization has used NG FP3 on Win2k for quite some time now. My duties, since being hired, have been to identify software/hardware that needs replacement/upgrade. Our firewall has been in my crosshairs for sometime.

***THE UPGRADE PROCESS--BACKGROUND***

We have two identical servers; 1, the active firewall running NG FP3 with all applicable hot fixes on win2k server and 2, the "backup firewall." I rebuilt the backup firewall with win2k, all applicable service packs, updates, etc. I configured the server's networking interfaces & routing tables to match the current active firewall.

I took the NGX R62 CD, put it in the active firewall, ran the export procedure as documented in the upgrade guide. The checkpoint "installation" program generated a tar-gzipped archive of the entire configuration of the active firewall as it should. That was relatively painless and simple, I was impressed.

I took the tar-gzipped configuration, moved it over to the "backup firewall" that I am using as the upgrade box--which will replace the active firewall. I ran the installation program and selected the option to import an exported configuration. The installation program for R62 ran without error. I rebooted, installed the upgraded UTM/NGX licenses, put hardware loopbacks into the Ethernet interfaces and fired up the smart dashboard. Logged in, the configuration looked the exact same. I did a few hours of in depth configuration comparisons between the two servers to make sure that the upgrade export/import didn't miss anything. Everything looked good.

***THE ROADBLOCKS***

When our maintenance window arrived, I rolled into work, cup of coffee in hand, fired up some ping/http/cifs tests to automatically run, verified active firewall is responding the way it has always, all good. I then swapped the firewalls cat5 cables for the internal/external interfaces to the "backup firewall" (fresh install of NGX R62 with exported/imported configuration from NG FP3), dumped the ARP tables on most applicable switches, looked at my tests, boom everything "looked good."

VPN Tunnel's: up and chirping
SecureRemote Login: active and ready
Internal to Internet traffic: flowing nicely

Aces right?

I then proceeded to test the services that we hosted, mail, some http, a little sftp, and a pinch https--nada, zip, zero, nuthin'

Went into SmartDashboard to check the config, automatic NAT rules existed. I checked the proxy arp table "fw ctl arp" -- looked good. Compared the NAT config on the NG FP3 firewall to the NGX, no difference. I got a little worried.

I logged into a box that has a NAT rule specified for it, couldn't get out to the internet, as a matter of fact, couldn't get past the firewall, no VPN, no nuthin'. Okay, dumped the ARP tables on the box, verified, checked the new MAC address against the actual interface on the firewall, same. Attempted external communication from that box again, nothin'.

Fired up SmartView Trakker, packets didn't even look like they were making it to the firewall. So I said to myself, okay, maybe this is isolated right? WRONG. I logged into another box that was configured for auto NAT, same deal.

I did some research and here is what I came up with. Maybe something about the automatic proxy arp was broken and not listening for that traffic and passing it through the engine. I manually created a local.arp file and put it in the appropriate directory. I deleted all the automatic NAT rules just to make sure that the manual proxy arp file was being used, it was. Re-created the auto NAT rules, based on the other firewalls configuration, tested, same results. I then played around with manual NAT for a while. I already had the proxy arp entries created manually and figured that this might due the trick. Same result.

I realized what I might be missing at this point is something obvious, routing. I read some posts from northlandboy and robertgraham, which incidently I used to work with, small Internet huh? Anyways their recommendations have been to abandon automatic NAT wherever possible in favor of manual NAT and routing statements on the CE. Here is the issue with that, our next hop is PE. Our firewall acts as the router as our circuit comes in via Ethernet. The routing entries on the CE, which would be the firewall itself, proved to be fruitless. I tried my hand at a few things there, didn't really get anywhere. I then read some more posts.

I found a post, moving to NGX : two questions which eluded to an issue using the export/import tool and then upgrading. joris said that he had to add some routes to the firewall to fix the automatic nat config.

I haven't tried the route entries yet, and really don't want to, there has got to be a better solution. I don't want to pair auto NAT with manual band-aids, proxy arp, routes, etc, if auto NAT should take care of all this in the first place.

I know I have probably provided a lot of useless information, so if you have any specific questions, I will be inclined to elaborate. Any help on why the automatic NAT rules are not working the way that they were designed? I'd like to stick to using them as opposed to the manual NAT rules for simplicity of management for other people who may touch the firewall in the future.

Thanks!

[RayPesek]

VPN Tunnel's: up and chirping
SecureRemote Login: active and ready
Internal to Internet traffic: flowing nicely
---------------------------------------
I just had this precise issue on Nokia following some maintenance. Everything hitting the true external IP worked but nothing using automatic proxy ARP worked.

I don't know whether it was that I pushed the policy with the Internet line disconnected or what, but a second policy push fixed everything.

Do you have the SmartCenter on the enforcement module? It sort of sounds like it.

If so, I'd consider breaking the SmartCenter off to its own box and re-doing the enforcement modules with SPLAT. You're working with an operating system that is pretty much end of life and a firewall vendor that is cutting edge. Maybe the two aren't playing nice.

I've never used FW-1 on Windows, so I don't know if the local.arp file even works. A small voice in my head is saying it doesn't.

FWIW,

Ray

[chillyjim]

Ohhh how I hate windows as an enforcement point but....


fw monitor -e 'accept src=<testmachineIP> or dst=<testmachineIP>;'

That will show you what the enforcement point is seeing, and what it is doing with the packet. Best thing is to try and telnet on port 25 to some external mail server. You should see 4 lines for each packet and be able to see the IP address change. Automatic NAT should work just fine you. I would bet its an ARP cache issue at your ISP.

[seizeadonai]

Again, I would like to start off in saying thank you for your time in replying.

RayPesek, to answer your question, it's one box. Standalone FW-1 I believe is along the lines of the terminology that checkpoint uses. The enforcement module, management module, licensing module, all one box. Unfortunately management will not allow me to break up modules across boxes, we just don't have the hardware budget for it; besides NG FP3 has been configured the same way and has worked since it came out. I have pushed multiple policies and still no change. I agree with your comments surrounding upgrading the OS, right now my task is to upgrade the FW software first, I can then upgrade the OS once it's up and running on 2k. I have been building a case to upgrade all our 2k server infrastructure, it's just a slow process with management. As far as the local.arp, I know for a fact it sees the manual proxy arp entries, it's just a matter of having the local.arp file in the correct place, the \conf dir.

chillyjim, I would prefer having a nokia IP 330 that would do the job nicely. I am very comfortable in linux, but I'm the only one around here. If I "disappear" they'd be stuck between a rock and a hard place, I don't want to leave them high and dry, not my style. As far as what your suggesting I do, I can try it. What lead me to believe it's specific to the firewall is that if I remove the auto NAT, the server get's out just fine. I understand that it still could be an upstream ARP cache issue, because when I do that, the server's that have issues can now access the internet, via the hide behind the intereface on the firewall. What doesn't make sense to me is when I swapped the cable for the external interface on the firewall from the old firewall to the new, the tunnel's and connectivity comes right up. If it was a cache issue on the PE, then wouldn't external communication fail until the upstream cleared their ARP entries?

[RayPesek]

Yeah, I know about hardware cost issues. Penny-wise and pound-foolish, usually. When I took my management server to NGX, it really started huffing and puffing. Did I say "server"? It's an old 800 MHz desktop with 512 MB of RAM that I got when one of our facilities closed down. It replaced the 600 MHz desktop that I got from another facility's junk closet. I did put new hard drives in it, though, and my concern over its age does make me image it more frequently than I would otherwise, so maybe "old" is good. :-)

When I was testing the upgrade, I installed Server 2003 on a Microsoft Virtual Server 2005 virtual instance and it worked perfectly, as it should. I'm thinking about making that a permanent solution because it is hardware agnostic and certainly easy to back up.

A big disadvantage is that it's harder to secure physically. That's because the web browser interface to the virtual server itself is now reachable from all IP's internally, and so is it's login screen. :-(

Take care,

Ray

[mekach22]

We just upgraded our FP3 (distrubuted environment with 2 gateways) to NGX R60 SPLAT last Thursday and Static NAT is not working. I used the NGX upgrade_export/upgrade_import utility to perform the upgrade. All new hardware. There had to be something lost in translation when doing this upgrade from FP3 to NGX.

We have several services hosted by NAT and we cannot access any of them from the Internet (unless we utilize a work-around by adding static routes for those devices on the upstream router). This workaround is limited though, therefore, it is not a solution. All NAT’ing was working on the FP3 platform.

I have a case open with Checkpoint, but they have yet to figure out the issue. We have checked the proxy arp table "fw ctl arp" – and the static NAT entries are there. We have verified the new MAC addresses with the actual interfaces on the firewalls. We have tried disabling Auto NAT on the said devices and creating Manual NAT rules. No luck. We have added a local.arp file, no luck. Added persistent ARPS with arp -v -n -i eth0 -s xx.xx.xx.xx 00:00:00:00:00:00 pub. No luck.

If I try to ping one of the NAT’d IPs from outside and watch the traffic with a TCPDUMP on eth0 on the firewall, I get “arp who-has xx.xx.xx.xx (NAT) tell xx.xx.xx.xx (upstream router). The firewall is not taking ownership of the NAT.

NGX R60 has been out for some time and I cannot believe that Checkpoint does not have resolution on this. Our case just got pushed to an escalation engineer at Checkpoint, so we will see if we get this resolved.