Failed R60 upgrade, Rolled Back, Cluster not syncing

[crucial]

We have a management server running R60 and two cluster members on R54. On Tuesday night, we attempted to upgrade our R54 servers to R60. We brought one of the servers down, rebuilt it and brought it online. We had planned to push a policy to this server, bounce the other one, and rebuild that one after the first was online.

After the first one was back up, we were unable to push out a policy to the servers. So, we backtracked, wiped the first firewall again and reinstalled R54. So both servers are on R54 again.

Now, I am unable to get the cluster to be stable. The sync interface on the second firewall (the one we never changed) shows down. Also, we don't have the password for the second firewall (which is why we were wiping/rebuilding them to upgrade to R60.)

Here is some information from the first FW. Maybe somebody can make some recommendations on what to try next?

[fw-ep1]# cphaprob state

Cluster Mode:   New High Availability (Active Up)

Number     Unique Address  Assigned Load   State

1 (local)  192.168.32.44   100%            active
2          192.168.16.45   0%              down

Sync:
        Version: new
        Status: Able to Send/Receive sync packets
        Sync packets sent:
         total : 15096585,  retransmitted : 0, retrans reqs : 0,  acks : 0
        Sync packets received:
         total : 0,  were queued : 0, dropped by net : 0
         retrans reqs : 0, received 0 acks
         retrans reqs for illegal seq : 0
         dropped updates as a result of sync overload: 0 

[fw-ep1]#

[maurox]

Which OS ( for the password recovery procedure) do you have on the firewall machines ?
If you have the problems only on the secondary ( down ) machine , why don't you resinstall( OS an fw-1) that machine ?
Which problem did you have when you tried to push the policy on the NGX fw machine ?

Regards,
Maurox

[crucial]

We are running splat on the enforcement points and W2k on the management server.

We are planning to wipe FW2 and rebuild it as R54 (with a known password), but currently it is passing all of the traffic and we do not know what will happen if we attempt to fail over. Our only scheduled downtime is almost a month out. Anytime we attempt to push a policy to the servers, they stop passing traffic and smartview monitor ClusterXL shows both as 'disconnected'.

After we brought the 1st server back online as R54, we were unable to push the cluster topology to the members. We get an error saying the interface configurations did not match, although they match as well as I can see.

[maurox]

Are you sure that all traffic is passing trough FW2 ?
In your previous post I read that the HA process on the fw2 is considered, by the fw1 ,down....

[crucial]

All of the logs in Smartview tracker are displaying FW2 as the log origin.


Since my last message, I attempted to push a policy to the cluster and got an error:

Installation Targets	Version	Policy	Type	Details
fw-ep-shrd	NG AI	Advanced Security		

Reason: TCP connectivity failure ( port = 18191 )( IP = aaa.bbb.191.244 )[ error no. 10 ].   ( message from member fw-1 )

After that, both members showed 'disconnected' in clusterXL. I was forced to down FW-1 and leave it running on FW-2. Right now we are running steady on FW-2, and FW-1 is powered off. I think I'm going to have to leave it like this until the next scheduled downtime or until somebody with more experience gets here to look at it.

Thanks for the help

[Lackie]

That connectivity error is usually due to SIC or the inability to communicate with the firewall, either because there is a policy or rule on the firewall (the firewall with the aaa.bbb.191.244 address in your case) or there is a routing problem. As this was working before I would say it's not a problem with the routing. After rebuilding FW-1, were you able to reset SIC with the firewall? Is there a policy currently installed on the firewall? Either the Initial Policy or the DefaultFilter will block a policy push.

[seanmac1904]

I had this sic error (on solaris) my issue was that in /etc/rc3.d S99cpboot happened before my S99staticroutes file

therefore my module had not route to mgmt server and loaded the default policy.

I did an fw unloadlocal and pushed a new policy after the routes were added and all was fine ( I aslo changed my staticroutes to S98staticroutes so it happened before the CPboot)

I noticed you said you pushed a policy in the middle of the upgrade, my reading of the NGX upgrade guide said this is a "bad idea" for a zero-downtime cluster upgrade.

this is how I did it (and it went reasonably smoothly)

I needed to add set nautopush=64 to my /etc/system and change my routes file (as above)


here my process

NGX Upgrade Process

1. run cphaconf set_ccp broadcast on all cluster members

2. choose cluster_member1 as the final cluster member (upgrade cluster_member2 first)

3. attach NGX licenses to both firewalls

4. upgrade cluster_member2 using smartupdate

5. issue cphaprob stat on cluster_member1 and verify it is active or active-attention

6. issue command fw ctl setsync off on cluster_member1

7. issue cphastop on cluster_member1at this point Cluster_member2 will take up the load

8. use smartupdate to upgrade cluster_member1

9. reboot cluster_member1

10. run cphaconf set_ccp multicast followed by cphastart on all cluster members


there are a couple of steps you need to do if you dont use smartupdate to do with compiling the policy

hope this is of some help

cheers for now

Sean in Perth

[joefav]

I tried the ZERO DOWNTIME upgrade and the NGX modules never came back on line, but remained in a READY state. I was using SecurePlatform though and did an 'in place' upgrade. How can I do some troubleshooting with ClusterXL?