R60-HFA4: lost ospf and static routes after changing interface speed

[pvaneynd]

Hello

Last weekend we had a strange issue. We had upgraded our firewalls to R60 HFA-4 and then proceeded to change the speed of the interfaces to auto. We did this by going to the standby node of the cluster (all SPLAT machines running clusterXL and ospf) and using "eth_set ethX autoneg" and then changing it on the cisco switches.

All went well until we changed the interface connecting the firewalls to the backbone on which they talk ospf to each other and the routers (not my design). When I did this I lost connectivity to the firewall from the management station. When I regained access via the active node of the cluster it turned out that all ospf routes were gone. But stranger still: even the static route which I had added towards the management station as a backup in case the ospf failed was gone. It was still there in /etc/sysconfig/netconf.C but not listed anymore in 'netstat -rn', nor did routing work. In the end we had to reboot the firewall to regain the routing.

This is not improving our confidence in using ospf on the SPLAT's. Did anymore have similar issues?

[northlandboy]

Just wondering, but did the interface come back up after you changed it?

This behaviour sounds to me exactly like if the interface was down - it would then not be able to participate in OSPF, and all routes out that interface would be removed from the routing table.

I doubt it was an OSPF issue at all. Did you check that interface, see if OSPF traffic was being received?

Also, a bit of an odd choice to change all interfaces to auto, usually it's the other way round.

Are those ports set to portfast?

[pvaneynd]

Originally Posted by northlandboy

Just wondering, but did the interface come back up after you changed it?

Yes after a few seconds, like all other interfaces.

Originally Posted by northlandboy

I doubt it was an OSPF issue at all. Did you check that interface, see if OSPF traffic was being received?

Well as it was the backup member of the cluster as far as I understand it doesn't handle ospf, it should get the updates through FIB. And rebooting it fixed it without further changes.

Originally Posted by northlandboy

Also, a bit of an odd choice to change all interfaces to auto, usually it's the other way round.

Recently we've noticed that more and more interface cards will not handle being forced to a certain speed. Forcing an intel card too 1000/FD is futile, it will never get a link with the cisco. So as 100 MB interfaces work with auto we've decided to go for auto for all interfaces.

Originally Posted by northlandboy

Are those ports set to portfast?

yes.

[northlandboy]

Thanks for the extra information.

I still think that that interface did not come back up properly on the firewall side, and so it was not processing traffic. That would explain the missing static route. Given that you had a missing static route too, I don't think that you should be worried about your confidence in OSPF on SPLAT - the issue is at a lower level than that.

I suspect it didn't deal properly with the negotiation change, but on reboot it did sort itself out, which is why you could access it again.

Did you do any troubleshooting on that interface before rebooting it?

[pvaneynd]

Originally Posted by northlandboy

Did you do any troubleshooting on that interface before rebooting it?

To regain control I went to the active member of the node, discovered that I could ping the standby node. Then I jumped from the active to the standby node using ssh across the ospf-enabled network, there I noticed that the firewall was still running, but that all ospf routes were gone. I then searched for the route towards the management station and it was not in the 'netstat -rn' output, but was in the netconf.C file.

I should have also done a 'cpwd_admin list' but as we were under severe time pressure I just decided to reboot the node.

[northlandboy]

A good course of action, getting onto the active, and trying to get across from there. Which interface was it that you pinged from the active - was it the one that had problems, or another one?

cpwd_admin list probably wouldn't have shed much light on the situation. I might have been more inclined to look at the ifconfig output for that interface, and maybe ethtool <interface>.

[pvaneynd]

I used the interface that had the problems. I've also reviewed the logs on the firewall itself, but they do not tell me much.