Not responding to arp-who-has

[jcstefansson]

Recently upgraded a single member firewall to R80.10. We have about 10 static NATs automatic in objects. Outbound traffic on those NATs seems to work fine (as well as all Hide NAT traffic). However, when I tcp dump on the external for inbound traffic to those static NAT IPs, I just see arp-who-has and my own outbound traffic. I went ahead and added the IPs to the proxy arp config, but this had no effect. Merge proxy arp is enabled in global properties. Any other ideas?

[ShadowPeak.com]

Recently upgraded a single member firewall to R80.10. We have about 10 static NATs automatic in objects. Outbound traffic on those NATs seems to work fine (as well as all Hide NAT traffic). However, when I tcp dump on the external for inbound traffic to those static NAT IPs, I just see arp-who-has and my own outbound traffic. I went ahead and added the IPs to the proxy arp config, but this had no effect. Merge proxy arp is enabled in global properties. Any other ideas?

Turn off clustering from cpconfig. You accidentally enabled it.

[jflemingeds]

How could outbound nat traffic work if arp isn’t being responded to?

[ShadowPeak.com]

How could outbound nat traffic work if arp isn’t being responded to?

Because the outbound connections are all almost certainly being hidden behind the firewall's NIC address. It will always respond for that one.

[EricAnderson]

Because the outbound connections are all almost certainly being hidden behind the firewall's NIC address. It will always respond for that one.

But if the Static NAT rules come before the Hide NAT (which they will if they're all Automatic), then even the outbound connections will be source-NATed as coming from their public address. If ARP isn't working, then replies wouldn't be able to find their way back.

OP (jcstefansson), check logs and see how the "working" outbound connections are actually going out. What NAT is being used? Is the Hide NAT automatic or manual? How is your Proxy ARP set up? I've seen quite a few issues (many very recently) with ARP in R80.10, but mostly cluster-related (like "friendly" interface names for clustered interfaces replacing actual names in topology, completely breaking automatic ARP).

-E

[jcstefansson]

But if the Static NAT rules come before the Hide NAT (which they will if they're all Automatic), then even the outbound connections will be source-NATed as coming from their public address. If ARP isn't working, then replies wouldn't be able to find their way back.

OP (jcstefansson), check logs and see how the "working" outbound connections are actually going out. What NAT is being used? Is the Hide NAT automatic or manual? How is your Proxy ARP set up? I've seen quite a few issues (many very recently) with ARP in R80.10, but mostly cluster-related (like "friendly" interface names for clustered interfaces replacing actual names in topology, completely breaking automatic ARP).

-E

We rolled back to an R77.30 snapshot because of pressure from above, but here's info from last night:

Static automatic NATs are what's being used in the config. We tried creating a manual NAT rule and creating a proxy arp config, but no effect, as mentioned in OP. Prior to that, there was no proxy arp config (I can confirm in R77.30 with it working properly this is still the case).

I am thinking ShadowPeak is probably correct, I do think clusterXL was enabled, but I didn't think anything of it at the time. We're planning on cutting over again tonight, and I'll double check this. If we have no issues, I think we can conclude that was the cause.

How could outbound nat traffic work if arp isn’t being responded to?

I should clarify that when I say outbound traffic was working, I meant that I saw static NAT'd IPs going outbound towards the router, I do not think replies to these packets were coming back. Anything that operated on a Hide NAT worked normally.

[EricAnderson]

If the Automatic Hide NAT is fine, and you're seeing the outbound Static's being NATed properly, but not getting replies, then yes, this seems to be an ARP issue, and yes, ClusterXL is a very likely culprit.

When ClusterXL is enabled, both Automatic and Proxy ARP only "broadcast" on the active member. If ClusterXL is enabled on the device, but not in policy, then it's very likely that those ARP's aren't working.

Good luck tonight, and let us know.

-E

[msjouw]

In the global settings check the NAT page for ARP settings, do not forget that Cisco routers have a 4 hour ARP cache and you need to use arping to make sure to overwrite that cache or reset the routers arp cache.
Especially when you have a cluster and start using a different cluster method (ClusterXL instead of VRRP or the other way around), however that is not your case, but when cluster is on there is a chance a Virtual MAC is used.

When using static Proxy arp make sure to always push a policy after adding these commands and check with 'fw ctl arp' to see what the gateway thinks the proxy arps should be.

For sending gratuitous arp's for IP's not on any interface:
goto expert mode and execute below commands, in the second replace the IP and interface to reflect your situation:
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
arping -c 4 -A -I eth0 177.177.177.177

[jflemingeds]

We rolled back to an R77.30 snapshot because of pressure from above, but here's info from last night:

Static automatic NATs are what's being used in the config. We tried creating a manual NAT rule and creating a proxy arp config, but no effect, as mentioned in OP. Prior to that, there was no proxy arp config (I can confirm in R77.30 with it working properly this is still the case).

I am thinking ShadowPeak is probably correct, I do think clusterXL was enabled, but I didn't think anything of it at the time. We're planning on cutting over again tonight, and I'll double check this. If we have no issues, I think we can conclude that was the cause.



I should clarify that when I say outbound traffic was working, I meant that I saw static NAT'd IPs going outbound towards the router, I do not think replies to these packets were coming back. Anything that operated on a Hide NAT worked normally.

Aaah. That makes sense. One last option if clustering doesn't fix it. On your proxy arp config are using the interface name or forcing a MAC address in the proxy arp? I just hit an issue a few days ago where using the interface name didn't work but using a static mac did. This was 77.x though.

[jflemingeds]

and if none of that works you can always check for dropped packet with fw ctl zdebug drop.

[jcstefansson]

Confirmation: We did the upgrade again last night with clusterXL not enabled, and encountered no issues.

[ShadowPeak.com]

Confirmation: We did the upgrade again last night with clusterXL not enabled, and encountered no issues.

Great, thanks for the update!

[jflemingeds]

Great, thanks for the update!

Nice going, i'll buy you one of those famous $18 buds lights at vegas for your abilities assuming your going to cpx.

[ShadowPeak.com]

Nice going, i'll buy you one of those famous $18 buds lights at vegas for your abilities assuming your going to cpx.

I plan to attend CPX Vegas, whether it will be as more than just an attendee remains to be seen. :-)