PDA

View Full Version : HOWTO: Deal with stale ARP issues on adjacent routers



ShadowPeak.com
2014-09-22, 16:27
Note: I originally posted this over at CPShared but wanted this article added for posterity at CPUG as well...

THE ISSUE

A new firewall has been procured and you've configured it with all the same IP addresses/routing and security policy as the old firewall. When cut into production, traffic doesn't seem to be flowing between the new firewall and a directly-attached Layer 3 switch/router. Running a:


Expert# tcpdump -eni ethX

Shows outbound traffic leaving your firewall to the router and replies coming back, but the destination MAC address on the reply frames is still showing the MAC address of the old firewall.

THE SOLUTION

Obviously the optimal solution is to clear the ARP cache on the router, but what if administrative access to it cannot be obtained? Physical access will of course allow a quick power-cycle of the router to clear the cache; one can also try unplugging the physical router interface which may flush any ARP cache entries associated with that interface. With a cluster of firewalls, initiating a failover will cause the new active member to send a gratuitous ARP for the firewall's cluster address and all NAT addresses for which proxy ARP is being performed to solve the problem (this assumes of course that "Use cluster MAC" is not set on the cluster object).

However if there is just a single firewall and not a cluster is there a way to force the firewall to send a gratuitous ARP not only for itself but all NAT addresses that it is supposed to perform proxy ARP for?

THE ARPING COMMAND

Built into SecurePlatform and Gaia is a little-known tool called arping. This command allows the creation of ARP traffic including gratuitous ARPs. For example if the external IP address of your firewall is 129.82.102.32/24 on interface eth1, the following command will cause the firewall to send 4 gratuitous ARPs for its own address, thus updating the ARP cache of any directly adjacent routers on that interface:


Expert# arping -c 4 -A -I eth1 129.82.102.32
ARPING 129.82.102.32 from 129.82.102.32 eth1
Sent 4 probes (4 broadcast(s))
Received 0 response(s)

This works great for the firewall's address, but what about all those other IP addresses we have plucked from 129.82.102.0/24 for purposes of NAT like 129.82.102.222?


Expert# arping -c 4 -A -I eth1 129.82.102.222
bind: cannot assign requested address


Oops. Well as it turns out this can be made to work even for addresses that are not physically assigned to interface eth1:


Expert# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
Expert# arping -c 4 -A -I eth1 129.82.102.222
ARPING 129.82.102.222 from 129.82.102.222 eth1
Sent 4 probes (4 broadcast(s))
Received 0 response(s)


Taking this a step further, the command fw ctl arp shows a list of addresses plucked from our ISP-assigned range for which the firewall believes it needs to perform proxy ARP so NAT will work for those addresses. For arping to send gratuitous ARPs for every IP address in this list the following command sequence will strip out the IP addresses and feed them one by one to the arping command:


Expert# fw ctl arp | cut -d\( -f2 | cut -d\) -f1 | xargs -i -t arping -c 4 -A -I eth1 {}

This command will update the ARP cache of a directly-attached Layer 3 switch/router without having to access it or wait the usual default ARP timeout of 4 hours. Obviously this will not work with a router that has been configured to ignore gratuitous ARPs

avdonzzz
2014-09-24, 01:34
Wow I didnt know about the ARPING command, thanks for sharing!

chillyjim
2014-09-24, 18:46
Wow I didnt know about the ARPING command, thanks for sharing!

Don't feel bad, I didn't know about either and I've been using Check Point 20 years.