Well.. this took a while to figure out.

High level

2 site to site VPN peers agree on phase 1 and 2 (using PSK and IKEv2).

peer 1 (EdgeRoutr) says.. heck yeah Phase II is great. lets do raw ESP (raw protocol 50)

peer 2 (SMB 730) says ... heck yeah Phase II is great.. let do ESP over UDP (4500) all day long!

Think this should work? In this case it sure didn't... I'm just not sure this should have worked in the first place.

We couldn't get the Edgerouter to unencapsulate the ESP over UDP packets until an option to force Nat-T was enabled on the Edgerouter. After that vpn tunnel started working (and thus the EdgeRouter stopped sending raw ESP packets).

I'm honestly not sure if this has happened before and i just haven't noticed it or not (ESP one way, ESP over UDP the other). Anyone seen something like that before? I'm thinking about it and i don't know.. if both sides say they support Nat-T does that mean they're going to use it for sure? Not sure what the trigger is. BTW both peers have routable IPs and shouldn't need Nat-T. I did find an option on the SMB that basically says force Nat-T. We didn't try disabling this on the SMB FYI.

Side note.... so many 4 letter words used against Edgerouter... which is really just Linux and StrongSwan.