Dropping connections for IP's ending in .255

[fdamstra]

I have a strange problem, and while I can't guarantee that it's the checkpoint that's dropping the traffic, it looks like it is, so I'll start here.

The scenario: Users with IP addresses that end in '.255' are unable to access web servers of ours on a particular VLAN.

Here's the architecture:
2 NGX R65 HFA2 Enforcement Nodes in an active-active ClusterXL configuration are connected to a single Cisco 2924XL switch. The switch is in turn connected to a pair of F5 BigIP load balancers in an active-standby configuration. The web servers sit behind these F5's.

The connections from the switch to the firewall and to the F5's are 802.1q trunks. There are two VLAN's going across these trunks: VLAN 80 and VLAN 443.

Users with IP addresses ending in 255 are able to access the web sites on VLAN 80 without issue. However, they are completely unable to access sites on VLAN 443.

Using port mirroring on the switch, I see the SYN from the user to the web server, and then SYN-ACK response sent out the port to the firewall. However, running 'tcpdump' or 'fw mon' on the firewalls, I see only the SYN. (I never see the SYN-ACK response).

So, I'm led to believe that either:
A) The switch isn't properly sending out the frame, or
B) The firewall is dropping the frame.

Because it's only affecting users with IP addresses ending in .255, I tend to think it's not the switch, since it's only a L2 switch and should not be paying any attention to the IP address.

Any thoughts?

[RayPesek]

Are these internal users or external users?

Since .255 addresses are usually reserved for broadcasts, I suspect something thinks they are broadcasts and not users.

Ray

[fdamstra]

These are external users, assigned a .255 address by their ISP.

I agree with the theory that something is assuming that IP's ending in 255 are broadcasts and dropping them, but that would be a bug in that software, not anything wrong with the 255 address.

It's a bizarre problem. Checkpoint Support started off with "I find this hard to believe." And I don't blame them.

The user can get to web servers that are off other interfaces, even other VLAN'd interfaces off the same physical interface. So it's just this one particular VLAN that's having an issue. However, other users are able to communicate to web servers on this VLAN just fine. (to the tune of 30k+ hits/day)

And the capture from the switch shows SYN-ACK packets going out the interface to the firewall, but the capture from the firewall doesn't show these packets. So the problem has to be either the firewall or the switch. Since the switch is a standard L2 Cisco switch, it shouldn't even be looking at the IP address, so I have to blame the firewall.

Checkpoint is having me do a kernel debug next time I can arrange it with the user.

[Thorpuse]

For Network objects, you can specify whether the broadcast address is included or not included in the object properties. You may need to toggle this for the relevant network object.

[fdamstra]

Originally Posted by Thorpuse

For Network objects, you can specify whether the broadcast address is included or not included in the object properties. You may need to toggle this for the relevant network object.

It's a good thought, but this is a regular Internet user who is attempting to use our a web application that our company provides. There is no network object defined for him. (The rule is allow source: any, destination: webserver, protocol: http/https)

The fact that a 'tcpdump -i eth1.443' on the interface doesn't show the SYN-ACKs concerns me, as I would think I'd see every packet, even if the firewall was discarding it. So it seems to me that it's the NIC or NIC driver that must be dropping the packet.

The are Intel Pro/1000 quad port cards, and on the list of supported NIC's.

[fdamstra]

I wanted to post an update to this, as some further experiments have determined that this is a problem related to ClusterXL Unicast Load-sharing.

We tried with a couple different IP addresses that ended in .255. We found that sometimes it would work for one vlan, sometimes for the other, and sometimes for neither. Further investigation revealed that when the traffic went through the pivot, the connection was successful, but when it went through the secondary firewall, the connection was unsuccessful.

So, we did a 'cpstop' on the pivot (failed over to the secondary), and all traffic was successful. Similarly with failing over to just the primary firewall. Provided there is only one node in the cluster, the traffic is successful.

We are planning on moving to new mode high availability to see if this solves the problem. Even so, we have enough traffic going through these firewalls that that's not an ideal solution, so we're continuing to push CheckPoint support for a fix.

(Sidenote: We're running Load Sharing Unicast because Load Sharing Multicast prevented all traffic to Cisco devices, though all other hosts seemed to be fine)

[Thorpuse]

Wow... nice find, that definitely sounds like a bug. I'd suggest you get your local friendly CP SE involved in getting that fixed as well.