PDA

View Full Version : Packets in tcpdump don't appear in fw monitor



jrrodrigo
2014-11-18, 07:35
Hi,

We are having an issue in a SG4807 kind of funny. It's a "snmp" petition whose reply doesn't appear in fw monitor, but in tcpdump it does.

The tests were made with acceleration disabled and fw monitor with "-p all" option in order to see if there was any point in the chain where packet was discarted (by the way, there is not message in "fw ctl zdebug drop" nor in Smart View Tracker).

It appears that that packet doesn't reach fw chain but it does tcpdump.


[Expert@fwcktic01]# fw monitor -p all -e "accept host(172.24.8.202) and host(213.0.254.12);"
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
in chain (12):
0: -7f800000 (f21aadc0) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 1fffff6 (f21ac0c0) (00000001) Stateless verifications (in) (asm)
2: - 1fffff5 (f2748530) (00000001) fw multik VoIP Forwarding
3: - 1000000 (f2201650) (00000003) SecureXL conn sync (secxl_sync)
4: 0 (f215ae00) (00000001) fw VM inbound (fw)
5: 1 (f21c4d00) (00000002) wire VM inbound (wire_vm)
6: 10000000 (f2207410) (00000003) SecureXL inbound (secxl)
7: 7f600000 (f21a14e0) (00000001) fw SCV inbound (scv)
8: 7f730000 (f22ee6b0) (00000001) passive streaming (in) (pass_str)
9: 7f750000 (f2457090) (00000001) TCP streaming (in) (cpas)
10: 7f800000 (f21ab150) (ffffffff) IP Options Restore (in) (ipopt_res)
11: 7fb00000 (f2419170) (00000001) HA Forwarding (ha_for)
out chain (9):
0: -7f800000 (f21aadc0) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (f2456f10) (00000001) TCP streaming (out) (cpas)
2: - 1ffff50 (f22ee6b0) (00000001) passive streaming (out) (pass_str)
3: - 1f00000 (f21ac0c0) (00000001) Stateless verifications (out) (asm)
4: 0 (f215ae00) (00000001) fw VM outbound (fw)
5: 1 (f21c4d00) (00000002) wire VM outbound (wire_vm)
6: 10000000 (f2207410) (00000003) SecureXL outbound (secxl)
7: 7f700000 (f2459100) (00000001) TCP streaming post VM (cpas)
8: 7f800000 (f21ab150) (ffffffff) IP Options Restore (out) (ipopt_res)
monitor: monitoring (control-C to stop)
[fw_2] Mgmt:i0 (IP Options Strip (in))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:i1 (Stateless verifications (in))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:i2 (fw multik VoIP Forwarding)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:i3 (SecureXL conn sync)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:i4 (fw VM inbound )[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I6 (SecureXL inbound)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I7 (fw SCV inbound)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I8 (passive streaming (in))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I9 (TCP streaming (in))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I10 (IP Options Restore (in))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I11 (HA Forwarding)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] Mgmt:I12 (Chain End)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:o0 (IP Options Strip (out))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:o1 (TCP streaming (out))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:o2 (passive streaming (out))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:o3 (Stateless verifications (out))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:o4 (fw VM outbound)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:O6 (SecureXL outbound)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:O7 (TCP streaming post VM)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:O8 (IP Options Restore (out))[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
[fw_2] eth1:O9 (Chain End)[67]: 172.24.8.202 -> 213.0.254.12 (UDP) len=67 id=0
UDP: 59325 -> 161
monitor: caught sig 2
monitor: unloading
[Expert@fwcktic01]#
[Expert@fwcktic01]#

The reply packet can be seen here:


[Expert@fwcktic01]# tcpdump -ni eth1 host 172.24.8.202 and host 213.0.254.12
tcpdump: listening on eth1
11:43:50.769262 172.24.8.202.59325 > 213.0.254.12.snmp: C=p3l3t4 GetNextRequest(24) .1.3.6.1.2.1 (DF)
11:43:50.770597 213.0.254.12.snmp > 172.24.8.202.59325: C=p3l3t4 GetResponse(37) .1.3.6.1.2.1.1.1.0="Cisco I"


Thanks.

JRRR

jdmoore0883
2014-11-18, 11:02
The fact that fw monitor does not show any of the packets is a clear indication that the packets are not being accepted up into the firewall kernel for inspection. We will see this kind of behavior with ARPs as well (for example). Try running the tcpdump with the '-e' flag to show MAC Addresses. Check to see that the proper MAC Addresses are in the packet. Also, do you see any ARPs for the firewall that it is not responding to? Or any other ARPs for that relevant traffic?

ShadowPeak.com
2014-11-18, 13:15
The fact that fw monitor does not show any of the packets is a clear indication that the packets are not being accepted up into the firewall kernel for inspection. We will see this kind of behavior with ARPs as well (for example). Try running the tcpdump with the '-e' flag to show MAC Addresses. Check to see that the proper MAC Addresses are in the packet. Also, do you see any ARPs for the firewall that it is not responding to? Or any other ARPs for that relevant traffic?

Yep use -e to check the destination MAC address of the replies, they are almost certainly not that of the firewall. Your local switch could be flooding all frames to that MAC address since it doesn't exist on your network. The packets will show up in tcpdump because it sets the interface to promiscuous mode while it runs. Rather than messing around with mac addresses and -e, try running tcpdump in non-promiscuous mode with the -p option. If the replies are no longer visible those frames will never be passed up to the INSPECT module for processing.

jflemingeds
2014-11-18, 14:31
Yep use -e to check the destination MAC address of the replies, they are almost certainly not that of the firewall. Your local switch could be flooding all frames to that MAC address since it doesn't exist on your network. The packets will show up in tcpdump because it sets the interface to promiscuous mode while it runs. Rather than messing around with mac addresses and -e, try running tcpdump in non-promiscuous mode with the -p option. If the replies are no longer visible those frames will never be passed up to the INSPECT module for processing.

Correct me if i'm wrong, but isn't it possible that a connection accelerated by securexl isn't going to show in a fw mon output because its not longer being passed all the way up the connection stack and just being passed around via securexl? I think this is also the reason support asks for fw mon output with securexl disabled. I have also seen issues using -p all before. May not apply, but i had issues where it wasn't working correctly (connection would show without -p and would not with -p).

That being said, i'm not a huge fan of fw mon. I've never really found it that helpful. At best i've had it show me something is being dropped, which always leads to debugging something else. I know it also shows natting but tracker does to (in my cases!). Also fw mon can't show layer 2 so you can miss packets on connectivity issues (not a lot happening down there, but it can be very useful).

msjouw
2014-11-18, 14:58
I can't agrre to the point that FW mon is not useful, we us both tcpdump[ and fw mon and the latter is mostly showing you where problems are when communications are not getting where they should be, the Interface information is very useful there.
We run into a lot of routing issues during setups of customer networks and knowing which of the 10 (sub)interfaces it is leaving or entering the firewall can just save the day.

jdmoore0883
2014-11-18, 15:01
In the end, fw monitor and tcpdump both show different details of a packet at different points in the firewall.

tcpdump shows us what is "on the wire"; what is actually being seen on the interface, at the interface level.

fw monitor shows us the firewall kernel, and the various points within.

Indeed, SecureXL *WILL* mess up the output of all fw monitors. Anytime I have to run an fw monitor, I turn off SecureXL (fwaccel off), otherwise, any fw monitor outputs are more or less useless.

jflemingeds
2014-11-18, 20:05
I can't agrre to the point that FW mon is not useful, we us both tcpdump[ and fw mon and the latter is mostly showing you where problems are when communications are not getting where they should be, the Interface information is very useful there.
We run into a lot of routing issues during setups of customer networks and knowing which of the 10 (sub)interfaces it is leaving or entering the firewall can just save the day.

Well, tracker logs the in interface and clish - show route destination $ip on src and dst ips is a much better low impact way to trouble shoot routing issues. I mean you pretty much know if the input nic in tracker doesn't match show route destination $src_ip then something is very wrong and you didn't have punch the firewall in the gonads with a packet capture.

Anyway, it doesn't really matter if what your doing doesn't impact anything and you get to the answer. I was just saying for me, it hasn't been as useful a tcpdump + fw ctl zdebug drop.

jflemingeds
2014-11-18, 20:11
In the end, fw monitor and tcpdump both show different details of a packet at different points in the firewall.

tcpdump shows us what is "on the wire"; what is actually being seen on the interface, at the interface level.

fw monitor shows us the firewall kernel, and the various points within.

Indeed, SecureXL *WILL* mess up the output of all fw monitors. Anytime I have to run an fw monitor, I turn off SecureXL (fwaccel off), otherwise, any fw monitor outputs are more or less useless.

brings up a point. I do wonder which would be better performance wise on say a 10 gig interface.

tcpdump + bpf filter

disabled securexl and fw mon + filter.

Personally i would feel safer with tcpdump, but i can't say i have any data to show that its the safer option so thats just an opinion.


Where are we with the issue anyway?

ShadowPeak.com
2014-11-18, 22:21
brings up a point. I do wonder which would be better performance wise on say a 10 gig interface.

tcpdump + bpf filter

disabled securexl and fw mon + filter.

Personally i would feel safer with tcpdump, but i can't say i have any data to show that its the safer option so thats just an opinion.


Where are we with the issue anyway?

I would feel safer with tcpdump as well. When a fw monitor is started it actually inserts a new chain module at the requested inspection points. You can run fw ctl chain to see the chain modules and if fw monitor is running you'll see a new chain module, sometimes up to 4 of them depending on your inspection points mask. So technically fw monitor is inserting itself "in-line" on your live traffic flow through the INSPECT module. Sounds a tad risky but I've never seen that method cause a problem. This is why an active fw monitor will get killed if the policy is reinstalled to the firewall since rebuilding the chain module sequences is a part of the policy load operation.

However with a tcpdump capture, libpcap is leveraged to essentially "register" to get traffic being sent or received on a particular interface(s). When a tcpdump is not running, the only entity registered to get traffic from an interface is the INSPECT driver (or the acceleration/SecureXL layer if SecureXL is active). When a tcpdump starts libpcap registers as well, and all traffic is split (or teed) and both INSPECT/SecureXL and libpcap gets a copy of the sent/received packets. So in this respect a tcpdump capture is not "in-line" quite as much as fw monitor although both techniques are sharing various kernel resources so there could always be a performance and/or stability issue possible.

With tcpdump in general even if SecureXL is active you should always see a packet inbound to an interface arrive but you may not be able to see it leave. I can seem to vaguely remember situations where tcpdump could not see packets arrive or leave but I think that was on Nokia IPSO which had some acceleration built right into IPSO that was independent of Check Point's code. Also bear in mind that tcpdump by default will put the monitored interface in promiscuous mode (but not if you specify interface "any" or -p) so you may see packets with tcpdump that would never make it to the SecureXL/INSPECT module. With fw monitor one might not see a packet arrive or leave if SecureXL is active and the packets in question are handled exclusively by the SecureXL/acceleration layer which is independent of the Firewall Path (INSPECT module).

Edit: Also note that SecureXL can be disabled on a per-interface basis to get a good capture without having to turn off all acceleration and subsequently auto-affinity. See my post here:

https://www.cpug.org/forums/showthread.php/18641-I-only-see-syn-packets-when-I-do-a-capture?p=85334#post85334

jflemingeds
2014-11-18, 22:49
I would feel safer with tcpdump as well. When a fw monitor is started it actually inserts a new chain module at the requested inspection points. You can run fw ctl chain to see the chain modules and if fw monitor is running you'll see a new chain module, sometimes up to 4 of them depending on your inspection points mask. So technically fw monitor is inserting itself "in-line" on your live traffic flow through the INSPECT module. Sounds a tad risky but I've never seen that method cause a problem. This is why an active fw monitor will get killed if the policy is reinstalled to the firewall since rebuilding the chain module sequences is a part of the policy load operation.

However with a tcpdump capture, libpcap is leveraged to essentially "register" to get traffic being sent or received on a particular interface(s). When a tcpdump is not running, the only entity registered to get traffic from an interface is the INSPECT driver (or the acceleration/SecureXL layer if SecureXL is active). When a tcpdump starts libpcap registers as well, and all traffic is split (or teed) and both INSPECT/SecureXL and libpcap gets a copy of the sent/received packets. So in this respect a tcpdump capture is not "in-line" quite as much as fw monitor although both techniques are sharing various kernel resources so there could always be a performance and/or stability issue possible.

With tcpdump in general even if SecureXL is active you should always see a packet inbound to an interface arrive but you may not be able to see it leave. I can seem to vaguely remember situations where tcpdump could not see packets arrive or leave but I think that was on Nokia IPSO which had some acceleration built right into IPSO that was independent of Check Point's code. Also bear in mind that tcpdump by default will put the monitored interface in promiscuous mode (but not if you specify interface "any" or -p) so you may see packets with tcpdump that would never make it to the SecureXL/INSPECT module. With fw monitor one might not see a packet arrive or leave if SecureXL is active and the packets in question are handled exclusively by the SecureXL/acceleration layer which is independent of the Firewall Path (INSPECT module).

Edit: Also note that SecureXL can be disabled on a per-interface basis to get a good capture without having to turn off all acceleration and subsequently auto-affinity. See my post here:

https://www.cpug.org/forums/showthread.php/18641-I-only-see-syn-packets-when-I-do-a-capture?p=85334#post85334

ok so how about this, which is safer. fw monitor (no -p) or tcpdump -i any ? :)

BTW i did see that other post after i made these. Very informative, your posts are so high quality. I hope everyone is paying attention, because Yoda is in the house. I really like pointing out how switch flooding would explain this issue (which keeps getting OT).

Side note (ok back to OT) are you doing anything at cpug con (assuming that happening)? If so i would push to go to it.

ShadowPeak.com
2014-11-19, 10:32
ok so how about this, which is safer. fw monitor (no -p) or tcpdump -i any ? :)

BTW i did see that other post after i made these. Very informative, your posts are so high quality. I hope everyone is paying attention, because Yoda is in the house. I really like pointing out how switch flooding would explain this issue (which keeps getting OT).

Side note (ok back to OT) are you doing anything at cpug con (assuming that happening)? If so i would push to go to it.

Probably a draw in those two scenarios, but I'd still give tcpdump a slight edge. If the tcpdump process cannot empty the packet buffer filled by the libpcap driver fast enough you'll see a nonzero value for "dropped by kernel" when the tcpdump terminates. Generally a polite clue that your reach is exceeding your grasp and you need to get more specific with your parameters. Newer versions permit setting the buffer to larger than the default 2MB via the -B option but the somewhat old version of tcpdump shipped with Gaia does not appear to permit that. Feature request to update the version of tcpdump for R80 maybe?

Hadn't thought too much about CPUGCon; never been to one before. Once dates & agenda are firmed up I'll consider it.

jflemingeds
2014-11-19, 12:06
Probably a draw in those two scenarios, but I'd still give tcpdump a slight edge. If the tcpdump process cannot empty the packet buffer filled by the libpcap driver fast enough you'll see a nonzero value for "dropped by kernel" when the tcpdump terminates. Generally a polite clue that your reach is exceeding your grasp and you need to get more specific with your parameters. Newer versions permit setting the buffer to larger than the default 2MB via the -B option but the somewhat old version of tcpdump shipped with Gaia does not appear to permit that. Feature request to update the version of tcpdump for R80 maybe?

Hadn't thought too much about CPUGCon; never been to one before. Once dates & agenda are firmed up I'll consider it.

yeah updated tcpdump would be nice. I would assume it would also support the option to rotate pcap files once they hit a size. I think its the -W option? I've used nohup tcpdump bla -w - | gzip > tcpdump.cap.gz & to keep the size down.

Only trick is you have to kill tcpdump with a sig interrupt (which is what ctrl-c is I think) to get it to cleanly shutdown, which makes gzip happy otherwise you'll loose some data at the end.

jrrodrigo
2014-11-26, 05:20
Hi,

We have found the problem, searching the involved MACs with tcpdump -e. The packet actually reach the firewall, but using the physical MAC, not the VMAC like the other valid packets being processed by the firewall. It seems it doesn't reach the kernel firewall module due to this behavior. The remote machine (a router) is using the wrong MAC because it has a loopback direction configured in the VLAN of the other remote machine and for some reason it's creating the packet with the physical MAC.

Thank you all.

ShadowPeak.com
2014-11-26, 16:41
Hi,

We have found the problem, searching the involved MACs with tcpdump -e. The packet actually reach the firewall, but using the physical MAC, not the VMAC like the other valid packets being processed by the firewall. It seems it doesn't reach the kernel firewall module due to this behavior. The remote machine (a router) is using the wrong MAC because it has a loopback direction configured in the VLAN of the other remote machine and for some reason it's creating the packet with the physical MAC.

Thank you all.

Interesting, thanks for the follow-up.