PDA

View Full Version : CP resetting connections



*tomo*
2015-12-16, 05:30
Hi,

Any thoughts about my issue. At customer site they implement new forward proxy in DMZ, sometimes user can't open pages with message that peer reset connection. They told me, that they see lot of RST packets hiting proxy.
I sniffed taffic with tcpdump on outside an dmz at same time, and on outside I see RST-ACK, an on dmz I see RST and RST-ACK. This would mean that checkpoint is resetting connections.
fw ctl zdebug shows me that firewall is droping packets from proxy with reaseon:
"dropped by fwpslglue_chain Reason: PSL Reject: HTTP_PSL;"
All I could find on knowladgebase is that this appear when you use Legacy URL filtering on versions 75.20 and above, which is not the case.
btw I'm using 77.10 on splat in HA cluster.
Connection table is on 1/3 of defined capacity
proxy is exempted from network quota and non compliant HTTP IPS protections (although not as an object, but as a member of group)

Regards

jflemingeds
2015-12-16, 10:50
You're going to need a more in depth kernel debug. I'll take a look at some debugging options. Might have to do a full kdebug.

jflemingeds
2015-12-16, 10:57
Try this, may increase load so do off hours if possible.

fw ctl zdebug drop tcp-str spii

Should give more info on the reason for the drop. If not you might try this (even more output and load).

fw ctl zdebug conn drop vm tcp-str spii

Look for the drop string you already found and then look around that.

laf_c
2015-12-17, 04:50
Try this, may increase load so do off hours if possible.

fw ctl zdebug drop tcp-str spii

Should give more info on the reason for the drop. If not you might try this (even more output and load).

fw ctl zdebug conn drop vm tcp-str spii

Look for the drop string you already found and then look around that.

Hi mate,

What's the difference between fw ctl zdebug + drop vs fw ctl zdebug drop vs fw ctl zdebug drop tcp-str spii

jflemingeds
2015-12-17, 11:16
Hi mate,

What's the difference between fw ctl zdebug + drop vs fw ctl zdebug drop vs fw ctl zdebug drop tcp-str spii

drop, tcpstr spii conn and vm are all flags passed to the fw module. Each flag turns on debugging at a different area.

drop is not much more then what tracker would log.

tcp-str i don't remember what this is but i think its related to tcp streaming. I'm hoping passive streaming.
spii is related to IPS i think.
conn and vm show much greater detail of the packet passing through the firewall kernel.

jdmoore0883 might be able to explain each of those better since he has access to the goods.

The issue is drop isn't logging enough info about the drop so you need a debug set deeper in the kernel to see where. PSL is passive streaming library, which i'm pretty sure is related to IPS. Once the correct debug is enabled you should see something about the IPS signature that is dorking things up.


zdebug itself is really just a short cut for set debug flags with fw ctl debug -m fw + drop tcp-str, setting the buffer and doing a fw ctl kdebug -T -f but it only (i think) applies to the firewall module vs say the clustering module or vpn or etc.

If you want to see all the modules and flags look around for

fw ctl debug

I didn't spend much time looking on google/checkpoint.

jdmoore0883
2015-12-17, 12:04
jflemingeds is MOSTLY bang-on... I will add a few more details.


drop is not much more then what tracker would log.

A - Not quite... Tracker only logs the FW rules that are set to log. So, if a rule that is dropping the traffic is NOT set to log, you will not see it in Tracker, but you will in the zdebug drop. In addition, there are implied rules that may or may not be logged, as well as errors of various kinds that will also not be logged. 'fw ctl zdebug drop' is *THE* *DEFINITIVE* source of whether or not a packet is being dropped by the firewall.


tcp-str i don't remember what this is but i think its related to tcp streaming. I'm hoping passive streaming.

A - You got it!


spii is related to IPS i think.

A - spii = Stateful Protocol Inspection Infrastructure (INSPECT streaming). This is the hear of the stateful inspection


conn and vm show much greater detail of the packet passing through the firewall kernel.

A - Correct.
conn = connection information and identification processing
vm = Virtual Machine chain decisions on traffic going through fw_filter_chain


jdmoore0883 might be able to explain each of those better since he has access to the goods.

A - Oh no my secret is out...! Though I won't be going around advertising it ;) It's like my hidden super power.


PSL is passive streaming library, which i'm pretty sure is related to IPS.

A - Correct, PSL is the heart of IPS. This is the mechanism that "watches" multiple packets over the course of the entire connection.


zdebug itself is really just a short cut for set debug flags with fw ctl debug -m fw + drop tcp-str, setting the buffer and doing a fw ctl kdebug -T -f

A - Correct.


but it only (i think) applies to the firewall module vs say the clustering module or vpn or etc.

A - Kinda... TECHNICALLY, it can be used for ANY and ALL debugs. This command will print all the debugs straight to the console or SSH session, and not to a file. Since most of the really technical debugs create WAY too many logs for this, we need to use the other command to log it all to a file. Typically, we only see the drops here because it produce far fewer logs, and can easily be grep'ed for an IP or port for quick, live troubleshooting.



If you want to see all the modules and flags look around for

fw ctl debug

A - Correct, though the output will only give you the flags, and no explanations. Search Chekpoint's SK Center for "Kernel Debug Flags", and we have a corresponding document for MOST Checkpoint versions.

HTH

ShadowPeak.com
2015-12-17, 13:31
Great post jdmoore0883, just to clarify one thing you said:


'fw ctl zdebug drop' is *THE* *DEFINITIVE* source of whether or not a packet is being dropped by the firewall.

To further clarify when jdmoore0883 says "firewall" here, it means Check Point's code which includes the INSPECT driver and even SecureXL. If a packet is discarded by a Linux OS function such as the Ethernet driver or IP, it will not show up in the fw ctl zdebug drop output. So if you see a packet arriving on an interface with tcpdump and don't see it leave in a "roach-motel" fashion, and fw ctl zdebug drop shows nothing about that packet, losing it had absolutely nothing to do with Check Point's code. You may be wondering how that heck that can occur, well here are two examples:

1) IP does not have a route to the destination network at all. Commonly manifests when the default route is missing or misconfigured upon cutting a new firewall into production, or more rarely the IP driver itself has an explicit drop/reject route (equivalent of null0 for you Cisco folks) to the destination IP address and throws it directly into the bit bucket.

2) You've just cut a new firewall into production and nothing can pass through it. The surrounding Layer 3 devices still have the MAC address of the old firewall and are sending frames to the wrong destination MAC address. The frames will show up in a tcpdump since it puts the capturing interface in promiscuous mode (and the switch is flooding the frame to all ports because it doesn't know where the firewall's old MAC address went), but the NIC driver will not actually pick up the frame off the wire and send it up the stack since the destination MAC doesn't match its own. Adding the -e option to tcpdump will show the MAC addresses and allow you to confirm if this is happening. Either clear the ARP cache on the Layer 3 device, or see my post here if you don't have access to the Layer 3 device for one reason or another: https://www.cpug.org/forums/showthread.php/20026-HOWTO-Deal-with-stale-ARP-issues-on-adjacent-routers

fw monitor can also be used to diagnose these; in situation 1 the packet will be seen at iI and no further, in situation 2 the packet will never arrive at i at all. These scenarios illustrate that there is a very clear line of demarcation between the Linux code and Check Point's code, even if it can get pretty blurry sometimes...

jflemingeds
2015-12-17, 13:48
geez man.. why don't you write a book about.. oh.. right.

ShadowPeak.com
2015-12-17, 14:00
geez man.. why don't you write a book about.. oh.. right.

Good thing my headset mic was muted when I read your post, as I'm teaching a class online today and the attendees currently doing lab exercises would have wondered what was wrong with me as I laughed my a** off.

laf_c
2015-12-18, 09:38
Good thing my headset mic was muted when I read your post, as I'm teaching a class online today and the attendees currently doing lab exercises would have wondered what was wrong with me as I laughed my a** off.

This got some speed for sure.
Thanks for the input gentlemen!

*tomo*
2015-12-23, 08:29
Hi,

Great stuff, this one goes to bookmarks, I'm sure I'll need it.
Btw we found out that IPS protection EOT files was causing this.

Tnx

jflemingeds
2015-12-23, 11:20
Hi,

Great stuff, this one goes to bookmarks, I'm sure I'll need it.
Btw we found out that IPS protection EOT files was causing this.

Tnx

So how did you figure that out?