dropped by fw_filter_chain Reason: chain hold failed

[Irek_Romaniuk]

I have traffic dropped on firewall for some users, see below example , source 10.29.121.88. Drop is seen only on 'fw ctl zdebug drop' , nothing in Tracker or Smartlog. It's the same after I made an IPS exception for destination 10.94.7.211.

[Expert@VSX-01:5]# fw ctl zdebug drop ^ grep 10.94.7.211
^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:52109 -^ 10.94.7.211:443 dropped by fw_filter_chain Reason: chain hold failed^
^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:42223 -^ 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed^
^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:42224 -^ 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed^

I'm not using domain objects

[Cory Webb]

I would double check the rulebase again and make doubly sure there arent any domain objects being used and if there are any then to remove them

[gyovel]

Hi

the hold_table is being filled up, most likey you use a rule containing domain objects on the beginning of the rulebase.
almost each packet that enters the rulebase and reaches this specific rule, needs to be resolved to check if it
matches the domain object and as a result the connection is listed on the hold_table.
if there is high traffic, the hold_table might be full and lead to traffic drop.

Move to the domain object rule towards the end of the rulebase and this will decrease the amount of connections being listed under the hold_table.

Regards,
Guy

[Irek_Romaniuk]

I'm not using domain objects

[ShadowPeak.com]

I'm not using domain objects

I seem to remember that there are some limited scenarios where this can be caused by Identity Awareness, specifically if there are invalid or unreachable domain servers configured. Assuming you are using IA, what does the "adlog a dc" command show when run on the firewall?

[Irek_Romaniuk]

I'm not using domain objects, but definitely hold_table size is growing when I generate traffic from specific source..Questions is why

[Expert@VSX-01:5]# fw tab -t hold_table -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             hold_table                       8183     9   200       0
[Expert@VSX-01:5]# fw tab -t hold_table -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             hold_table                       8183    22   200       0

Also when I look inside hold_table and translate hex to ip , I don't see that specific source

[ShadowPeak.com]

I'm not using domain objects, but definitely hold_table size is growing when I generate traffic from specific source..Questions is why

[Expert@VSX-01:5]# fw tab -t hold_table -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             hold_table                       8183     9   200       0
[Expert@VSX-01:5]# fw tab -t hold_table -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             hold_table                       8183    22   200       0

Also when I look inside hold_table and translate hex to ip , I don't see that specific source

The hold_table is used to keep track of packets that the firewall kernel has sent to a daemon process for handling. The classic case is of course DNS lookups for domain objects but it is used for many other things. I would have a look at what is going on in user/process space on the firewall:

cpwd_admin list (any processes crashing and restarting?)
top (any daemon processes eating huge amounts of CPU off in the weeds somewhere?)
ls -ltr $FWDIR/log (any daemon log files getting constantly written to with errors/warnings?)
enabled_blades (any features enabled that rely on daemons for processing?)

Also see my posting above about IA possibly being involved...

[Irek_Romaniuk]

I missed that IA comment, thnx.

[Expert@VSX-01:5]# enabled_blades
fw ips identityServer

But 'adlog a dc' in 'Connection state' gives solid 'has connection'

'cpwd_admin list' showing [21:25:48] 15/4/2015 for all processes which is time/day when I applied R77_20_jumbo_hf, 'top' is not bad since that time;) Will look again

But 'stickness' to specific source IP addresses bothers me the most

See below , I spoofed source IP (1) and it was dropped. And I spoofed another IP (2) and got response from server, magic ? Very repeatable.

1) #sudo nmap -e eth0 -Pn -S 10.29.121.88 10.94.7.211 -p 80

[Expert@VSX-01:5]# fw ctl zdebug drop | grep 10.29.121.88
;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.29.121.88:50352 -> 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed;
;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.29.121.88:50353 -> 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed;

2) # sudo nmap -e eth0 -Pn -S 10.29.121.244 10.94.7.211 -p 80

[Expert@VSX-01:5]# tcpdump -ni Lan5.373 host 10.29.121.244
07:41:27.889310 IP 10.29.121.244.59291 > 10.94.7.211.http: S 3853270289:3853270289(0) win 3072 <mss 1460>
07:41:27.889411 IP 10.94.7.211.http > 10.29.121.244.59291: S 3866447525:3866447525(0) ack 3853270290 win 8192 <mss 1460>

When looking at 'fw monitor' I can see only 'i' in the first case...

And $FWDIR/log/ is showing IA logs, but nothing interesting there. I am surprised these are present without running debug thou

[Expert@VSX-01:5]# ls -ltr $FWDIR/log/
-rw-rw-r-- 1 admin root    3002209 Apr 19 16:48 pdpd.elg
-rw-rw-r-- 1 admin root    5692894 Apr 19 16:58 pepd.elg

[Irek_Romaniuk]

Hm.. there is predictability in this madness. The rule allowing traffic for the subnet 10.29.121.0/24 is IP, not IA based.
I can pick any address from this allowed subnet not being identified by IA and it will have the same problem !
'fw monitor' will stops after 'i'. And any address from allowed subnet which is recognized by IA is fine, wow..does it make any sense ?

So simple rule:

If AD Query finds the owner of IP_Address 
[Expert@VSX-01:5]# adlog a query ip IP_Address
OUTPUT

If OUTPUT not empty then IP_Address is passing that IP, not IA based rule;)

[ShadowPeak.com]

I'd say it is almost certainly IA causing the drops based on the output of enabled_blades, although based on the non-IA rules getting caught in this you could quickly try disabling IPS with an "ips off" command, initiate some new traffic to see if the drops are still occurring, then re-enable IPS with an "ips on" command just to rule IPS out.

However IA shouldn't need to hold anything up like that unless it is having to shovel a captive portal at a user and waiting for a response or possibly attempting communication with the optional Identity Awareness Agent on a user's system or the Terminal Server Agent. What IA checkboxes do you have enabled on your gateway (i.e. Browser-Based Authentication, AD Query, Identity Agents, Terminal Servers, Remote Access)?

[Irek_Romaniuk]

Thnx, only AD Query enabled. I put an network exception in IPS at the very begining and it didn't help

[Irek_Romaniuk]

Looks like moving IP based rules UP before IA rules fixes the issue... I can see 'iIoO' in 'fw monitor' now after I moved IP rule before not directly related IA rule

[ShadowPeak.com]

Looks like moving IP based rules UP before IA rules fixes the issue... I can see 'iIoO' in 'fw monitor' now after I moved IP rule before not directly related IA rule

Thanks for the followup, interesting. Are you running R75.47 or R77.20? There were quite a few IA fixes included in those particular releases.

[Irek_Romaniuk]

I have 77.20 jumbo patched week ago;)

[Irek_Romaniuk]

plus I was told by CP that root cause was that I was obtaining IA on my VS here from 3 other VSs (10.93.17.1, 10.83.21.1, 10.73.41.20)

[Expert@VSX-01:5]# pep show network pdp | grep 10.29.121.
| 10.29.121.0 | 255.255.255.0 | <10.93.17.1,0>;<10.83.21.1,0>;<10.73.41.20,0>; |

I don't believe until I test it

[philuxe]

plus I was told by CP that root cause was that I was obtaining IA on my VS here from 3 other VSs (10.93.17.1, 10.83.21.1, 10.73.41.20)

[Expert@VSX-01:5]# pep show network pdp | grep 10.29.121.
| 10.29.121.0 | 255.255.255.0 | <10.93.17.1,0>;<10.83.21.1,0>;<10.73.41.20,0>; |

I don't believe until I test it

Hi,

Did you manage to resolve this issue ? I have the same setup (IA on VSX with identity sharing between gateways)

Also I noticed that it makes the hit counts wrong in smartdashboard : my IA rule is just before the drop/any/any but 99% of the dropped traffic is accounted against that rule ...

[philuxe]

any news on this issue ? I think i'm going to stop using IA because of that problem.

last option would be to switch to the new Checkpoint Identity Collector agent ?

regards