CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 17 of 17

Thread: dropped by fw_filter_chain Reason: chain hold failed

  1. #1
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default dropped by fw_filter_chain Reason: chain hold failed

    I have traffic dropped on firewall for some users, see below example , source 10.29.121.88. Drop is seen only on 'fw ctl zdebug drop' , nothing in Tracker or Smartlog. It's the same after I made an IPS exception for destination 10.94.7.211.

    Code:
    [Expert@VSX-01:5]# fw ctl zdebug drop ^ grep 10.94.7.211
    ^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:52109 -^ 10.94.7.211:443 dropped by fw_filter_chain Reason: chain hold failed^
    ^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:42223 -^ 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed^
    ^[vs_5]^[tid_0]^[fw4_0]^fw_log_drop_ex: Packet proto=6 10.29.121.88:42224 -^ 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed^
    I'm not using domain objects

  2. #2
    Join Date
    2014-09-23
    Location
    Austin, TX
    Posts
    136
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I would double check the rulebase again and make doubly sure there arent any domain objects being used and if there are any then to remove them

  3. #3
    Join Date
    2007-06-27
    Posts
    22
    Rep Power
    0

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Hi

    the hold_table is being filled up, most likey you use a rule containing domain objects on the beginning of the rulebase.
    almost each packet that enters the rulebase and reaches this specific rule, needs to be resolved to check if it
    matches the domain object and as a result the connection is listed on the hold_table.
    if there is high traffic, the hold_table might be full and lead to traffic drop.

    Move to the domain object rule towards the end of the rulebase and this will decrease the amount of connections being listed under the hold_table.

    Regards,
    Guy

  4. #4
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I'm not using domain objects

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Quote Originally Posted by Irek_Romaniuk View Post
    I'm not using domain objects
    I seem to remember that there are some limited scenarios where this can be caused by Identity Awareness, specifically if there are invalid or unreachable domain servers configured. Assuming you are using IA, what does the "adlog a dc" command show when run on the firewall?
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  6. #6
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I'm not using domain objects, but definitely hold_table size is growing when I generate traffic from specific source..Questions is why

    Code:
    [Expert@VSX-01:5]# fw tab -t hold_table -s
    HOST                  NAME                               ID #VALS #PEAK #SLINKS
    localhost             hold_table                       8183     9   200       0
    [Expert@VSX-01:5]# fw tab -t hold_table -s
    HOST                  NAME                               ID #VALS #PEAK #SLINKS
    localhost             hold_table                       8183    22   200       0
    Also when I look inside hold_table and translate hex to ip , I don't see that specific source

  7. #7
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Quote Originally Posted by Irek_Romaniuk View Post
    I'm not using domain objects, but definitely hold_table size is growing when I generate traffic from specific source..Questions is why

    Code:
    [Expert@VSX-01:5]# fw tab -t hold_table -s
    HOST                  NAME                               ID #VALS #PEAK #SLINKS
    localhost             hold_table                       8183     9   200       0
    [Expert@VSX-01:5]# fw tab -t hold_table -s
    HOST                  NAME                               ID #VALS #PEAK #SLINKS
    localhost             hold_table                       8183    22   200       0
    Also when I look inside hold_table and translate hex to ip , I don't see that specific source
    The hold_table is used to keep track of packets that the firewall kernel has sent to a daemon process for handling. The classic case is of course DNS lookups for domain objects but it is used for many other things. I would have a look at what is going on in user/process space on the firewall:

    cpwd_admin list (any processes crashing and restarting?)
    top (any daemon processes eating huge amounts of CPU off in the weeds somewhere?)
    ls -ltr $FWDIR/log (any daemon log files getting constantly written to with errors/warnings?)
    enabled_blades (any features enabled that rely on daemons for processing?)

    Also see my posting above about IA possibly being involved...
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  8. #8
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I missed that IA comment, thnx.

    [Expert@VSX-01:5]# enabled_blades
    fw ips identityServer

    But 'adlog a dc' in 'Connection state' gives solid 'has connection'

    'cpwd_admin list' showing [21:25:48] 15/4/2015 for all processes which is time/day when I applied R77_20_jumbo_hf, 'top' is not bad since that time;) Will look again

    But 'stickness' to specific source IP addresses bothers me the most

    See below , I spoofed source IP (1) and it was dropped. And I spoofed another IP (2) and got response from server, magic ? Very repeatable.

    1) #sudo nmap -e eth0 -Pn -S 10.29.121.88 10.94.7.211 -p 80

    Code:
    [Expert@VSX-01:5]# fw ctl zdebug drop | grep 10.29.121.88
    ;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.29.121.88:50352 -> 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed;
    ;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.29.121.88:50353 -> 10.94.7.211:80 dropped by fw_filter_chain Reason: chain hold failed;
    2) # sudo nmap -e eth0 -Pn -S 10.29.121.244 10.94.7.211 -p 80

    Code:
    [Expert@VSX-01:5]# tcpdump -ni Lan5.373 host 10.29.121.244
    07:41:27.889310 IP 10.29.121.244.59291 > 10.94.7.211.http: S 3853270289:3853270289(0) win 3072 <mss 1460>
    07:41:27.889411 IP 10.94.7.211.http > 10.29.121.244.59291: S 3866447525:3866447525(0) ack 3853270290 win 8192 <mss 1460>
    When looking at 'fw monitor' I can see only 'i' in the first case...

    And $FWDIR/log/ is showing IA logs, but nothing interesting there. I am surprised these are present without running debug thou

    Code:
    [Expert@VSX-01:5]# ls -ltr $FWDIR/log/
    -rw-rw-r-- 1 admin root    3002209 Apr 19 16:48 pdpd.elg
    -rw-rw-r-- 1 admin root    5692894 Apr 19 16:58 pepd.elg

  9. #9
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Hm.. there is predictability in this madness. The rule allowing traffic for the subnet 10.29.121.0/24 is IP, not IA based.
    I can pick any address from this allowed subnet not being identified by IA and it will have the same problem !
    'fw monitor' will stops after 'i'. And any address from allowed subnet which is recognized by IA is fine, wow..does it make any sense ?

    So simple rule:

    Code:
    If AD Query finds the owner of IP_Address 
    [Expert@VSX-01:5]# adlog a query ip IP_Address
    OUTPUT
    If OUTPUT not empty then IP_Address is passing that IP, not IA based rule;)

  10. #10
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I'd say it is almost certainly IA causing the drops based on the output of enabled_blades, although based on the non-IA rules getting caught in this you could quickly try disabling IPS with an "ips off" command, initiate some new traffic to see if the drops are still occurring, then re-enable IPS with an "ips on" command just to rule IPS out.

    However IA shouldn't need to hold anything up like that unless it is having to shovel a captive portal at a user and waiting for a response or possibly attempting communication with the optional Identity Awareness Agent on a user's system or the Terminal Server Agent. What IA checkboxes do you have enabled on your gateway (i.e. Browser-Based Authentication, AD Query, Identity Agents, Terminal Servers, Remote Access)?
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  11. #11
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Thnx, only AD Query enabled. I put an network exception in IPS at the very begining and it didn't help

  12. #12
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Looks like moving IP based rules UP before IA rules fixes the issue... I can see 'iIoO' in 'fw monitor' now after I moved IP rule before not directly related IA rule

  13. #13
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Quote Originally Posted by Irek_Romaniuk View Post
    Looks like moving IP based rules UP before IA rules fixes the issue... I can see 'iIoO' in 'fw monitor' now after I moved IP rule before not directly related IA rule
    Thanks for the followup, interesting. Are you running R75.47 or R77.20? There were quite a few IA fixes included in those particular releases.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  14. #14
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    I have 77.20 jumbo patched week ago;)

  15. #15
    Join Date
    2014-10-10
    Posts
    250
    Rep Power
    6

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    plus I was told by CP that root cause was that I was obtaining IA on my VS here from 3 other VSs (10.93.17.1, 10.83.21.1, 10.73.41.20)

    [Expert@VSX-01:5]# pep show network pdp | grep 10.29.121.
    | 10.29.121.0 | 255.255.255.0 | <10.93.17.1,0>;<10.83.21.1,0>;<10.73.41.20,0>; |

    I don't believe until I test it

  16. #16
    Join Date
    2006-02-27
    Posts
    93
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    Quote Originally Posted by Irek_Romaniuk View Post
    plus I was told by CP that root cause was that I was obtaining IA on my VS here from 3 other VSs (10.93.17.1, 10.83.21.1, 10.73.41.20)

    [Expert@VSX-01:5]# pep show network pdp | grep 10.29.121.
    | 10.29.121.0 | 255.255.255.0 | <10.93.17.1,0>;<10.83.21.1,0>;<10.73.41.20,0>; |

    I don't believe until I test it
    Hi,

    Did you manage to resolve this issue ? I have the same setup (IA on VSX with identity sharing between gateways)

    Also I noticed that it makes the hit counts wrong in smartdashboard : my IA rule is just before the drop/any/any but 99% of the dropped traffic is accounted against that rule ...

  17. #17
    Join Date
    2006-02-27
    Posts
    93
    Rep Power
    14

    Default Re: dropped by fw_filter_chain Reason: chain hold failed

    any news on this issue ? I think i'm going to stop using IA because of that problem.

    last option would be to switch to the new Checkpoint Identity Collector agent ?

    regards

Similar Threads

  1. SIP UDP packets dropped with strange reason
    By TodorPetkov in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 5
    Last Post: 2015-12-03, 11:04
  2. dropped by vpn_encrypt_chain Reason: no reason
    By crosspopz in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 3
    Last Post: 2013-12-03, 11:07
  3. Installation Failed. Reason: Failed to Load Policy on Module. (Message from member)
    By rdbalaji in forum Security Management Server (Formerly SmartCenter Server ((Formerly Management Server))
    Replies: 2
    Last Post: 2012-11-15, 04:20
  4. Replies: 6
    Last Post: 2012-10-05, 13:47
  5. dropped by vpn_inbound_policy_chain Reason: vpn inbound nat after vm failed
    By zyz101z in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 7
    Last Post: 2007-01-25, 05:48

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •