CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Page 1 of 2 12 LastLast
Results 1 to 20 of 27

Thread: URL filtering, is this a joke?

  1. #1
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default URL filtering, is this a joke?

    Hey,

    Say I simply want to allow access to example.com and all of its subdomains. From what I read in sk106623 this is pure horror:

    1) You have to enable regex filtering for a task trivial as this.
    2) It makes no sense why \.example\.com would include subdomains. You'd expect that would need to be ".*\.example\.com".
    3) In fact I tried and it does include all subdomains but not the base domain, which makes sense...
    4) ...and which also makes sk106623 wrong.

    Just like on a ProxySG for example, I'd want to put "example.com" there and be done.

    What a pain...
    Last edited by jeronimo; 2018-02-16 at 12:08.

  2. #2
    Join Date
    2012-08-16
    Posts
    182
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Most of the documentation surrounding their regex and what works and what doesn't work is frustrating. Two articles will say two different things. Ultimately it is usually a combo of a regex and standard *.cnn.com cnn.com to cover all the bases

  3. #3
    Join Date
    2011-08-02
    Location
    http://spikefishsolutions.com
    Posts
    1,648
    Rep Power
    9

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by jeronimo View Post
    Hey,

    Say I simply want to allow access to example.com and all of its subdomains. From what I read in sk106623 this is pure horror:

    1) You have to enable regex filtering for a task trivial as this.
    2) It makes no sense why \.example\.com would include subdomains. You'd expect that would need to be ".*\.example\.com".
    3) In fact I tried and it does include all subdomains but not the base domain, which makes sense...
    4) ...and which also makes sk106623 wrong.

    Just like on a ProxySG for example, I'd want to put "example.com" there and be done.

    What a pain...
    Not to down play your pain or anything. Just pointing out that from a regex point of view

    \.example\.com

    and

    .*\.example\.com

    should in theory match the same thing from a regex point of view.

    .* means zero or more of any character. (optional anything)

  4. #4
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by jflemingeds View Post
    Not to down play your pain or anything. Just pointing out that from a regex point of view

    \.example\.com

    and

    .*\.example\.com

    should in theory match the same thing from a regex point of view.

    .* means zero or more of any character. (optional anything)
    Yeah but if you want the base domain to match, you don't want ".example.com" to match but only "example.com".

  5. #5
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by jeronimo View Post
    Yeah but if you want the base domain to match, you don't want ".example.com" to match but only "example.com".
    Forget my remark, you are right of course. I didn't immediately get the point you were making.

    It remains a pain anyway. To solve this you'll have to go to regex mode and include two URLs:
    1) \.example\.com
    2) example\.com

    In sk106623 they are indeed also pointing out that someone tried combining this into (.*\.|)example\.com* which made the thing crash. Great.

  6. #6
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by aweldon View Post
    standard *.cnn.com cnn.com
    What do you call "standard"? How would you define *.example.com without a regex?

  7. #7
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Oh and BTW once you're in with regexes, who says that "\.example\.com" wouldn't match "x.example.com.bla"?

    To be on the safe side you'd have to make it
    ^(.*\.|)example\.com$

  8. #8
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by jeronimo View Post
    What do you call "standard"? How would you define *.example.com without a regex?
    I see you mean this: https://sc1.checkpoint.com/documents...17.htm#o101841

    The meaning of the asterisk ( * ) depends on its use.
    In regular expressions, the asterisk is a metacharacter for zero or more instances of the preceding character.
    Without regular expressions, the asterisk is a wildcard, for zero or more instances of any character.

    They also suggest:
    More examples of regular expressions:
    To match subdomains of mydomain.com: (^|.*\.)mydomain\.com
    To match domain and subdomains of mydomain.com: (^|.*\.)*mydomain\.com

    Well usually that would also match mydomain.com.bla, not sure if they are putting implicit anchors around this...

  9. #9
    Join Date
    2012-08-16
    Posts
    182
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    So, what I usually do, is we have a bypass custom category defined as regex - meaning box checked and a secondary custom with the box not checked (non regular expression) defined in a bypass rule. Again this is by category. The regex category uses variations of regex - usually \.example\.com then the secondary category has example.com, *.example.com defined. This acts as a "please somehow catch this URL"

  10. #10
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    303
    Rep Power
    13

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by jeronimo View Post
    Oh and BTW once you're in with regexes, who says that "\.example\.com" wouldn't match "x.example.com.bla"?

    To be on the safe side you'd have to make it
    ^(.*\.|)example\.com$
    Oh, it's worse than that.

    I tried matching ".ar" once. It matched Argentinian websites, yes, but it also caught
    Code:
    www.some.site.here/and/now/a/path/images/32x32.arrow.png
    Also keep in mind the regular expression engine is case-sensitive, while domain names are case-insensitive but case-preserving.
    Zimmie

  11. #11
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    If anyone asked me how I would want it to be, I'd say: Do it like ProxySGs:

    If the host specified is a domain name, all hosts in that domain (or any subdomain) will match. If a path is specified, all paths with that prefix will match. If a scheme or port number is specified, only URLs with that scheme or port will match. (Additionally you can switch to regex mode or advanced matching.)

  12. #12
    Join Date
    2005-08-14
    Location
    Gig Harbor, WA, USA
    Posts
    2,494
    Rep Power
    16

    Default Re: URL filtering, is this a joke?

    We have updated the contents of sk106623 based on the feedback in this thread.
    Please review it and let me know if there are further problems.
    http://phoneboy.org
    Unless otherwise noted, views expressed are my own

  13. #13
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by PhoneBoy View Post
    We have updated the contents of sk106623 based on the feedback in this thread.
    Please review it and let me know if there are further problems.
    I've taken a look at the update and it still seems strange. They write you should use:
    ^example.com, and
    .example.com

    I'm not sure what the caret does there, because usually that's part of a regex. Albeit if this were a regex, the periods probably needed to be escaped.

    I tried again, and it still seems that what you need to include in non-regex mode is:
    example.com
    *.example.com
    if you want to allow both anything.example.com as well as example.com.

    I'm not sure where they came up with what they wrote.

  14. #14
    Join Date
    2012-08-16
    Posts
    182
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    The caret just represents the beginning of the line.

    https://regexr.com/3lgj0

    So yes you will need multiple entries for a single site. Regex and non-regex.

  15. #15
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by aweldon View Post
    The caret just represents the beginning of the line.

    https://regexr.com/3lgj0

    So yes you will need multiple entries for a single site. Regex and non-regex.
    You didn't read all I wrote. The caret means a caret when used in non-regex mode, nothing more and nothing less.

    Non-regex mode: example.com
    Regex mode: ^example\.com

    What they wrote in the sk article makes no sense, and it doesn't work (I tried).

  16. #16
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by aweldon View Post
    The caret just represents the beginning of the line.

    https://regexr.com/3lgj0

    So yes you will need multiple entries for a single site. Regex and non-regex.
    Your example also isn't entirely correct BTW; in the example you provided you didn't use multi-line mode so it wouldn't have matched anything beyond the first line anyway

    See here: https://regexr.com/3lgnm (which shows that you would need to use \. instead of .)

  17. #17
    Join Date
    2012-08-16
    Posts
    182
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Why would you need multi line mode for a single URL?

  18. #18
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by aweldon View Post
    Why would you need multi line mode for a single URL?
    The mode applies to the text you apply it to. You had five possible matches (in five lines), and even though your regex only matched the first, you would never have seen if one of the others had matched. If you want to test your regex against multiple lines of possible matches, you need to use multi-line mode. Your usage of the period (wildcard) was also incorrect as you can see in my example.

  19. #19
    Join Date
    2012-08-16
    Posts
    182
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Yeah, I see what you mean with multi-line and carets. Don't think I'll be using them anyway - stick to what's working.
    Last edited by aweldon; 2018-02-28 at 23:05.

  20. #20
    Join Date
    2012-08-06
    Posts
    62
    Rep Power
    7

    Default Re: URL filtering, is this a joke?

    Quote Originally Posted by aweldon View Post
    Yeah, I see what you mean with multi-line and carets. Don't think I'll be using them anyway - stick to what's working.
    Yeah, definitely what's in the sk article won't work.

Page 1 of 2 12 LastLast

Similar Threads

  1. URL Filtering
    By rotherdrummer in forum Miscellaneous
    Replies: 5
    Last Post: 2014-09-18, 05:10
  2. R75 URL Filtering Reports
    By mmazz in forum Eventia Analyzer/Reporter/SmartView Reporter
    Replies: 2
    Last Post: 2012-04-20, 13:45
  3. URI filtering on HTTPS
    By ppetrovic in forum Content Security/Security Servers/CVP/UFP
    Replies: 1
    Last Post: 2010-10-09, 18:47
  4. Web filtering - Centralized
    By sroghen in forum Check Point UTM-1 Appliances
    Replies: 8
    Last Post: 2009-06-18, 14:38
  5. Filtering URL's
    By imwings in forum Content Security/Security Servers/CVP/UFP
    Replies: 2
    Last Post: 2008-04-22, 20:16

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •