URL filtering, is this a joke?

[jeronimo]

Hey,

Say I simply want to allow access to example.com and all of its subdomains. From what I read in sk106623 this is pure horror:

1) You have to enable regex filtering for a task trivial as this.
2) It makes no sense why \.example\.com would include subdomains. You'd expect that would need to be ".*\.example\.com".
3) In fact I tried and it does include all subdomains but not the base domain, which makes sense...
4) ...and which also makes sk106623 wrong.

Just like on a ProxySG for example, I'd want to put "example.com" there and be done.

What a pain...

[aweldon]

Most of the documentation surrounding their regex and what works and what doesn't work is frustrating. Two articles will say two different things. Ultimately it is usually a combo of a regex and standard *.cnn.com cnn.com to cover all the bases

[jflemingeds]

Hey,

Say I simply want to allow access to example.com and all of its subdomains. From what I read in sk106623 this is pure horror:

1) You have to enable regex filtering for a task trivial as this.
2) It makes no sense why \.example\.com would include subdomains. You'd expect that would need to be ".*\.example\.com".
3) In fact I tried and it does include all subdomains but not the base domain, which makes sense...
4) ...and which also makes sk106623 wrong.

Just like on a ProxySG for example, I'd want to put "example.com" there and be done.

What a pain...

Not to down play your pain or anything. Just pointing out that from a regex point of view

\.example\.com

and

.*\.example\.com

should in theory match the same thing from a regex point of view.

.* means zero or more of any character. (optional anything)

[jeronimo]

Not to down play your pain or anything. Just pointing out that from a regex point of view

\.example\.com

and

.*\.example\.com

should in theory match the same thing from a regex point of view.

.* means zero or more of any character. (optional anything)

Yeah but if you want the base domain to match, you don't want ".example.com" to match but only "example.com".

[jeronimo]

Yeah but if you want the base domain to match, you don't want ".example.com" to match but only "example.com".

Forget my remark, you are right of course. I didn't immediately get the point you were making.

It remains a pain anyway. To solve this you'll have to go to regex mode and include two URLs:
1) \.example\.com
2) example\.com

In sk106623 they are indeed also pointing out that someone tried combining this into (.*\.|)example\.com* which made the thing crash. Great.

[jeronimo]

standard *.cnn.com cnn.com

What do you call "standard"? How would you define *.example.com without a regex?

[jeronimo]

Oh and BTW once you're in with regexes, who says that "\.example\.com" wouldn't match "x.example.com.bla"?

To be on the safe side you'd have to make it
^(.*\.|)example\.com$

[jeronimo]

What do you call "standard"? How would you define *.example.com without a regex?

I see you mean this: https://sc1.checkpoint.com/documents...17.htm#o101841

The meaning of the asterisk ( * ) depends on its use.
In regular expressions, the asterisk is a metacharacter for zero or more instances of the preceding character.
Without regular expressions, the asterisk is a wildcard, for zero or more instances of any character.

They also suggest:

More examples of regular expressions:
To match subdomains of mydomain.com: (^|.*\.)mydomain\.com
To match domain and subdomains of mydomain.com: (^|.*\.)*mydomain\.com

Well usually that would also match mydomain.com.bla, not sure if they are putting implicit anchors around this...

[aweldon]

So, what I usually do, is we have a bypass custom category defined as regex - meaning box checked and a secondary custom with the box not checked (non regular expression) defined in a bypass rule. Again this is by category. The regex category uses variations of regex - usually \.example\.com then the secondary category has example.com, *.example.com defined. This acts as a "please somehow catch this URL"

[Bob_Zimmerman]

Oh and BTW once you're in with regexes, who says that "\.example\.com" wouldn't match "x.example.com.bla"?

To be on the safe side you'd have to make it
^(.*\.|)example\.com$

Oh, it's worse than that.

I tried matching ".ar" once. It matched Argentinian websites, yes, but it also caught

www.some.site.here/and/now/a/path/images/32x32.arrow.png

Also keep in mind the regular expression engine is case-sensitive, while domain names are case-insensitive but case-preserving.

[jeronimo]

If anyone asked me how I would want it to be, I'd say: Do it like ProxySGs:

If the host specified is a domain name, all hosts in that domain (or any subdomain) will match. If a path is specified, all paths with that prefix will match. If a scheme or port number is specified, only URLs with that scheme or port will match. (Additionally you can switch to regex mode or advanced matching.)

[PhoneBoy]

We have updated the contents of sk106623 based on the feedback in this thread.
Please review it and let me know if there are further problems.

[jeronimo]

We have updated the contents of sk106623 based on the feedback in this thread.
Please review it and let me know if there are further problems.

I've taken a look at the update and it still seems strange. They write you should use:
^example.com, and
.example.com

I'm not sure what the caret does there, because usually that's part of a regex. Albeit if this were a regex, the periods probably needed to be escaped.

I tried again, and it still seems that what you need to include in non-regex mode is:
example.com
*.example.com
if you want to allow both anything.example.com as well as example.com.

I'm not sure where they came up with what they wrote.

[aweldon]

The caret just represents the beginning of the line.

https://regexr.com/3lgj0

So yes you will need multiple entries for a single site. Regex and non-regex.

[jeronimo]

The caret just represents the beginning of the line.

https://regexr.com/3lgj0

So yes you will need multiple entries for a single site. Regex and non-regex.

You didn't read all I wrote. The caret means a caret when used in non-regex mode, nothing more and nothing less.

Non-regex mode: example.com
Regex mode: ^example\.com

What they wrote in the sk article makes no sense, and it doesn't work (I tried).

[jeronimo]

The caret just represents the beginning of the line.

https://regexr.com/3lgj0

So yes you will need multiple entries for a single site. Regex and non-regex.

Your example also isn't entirely correct BTW; in the example you provided you didn't use multi-line mode so it wouldn't have matched anything beyond the first line anyway

See here: https://regexr.com/3lgnm (which shows that you would need to use \. instead of .)

[aweldon]

Why would you need multi line mode for a single URL?

[jeronimo]

Why would you need multi line mode for a single URL?

The mode applies to the text you apply it to. You had five possible matches (in five lines), and even though your regex only matched the first, you would never have seen if one of the others had matched. If you want to test your regex against multiple lines of possible matches, you need to use multi-line mode. Your usage of the period (wildcard) was also incorrect as you can see in my example.

[aweldon]

Yeah, I see what you mean with multi-line and carets. Don't think I'll be using them anyway - stick to what's working.

[jeronimo]

Yeah, I see what you mean with multi-line and carets. Don't think I'll be using them anyway - stick to what's working.

Yeah, definitely what's in the sk article won't work.