SmartLog via CLI

[Chili]

Hi all,

is there a way to get access to the SmartLog database via CLI?
I am looking for a special logs over an long time period (2 month). The logfile is switch nearly every hour.
So using "fw log" or SmartTracker is not so much fun.

When I use SmartLog, I can search a longer time period, but there are a lot of matches (5-10k) for my query and so
I have to view all matches before you can export all the logs into a seperate CSV-file.

Thanks for your support!

Regards
Chili

[jflemingeds]

Hi all,

is there a way to get access to the SmartLog database via CLI?
I am looking for a special logs over an long time period (2 month). The logfile is switch nearly every hour.
So using "fw log" or SmartTracker is not so much fun.

When I use SmartLog, I can search a longer time period, but there are a lot of matches (5-10k) for my query and so
I have to view all matches before you can export all the logs into a seperate CSV-file.

Thanks for your support!

Regards
Chili

I don't know of any cli tools. Smartlogd (the system process) is basically a webserver from what i've seen. Smartlog (the log verview) i think it just sending http requests with a speical cookie for auth. if you can figure out how to authinicate then i think you should be able to hack up some http calls in your fav scripting language.

2nd option: I think its using a google database on the back end but i'm not sure.

Let us know what you find if anything. I think this will be something everyone would like to know.

[PhoneBoy]

At the moment, there are no CLI tools for SmartLog.
This issue with exporting all matched records is familiar, let me check with my contacts in R&D.

[Christoph]

Necro an old thread.
Has there ever been a solution to this problem?

[jflemingeds]

Necro an old thread.
Has there ever been a solution to this problem?

I don't know why i thought it was some google database. Seems to be apache solr after looking at it again. I never did figure out how to hack a http session, but this was many moons ago.

Not really a very useful follow up on my part. :D

night!

[Christoph]

I don't know why i thought it was some google database. Seems to be apache solr after looking at it again. I never did figure out how to hack a http session, but this was many moons ago.

Not really a very useful follow up on my part. :D

night!

Thank you. I took a quick look:

# curl_cli -b "CPToken=XYZ123XYZ123XYZ123DUMMY" "http://localhost:18242/search_tops?query=blade:%22Threat%20Emulation%22&c ount=10&time_effort=10&sub_session_action=update&s ub_session=0"

This produces valid output.

EDIT: But you're right. I should have read your earlier comment better. I have no idea how long this Token is valid, so it may change.

[jflemingeds]

Thank you. I took a quick look:

# curl_cli -b "CPToken=XYZ123XYZ123XYZ123DUMMY" "http://localhost:18242/search_tops?query=blade:%22Threat%20Emulation%22&c ount=10&time_effort=10&sub_session_action=update&s ub_session=0"

This produces valid output.

EDIT: But you're right. I should have read your earlier comment better. I have no idea how long this Token is valid, so it may change.

Interesting, where did you find the cookie? I remember seeing urls flying around when i was debugging but i didn't break them down any further.

Is the data returned in json format?

[Christoph]

Interesting, where did you find the cookie? I remember seeing urls flying around when i was debugging but i didn't break them down any further.

The cookie was in the first http request when the initial or any subsequent request by smartlog. Not sure when the cookie was set and how persistent it is.

tcpdump -nnvXSs 0 -i any port 18242 -w x.pcap
Personally I found Microsoft Message Analyzer in this case more convenient than Wireshark to find the relevant part.

Is the data returned in json format?

Not 100% sure as I am home right now. From my memory I cannot tell if it was XML (iirc the packet capture had some XML labels) or JSON, but I just skimmed over the data as I just wanted to have a valid query to begin with.

Maybe I find some time over the weekend to check the persistence of the cookie, the data format returned and some more meaningful queries.

[jflemingeds]

The cookie was in the first http request when the initial or any subsequent request by smartlog. Not sure when the cookie was set and how persistent it is.

tcpdump -nnvXSs 0 -i any port 18242 -w x.pcap
Personally I found Microsoft Message Analyzer in this case more convenient than Wireshark to find the relevant part.


Not 100% sure as I am home right now. From my memory I cannot tell if it was XML (iirc the packet capture had some XML labels) or JSON, but I just skimmed over the data as I just wanted to have a valid query to begin with.

Maybe I find some time over the weekend to check the persistence of the cookie, the data format returned and some more meaningful queries.

aah -i any. I didn't think about that. I think i tried -i lo but again its been a while.

Silly question, have you looked through the R80 API docs? This might be covered in there in some manner.

[Chili]

Hi all,

regarding to my initial question there was some Kind of workaround in newer versions.
Check Point added a new field in the export form where you can define the number of logs to export.
The number is limited to a million logs. This should handle nearly all requests even if I would prefer "export all logs <from date> <to date>".

I already took a look at the R80 api reference guide.
It does not show any possibilities for gettings logs.
I would say right now it can only handle actions that are performed by SmartDashboard.

Cheers,
Sven

[Christoph]

Looked a bit further into it. You can gather all the relevant information in XML form from the smartlog database via i.e. curl.
Default offset is a start at line 1 from your request with an end of 50. If you scroll the offset moves (1-50, 50-100..). So a good way for the cli to gather data is by using a large offset in a defined time peroid.

The only problem left is the session cookie, which expires at the moment you close smart log.
The generation of the session cookie takes place when you start smart log with a:

QUERY:
POST /login HTTP/1.0
Content-Length: 283

<login>
<user><![CDATA[username]]></user>
<magic_number><![CDATA[CP_Etude_2055]]></magic_number>
<password><![CDATA[]]></password>
<sso_token><![CDATA[1234567890ABCDEF]]></sso_token>
<get_all_columns_def />
</login>

REPLY
HTTP/1.1 200 OK
Content-Type: text/plain
Cache-Control: no-cache
Connection: keep-alive

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<root><read_only>false</read_only>...<token>Z1234567890ABCDEFZ</token>...

I will try to use the </password> field. Without it one would have to have a read only smartlog active all the time use its credentials.

EDIT: Ok this works, remove the sso token, enter a cleartext password and you get you token. The token looks to stay valid for a reasonable time. No idea when it expires serverwise, but if you fetch a token for every query one should be fine.
This should be sufficient to automatically query smartlog.

[jflemingeds]

nice bit of hacking there.