Extended ASCII log processing tool fw1logsearch.pl

[edhacker]

I have had several friends ask me about my Checkpoint log processing scripts and recently found myself having to sift through nearly a billion log lines, looking for specific patterns of access. The problem was that many of the exported fwm log files had differing column positions and there had been many ruleset changes over the course of 11 months worth of log data. Many of the excellent FW1 log summarization tools (such as Peter Sundstrom’s fwlogsum) didn’t handle the hundreds of files and differing column positions.

I wrote (or rather heavily hacked together) a perl script to read the header line and process the log file based on the column alignment shown in the header vs a predetermined column sequence. The fw1logsearch.pl script can be used to find complex patterns of interest. Any matching records found by fw1logsearch will be output with an initial FW1 header line so that fw1logsearch can be used iteratively, to build very complex search criteria (additive includes and excludes on (nearly) any column). The script can also write out a discard file allowing completely negative logic searches resulting in 100% of the input data separated into match and reject files.

This saved me countless hours, and while there are undoubtedly better scripts out there that I just didn't see or find, I hope this will help someone on this list. I've posted the script on my website at Sifting through Checkpoint FW1 logs Allen Pomeroy

Cheers,
AP

[fwlogsum]

Please note that fwlogsum can handle as many merged fwlogexport files as your system resources can handle (mostly memory) as it uses the header description line that is the first line of each log export.

This means it doesn't matter if the order of the columns changes between the logexports, as fwlogsum simply reads in each new header.

Cheers

Peter Sundstrom
fwlogsum