CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 6 of 6

Thread: fwm export - File size limit exceeded

  1. #1
    Join Date
    2007-10-31
    Location
    Great Plains - USA
    Posts
    158
    Rep Power
    12

    Default fwm export - File size limit exceeded

    SM225 running R77.30

    Our logs rollover at midnight every day. Typical log file contains 6-7 million records and a size of approx 1.3 GB. Having a need to export multiple days of logs into a .csv format, I looked to using "fwm export". Tracker works fine for this but takes a long time to complete. I was hoping fwm export would be speedier and it seems to be. But there is this:

    [Expert@]# fwm logexport -d , -i 2018-10-08_235900.log -o Oct08.csv -n -p -m raw
    Starting... There are 7053861 log records in the file
    File size limit exceeded (core dumped)ds (48%)

    Couple questions I need help with:
    - Why do I have an apparent file size limit of 2 GB? (-rw-rw---- 1 admin users 2147483647 Oct 9 16:21 Oct08.csv)
    - Is it expected that my original 1.3 GB file (-rw-rw---- 1 admin root 1399574291 Oct 8 23:59 2018-10-08_235900.log) expands to over 2 GB when converted to a .csv?

    Thanks for any insights on this.

  2. #2
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    265
    Rep Power
    12

    Default Re: fwm export - File size limit exceeded

    I'll answer your second question first. Check Point's native log format records some fields like IP address in binary rather than text. An IP address in binary is four bytes. The same address as text would be up to 15 bytes in ASCII or potentially even more in various Unicode encodings. Yes, it is expected for logs exported as plain text to be significantly larger than the logs in the original binary format.

    As for the file size limitation, what OS are you running on the box (check with 'uname -a')? What is the output of the command 'mount'? File size limitations most often come from the filesystem used on the drive. It's possible the fwm process doesn't like dealing with files larger than 2 GB, so you may have to use ordinary output redirection.

    Peripherally related, I recommend against delimiting the fields with a comma. Certain fields can contain a comma, which makes automated processing more difficult. I generally export logs like this:

    Code:
    fwm logexport -s -z -n -p -i ./<file>.log > <file>.ffsv
    -s sets the field delimiter to be ASCII character 0xff, which can never occur inside a log field.
    Zimmie

  3. #3
    Join Date
    2007-10-31
    Location
    Great Plains - USA
    Posts
    158
    Rep Power
    12

    Default Re: fwm export - File size limit exceeded

    Quote Originally Posted by Bob_Zimmerman View Post
    I'll answer your second question first. Check Point's native log format records some fields like IP address in binary rather than text. An IP address in binary is four bytes. The same address as text would be up to 15 bytes in ASCII or potentially even more in various Unicode encodings. Yes, it is expected for logs exported as plain text to be significantly larger than the logs in the original binary format.

    As for the file size limitation, what OS are you running on the box (check with 'uname -a')? What is the output of the command 'mount'? File size limitations most often come from the filesystem used on the drive. It's possible the fwm process doesn't like dealing with files larger than 2 GB, so you may have to use ordinary output redirection.

    Peripherally related, I recommend against delimiting the fields with a comma. Certain fields can contain a comma, which makes automated processing more difficult. I generally export logs like this:

    Code:
    fwm logexport -s -z -n -p -i ./<file>.log > <file>.ffsv
    -s sets the field delimiter to be ASCII character 0xff, which can never occur inside a log field.
    Thank you very much!

    I was using CP_R77_CLI_ReferenceGuide for command structure, they do not note the -z or -s switches, this is nice to know. I will be trying your string shortly and reporting back. My requested outputs:

    [Expert@:0]# uname -a
    Linux fw-mgmt 2.6.18-92cpx86_64 #1 SMP Wed Nov 8 17:55:29 IST 2017 x86_64 x86_64 x86_64 GNU/Linux

    [Expert@:0]# mount
    /dev/mapper/vg_splat-lv_current on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)
    /dev/mapper/vg_splat-lv_log on /var/log type ext3 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

  4. #4
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    265
    Rep Power
    12

    Default Re: fwm export - File size limit exceeded

    x86_64 in the uname output indicates it's running in 64-bit mode, and vg_splat-lv_current indicates GAiA. Files bigger than 2 GB aren't an issue on ext3. This must be a limitation of the fwm binary file output functionality. Output redirection with > should work.

    The -z switch just sets it to ignore non-fatal errors and keep exporting log data. This can cause corrupt entries in the output, but those can be handled after the fact.
    Zimmie

  5. #5
    Join Date
    2007-10-31
    Location
    Great Plains - USA
    Posts
    158
    Rep Power
    12

    Default Re: fwm export - File size limit exceeded

    Quote Originally Posted by Bob_Zimmerman View Post
    x86_64 in the uname output indicates it's running in 64-bit mode, and vg_splat-lv_current indicates GAiA. Files bigger than 2 GB aren't an issue on ext3. This must be a limitation of the fwm binary file output functionality. Output redirection with > should work.

    The -z switch just sets it to ignore non-fatal errors and keep exporting log data. This can cause corrupt entries in the output, but those can be handled after the fact.
    Well, "other duties as assigned" prevented my testing yesterday. I did so today and have encouraging results. Thank you for the great insights Zimmie.

    I now have an Excel error stating not enough memory and to consider using 64-bit version of Excel, but that is a different story I will leave to our helpdesk.......

    Kind regards,
    dbrown

  6. #6
    Join Date
    2007-03-30
    Location
    DFW, TX
    Posts
    265
    Rep Power
    12

    Default Re: fwm export - File size limit exceeded

    Rather than importing into Excel, it may be worth processing the file with PowerShell first. For example, I use this to trim a file down to just the columns I care about, then process down to unique lines:

    Code:
    Import-Csv -Path .\someFile.csv | Select Source,Destination,"Destination Port",Rule,Action | Export-Csv .\filtered.csv
    Import-Csv -Path .\filtered.csv | Sort-Object -Property Source,Destination,"Destination Port",Rule,Action -Unique | Export-Csv .\uniq.csv
    You should be able to chain them together without a round-trip through filtered.csv. I use intermediate files to be sure I only need to rerun a few shorter steps if something goes wrong.

    Import-Csv has a -Delimiter option, but I haven't tried it with 0xff-separated files before. It may understand quote-delimited, comma-separated files, so you may not need to use the -s switch in the export. I have only started using PowerShell for this kind of processing relatively recently. Before, I used awk or Perl.
    Zimmie

Similar Threads

  1. Limit file upload size checkpoint
    By Flexible in forum Application Control Blade
    Replies: 1
    Last Post: 2013-10-17, 10:17
  2. Exporting log files to text and 2GB file limit
    By nolan.rumble in forum Scripts and Tools
    Replies: 0
    Last Post: 2012-05-10, 09:42
  3. Log file size
    By him007 in forum SmartView Tracker
    Replies: 10
    Last Post: 2008-03-28, 12:47
  4. Replies: 7
    Last Post: 2007-04-17, 11:31
  5. ftp file size
    By Morph in forum Check Point SecurePlatform (SPLAT)
    Replies: 8
    Last Post: 2006-06-25, 03:11

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •