Logexport randomises fields?

[rtfmoz]

Hi,

We do a fwm logexport of binary log files nightly on a SPLAT box. These files are transferred from a P-1 NGX R62 server. What we are finding is nearly every daily log export has different field positions. This is making processing the logs for analysis extremely difficult. Can anyone else please verify they are seeing the same behaviour?

It seems the log header identifies the fields correctly. Does anyone have an awk script that will read the log header field then use info to pull the right information from the logfile? I tried to write one but its a little beyond me.

Log header from 1st of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;service_id;src;dst;proto;rule;message_info;service;s_port;rule_uid;rule_name;ICMP;ICMP Type;ICMP Code;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs;H.323 message;src phone number;dst phone number

Log header from 2nd of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;rule;rule_uid;rule_name;service_id;src;dst;proto;service;s_port;ICMP;ICMP Type;ICMP Code;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;message_info;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs

Log header from 26th of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;rule;rule_uid;rule_name;service_id;ICMP;src;dst;proto;ICMP Type;ICMP Code;service;s_port;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;message_info;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs;H.323 message;src phone number;dst phone number

Regards

Moz

[dys152]

We had this problem, not in work at the moment so cant give exact details but there is an article on Secureknowledge covering this. You have to create a file that determines the order of the fields then specify this as an option to logexport (this is from memory by the way!)

Jon

[tnkflx]

The file you have to create is "$FWDIR/conf/logexport.ini" with the following content:

[Fields_Info]
included_fields = num,date,time,orig,type,action,alert,i/f_name,i/f_dir,product,log_sys_message,rule,rule_uid,rule_n ame,service_id,src,dst,service,<REST_OF_FIELDS>

If you need those columns to be different, check the heading of a logfile and put the columns you want in the correct order after "included_fields ="

[edhacker]

I too have had to deal with FW1/VPN1 writing out columns in changing order. Here is a perl script I use to grab the columns I'm interested in .. based on the header line in the log file. You could change the output order of the columns if needed.
Hope this helps.
AP

#!/usr/bin/perl
#
# Use the first line (header) to determine the column layout for this
# specific log.

# cat log file into this perl script
while(<STDIN>) {
 my $line = $_; # remove all whitespace from end of line, including dos cr
 $line =~ s/\s+$//;
 my @column = split (";", $line);
 if ($column[0] =~ /^num/) {
  # find index position of various keywords - columns youre looking for
  for (my $idx=1; $idx<$#column; $idx++) {
   if ($column[$idx] =~ /^Date$/) { $cdate = $idx; }
   if ($column[$idx] =~ /^Time$/) { $ctime = $idx; }
   if ($column[$idx] =~ /^Action$/) { $caction = $idx; }
   if ($column[$idx] =~ /^Origin$/) { $corigin = $idx; }
   if ($column[$idx] =~ /^Service$/) { $cservice = $idx; }
   if ($column[$idx] =~ /^Source$/) { $csource = $idx; }
   if ($column[$idx] =~ /^Source Port$/) { $csport = $idx; }
   if ($column[$idx] =~ /^Rule$/) { $crule = $idx; }
   if ($column[$idx] =~ /^User$/) { $cuser = $idx; }
   if ($column[$idx] =~ /^Protocol$/) { $cprotocol = $idx; }
  }
 }
 print
 $column[$cdate], ",", 
 $column[$ctime], ",", 
 $column[$caction], ",", 
 $column[$corigin], ",", 
 $column[$cservice], ",", 
 $column[$csource], ",", 
 $column[$csport], ",", 
 $column[$cservice], ",", 
 $column[$crule], ",", 
 $column[$cuser], ",", 
 $column[$cprotocol], " "; 
}

Originally Posted by rtfmoz

Hi, We do a fwm logexport of binary log files nightly on a SPLAT box. These files are transferred from a P-1 NGX R62 server. What we are finding is nearly every daily log export has different field positions. This is making processing the logs for analysis extremely difficult. Can anyone else please verify they are seeing the same behaviour? It seems the log header identifies the fields correctly. Does anyone have an awk script that will read the log header field then use info to pull the right information from the logfile? I tried to write one but its a little beyond me. Log header from 1st of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;service_id;src;dst;proto;rule;message_info;service;s_port;rule_uid;rule_name;ICMP;ICMP Type;ICMP Code;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs;H.323 message;src phone number;dst phone number

Log header from 2nd of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;rule;rule_uid;rule_name;service_id;src;dst;proto;service;s_port;ICMP;ICMP Type;ICMP Code;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;message_info;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs

Log header from 26th of Month

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;log_sys_message;rule;rule_uid;rule_name;service_id;ICMP;src;dst;proto;ICMP Type;ICMP Code;service;s_port;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;xlatedport;xlatesport;message_info;rpc_prog;TCP packet out of state;tcp_flags;Attack Info;attack;sys_message:;Total logs;Suppressed logs;H.323 message;src phone number;dst phone number

Regards Moz

[marklar]

And in one line of awk:

awk '{q=split($0,a,";");if (NR==1){for (v=1;v<=q;v++) c[a[v]]=v} printf("%s;%s;%s;%s;%s;%s;%s;%s;%s;%s;%s;%s;%s\n", a[c["date"]],a[c["time"]],a[c["orig"]],a[c["action"]],a[c["i/f_name"]],a[c["src"]],a[c["dst"]],a[c["proto"]],a[c["service"]],a[c["rule"]],a[c["xlatesrc"]],a[c["xlatedst"]],a[c["xlatedport"]])}'

yes, I know I'm sick :)

m.