Firewalls Not Logging to Remote Server

[lnx32]

I am running Checkpoint R62 on 2 Nokia IP390s and Smartcenter on a Windows server. My Appliances started logging locally a few weeks ago and stopped sending the log files to the remote Smartcenter Server. When I noticed the current logs weren't showing up in Tracker I checked the appliances and found that when I did a df -k the /var directory was over 108% on both appliances. I checked the $FWDIR/logs folder and all the logs were being logged here.
What can I do to get the files to stop logging locally and start logging back to the SmartCenter server? I've tried clearing the $FWDIR/logs folder and rebooting both the appliances and the SmartCenter server but they are still logging to the appliances causing the /var directory to to fill up every 3 days or so. What can I do to correct this and get them back on track?

[belvdr]

Is SIC working? Test that first.

Then under Logs and Masters -> Log Servers for each node, uncheck the option "Save logs locally, on this machine". Note, if this is a cluster, the local option is not there.

Also ensure the SCS is under the "Always send logs to" box. If it is not, add your SCS in there and check both Logs and Alerts. Push policy.

[lnx32]

Yes, SIC is working to both appliances from the SCS. I checked the logging to SCS and these settings are correct! I can figure it out. I was toying with the Idea of just resetting the state tables on both firewalls but that would impact my production traffic.

[belvdr]

So you can push policy, but can you fetch it? Go to one of the boxes and do:

fw fetch <ip of SCS>

It really sounds as though SIC is initialized only one way, so the SCS can send to the gateway but not vice versa. By running the above command, it will verify it for sure, and it outputs errors too, if there are any.

[lnx32]

This is what the output looks like on both Appliances........

cp1[admin]# fw fetch 172.29.5.240

Fetching Security Policy From: 172.29.5.240

Local Policy is Up-To-Date.
Reinstalling Local Policy.

Installing Security Policy Standard_1a on all.all@cp1
Oct 30 12:05:53 cp1 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:05:53 cp1 [LOG_CRIT] kernel: fwdynlog_commit: failed translating alert param. i=
121, alert = 1 - setting alert = "log"
Oct 30 12:05:53 cp1 [LOG_CRIT] kernel: fwdynlog_commit: failed translating alert param. i=
122, alert = 1 - setting alert = "log"
Oct 30 12:05:53 cp1 [LOG_CRIT] kernel: FW-1: lost 9459 log/trap messages
Successfully compiled file types magic file.

Oct 30 12:06:04 cp1 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:06:04 cp1 [LOG_CRIT] kernel: FW-1: lost 21 log/trap messages
Oct 30 12:06:04 cp1 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:06:04 cp1 [LOG_CRIT] kernel: FW-1: lost 16 log/trap messages
Oct 30 12:06:04 cp1 [LOG_CRIT] kernel: FW-1: Log buffer is full
Fetching Security Policy Succeeded

cp1[admin]# Oct 30 12:06:21 cp1 [LOG_CRIT] kernel: FW-1: lost 2075 log/trap me
ssages

cp1[admin]#
__________________________________________________ _______________
cp2[admin]# fw fetch 172.29.5.240

Fetching Security Policy From: 172.29.5.240

Local Policy is Up-To-Date.
Reinstalling Local Policy.

Installing Security Policy Standard_1a on all.all@cp2
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: lost 8926 log/trap messages
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: fwdynlog_commit: failed translating alert param. i=
121, alert = 1 - setting alert = "log"
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: fwdynlog_commit: failed translating alert param. i=
122, alert = 1 - setting alert = "log"
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: lost 12 log/trap messages
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:12:50 cp2 [LOG_CRIT] kernel: FW-1: lost 64 log/trap messages
Successfully compiled file types magic file.
Oct 30 12:13:01 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: lost 52 log/trap messages
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: lost 5 log/trap messages
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: lost 20 log/trap messages
Oct 30 12:13:02 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: lost 60 log/trap messages
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: lost 4 log/trap messages
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: lost 26 log/trap messages
Oct 30 12:13:03 cp2 [LOG_CRIT] kernel: FW-1: Log buffer is full

Fetching Security Policy Succeeded

cp2[admin]#

[msjouw]

try the following: on the Gateway Object in the dashboard set the Master and logserver to use local defenition in the Masters file.
On the gateway itself edit $FWDIR/conf/masters and make sure it looks like this (it is from memory so pardon me missing the right words/syntax):
[Policy]
172.29.5.240
[Log]
172.29.5.240
[Alert]
172.29.5.240

And push the policy

[doron.linder]

Hi lnx32,

I'd suggest you fire up the Tracker on your log server (172.29.5.240) and filter the logs to check that you aren't getting any logs from the two Nokia machines. According to the "kernel: FW-1: Log buffer is full" message my guess is that there is too much traffic going on and although some of it reach the log server, the gateway isn't handling it fast enough and start to log locally so that no log will be lost. Have you tried running "fw debug fwd on TDERROR_ALL_ALL=5" and checking fwd.elg? It might support what I wrote, or at least give you the reason for the disconnection from the log server (if there is a disconnection causing the local logging).

Doron Linder
Logging Matters - Check Point Logging Related Services

[doron.linder]

And in addition to what I wrote below, first try and make some space available. Move the logs to the log server manually by using the Tracker's Tools -> Remote Files Management -> Fetch files option. It might be that the logging doesn't work since you're out of disk space.

Doron Linder
Logging Matters - Check Point Logging Related Services

[krugger]

Have a similar problem. My R70 ran out of disk space and after I cleaned some of the files the logging is still done locally at the firewalls and not at the SmartCenter server.

SIC is also ok, and have pushed the config again but still it is not resuming logging to the SmartCenter.

fw fetch does not give any errors. The SmartCenter server is set as the logging server. Is there a way to define the logging server through the command line? I am starting to think about a late night reboot of the firewall nodes to try and get the logging back to the SmartCenter server.

It is an active active cluster of R70 firewalls.

[boldin]

This happened with our R65 boxes from time to time and a reboot usually fixes the problem. I think on some occasions a cprestart would work, but not every time if my memory serves me correctly.

[krugger]

So this is some sort of bug that affects multiple FW-1 versions?

Surely there must be a solution I haven't found in the Knowledge Base.

[BillM]

I too am having this problem.

We upgraded our SCS from R61 to R70. It stopped receiving logs from the R61 gateways immediately. The next day, as planned, we upgraded the gateways to R65 and applied the latest HFA. R65 is as high as we could go on our appliance.

SIC is working. I can push or fetch. I have edited the $FWDIR/conf/masters with the IP address of my SCS. I set the object to use local.

Still nothing showing up in tracker.

Bill M

[ShadowPeak.com]

Originally Posted by BillM

I too am having this problem.

We upgraded our SCS from R61 to R70. It stopped receiving logs from the R61 gateways immediately. The next day, as planned, we upgraded the gateways to R65 and applied the latest HFA. R65 is as high as we could go on our appliance.

SIC is working. I can push or fetch. I have edited the $FWDIR/conf/masters with the IP address of my SCS. I set the object to use local.

Still nothing showing up in tracker.

Bill M

Seen this problem with R70 vanilla numerous times where the SmartCenter refuses to accept logs from older gateways but SIC works fine both ways. Try doing a Policy->Install Database. If that doesn't work you'll have to upgrade your SmartCenter Server to R70.1 or later which will fix the problem.

[mcnallym]

After performing an R65 to R70.20 migration on a P-1 CMA we have had an issue where the logging part of the CMA was not receiving the logs even though we could see the logs appearing on the NIC of the MDS.

Was that the CMA was showing the logging as started but did not stop and start correctly.

Had to do the kill -9 to stop the process, after which starting up again we got the logs.

Might be worth performing a cpstop on the Management Server and verifying that all of the services have stopped relating to check point.

If after a cpstop the check point services are still running then manually stop them, then restart the services afterwards.

[BillM]

This worked for me:

1.) Exit SmartDashboard

2.) Run cpstop

3.) Run the following commands:

rm $FWDIR/conf/asm.C*
rm $FWDIR/conf/profiles.C*
rm $FWDIR/conf/ips_tables_sqlite*
rm $FWDIR/conf/CPMILinksMgr.db*

4.) Replace the first three files you deleted with the ones I am attaching (zip file) <-- from our partners at Cadre.

5.) Run cpstart

6.) Log into SmartDashboard. If there are any error messages please take screen shots. If no errors, attempt to install database (Policy -> Install Database) on DRACO.

I am logging again.

Bill M

[regmartin]

For R70 environments, take a look at this post for a possible solution Logging not working -- need help :(