Strange Error - Load on Module Failed - No Memory

[markw]

Here's a strange one for you all, which I'm still trying to troubleshoot without any success.

I have a pair of Nokia IP350's clustered with VRRP - one primary, one failover. Both running NGX R60. I also have a SPlat box which is my management console. Loads of interfaces (a few trunked interfaces too) and a 209 rule rulebase. Firewalls have been up for about 250 days since last reboot running pretty well IMHO.

This combo has been running without problem for 3 years. Policies apply fine, things have just worked. Never had an issue.

Until last week - where I now have an issue when I try to apply a rulebase.

I have changed one thing in the rulebase and applied it (disabled a rule). The verifying and installing seems to be much faster than usual - it doesn't take the same pattern it would do normally, and then the rulebase won't take - it comes up with;

'Load on Module Failed - No Memory'

on both firewalls.

So, I thought I'd try just rolling back, and re-applying the same rulebase that's on them with just a word in the comments changed.

No - same again.

In fact, whatever I try, I get the same issue. I've removed the HTTP settings from SmartDefense, There are no rule names in the rulebase at all, being British English, I don't have any accented characters in my comments. I've searched this forum for other similar issues and tried most of the existing suggestions without success :(

So I'm stuck - and I'm loathe to reboot the boxes in case they don't boot again as I've read of issues where the firewalls don't boot again for some reason with this issue.

Each of the 350's has 256Mb RAM in them and both are running with about 50Mb RAM free. Disk space isn't an issue with 97% free on both boxes.

What's strange is that nothing has changed on these boxes for them to start doing this - the only person who has access to the management day-to-day is me. I've not updated them or patched them or done anything apart from apply rulebases in the last few months.


Any suggestions?

All help gratefully recieved!

Mark.

[markw]

Further investigation reveals that, on the IP350's the following log entry occurs when trying to apply the rulebase;

May 18 08:45:54 (myfirewallname) [LOG_CRIT] kernel: FW1: fwloghandle_register_string: unable to put entry into table.

Apart from that the IP350's don't look any different to normal. Historic memory use is 200Mb of 256Mb (so nothing unusual.) There are no other errors in the error log apart from the usual LOG_INFO for cron jobs.

Is this just a case that I'm going to have to restart the firewalls and hope for the best?

[donshoutarp]

Hi,

I encountered something similar when I had some non-English characters in my Name field or Comment field. When I deleted those characters, the policy pushed fine.

[markw]

Originally Posted by donshoutarp

Hi,

I encountered something similar when I had some non-English characters in my Name field or Comment field. When I deleted those characters, the policy pushed fine.

Hi Don,

Cheers for your response - I did check through that one as it's an old chestnut that used to affect way back to Firewall-1 3.2...

We don't have any rulenames on the rulebase, and the comments are all plain basic ascii without any non-english characters.

Like I said - this rulebase has been pushing fine up until the last couple of days - and the only change I made was to comment out a rule denying an internal machine access to another network in the group because it was no longer required!

As it happens, I notice that my primary firewall is also showing the error;

[LOG_CRIT] kernel: FW-1: bpush: push block size error sz=0 at 0x62f

Which is the SNMP inspection issue I've seen before.

[markw]

Well, here's an update for anyone who is interested.

I rebooted the SPLAT box yesterday, and that made no difference.

So I was very very cautious about rebooting the IP350's themselves, not knowing if they would actually come back up, and especially with them being live firewalls.

I took the gamble to reboot the failover device of the pair last night, and it came back fine, so this evening I did the same to the primary firewall.

It failed over just as it should do, to the secondary device, and it slowly booted and came back on line and took back duty.

And I'm happy to report that by doing that, it's cured the problem. I can successfully apply an updated rulebase!

So if it helps, that was my fix.

[hwallen172]

Here's what I was told when I ran into the same problem:

Symptoms
From time to time a policy push fails with error message 'Load on module failed. No memory'.

After rebooting the firewall the policy pushes successfully.

The module is not using swap space and has plenty of memory.

Answer
By default the rulebase_uid_in_log parameter is set to 'true'. When set to 'true', each rule in the rulebase is logged during the policy installation and this can intermittently cause memory problems particularly with large policies.

By setting the parameter to 'false', these logs are no longer generated during the policy installation and consequently the memory usage on the module is no longer affected during the policy installation. To make the change to the rulebase_uids_in_log property , use Check Point SmartDashboard to go to Global Properties > SmartDashboard Customization > Advanced Configuration > Configure > FireWall-1 > General > rulebase_uids_in_log. Set the property to false. Install the security policy.



I can't tell you if that's a valid fix because I had already rebooted before I received this advice, but I did it anyway just to keep the problem from recurring.

Wiley

[Jay_D]

I got the error when certain things in SmartDefense got checked. Forgot which though...was on a R55.

When we rebooted the ruleset wasn't loaded. I had to use fw unloadlocal to get to my SmartDashboard, uncheck the new SmartDefense settings and then install went ok.

I assume this was a bad SmartDefense update.

[freekazoid]

We also ran into this problem, and we tried the solution suggested by Wiley. I am happy to report that it solved our policy installation problem. We have reported it to Check Point, as they should solve this.

[dfwboiler]

For the first time... I actually got this error and it was a memory issue.
There was a change to add some more servers behind the firewall and that caused the firewalls to have more connections than they could handle.
Added more memory and the issue was resolved.

[pp2hkg]

All,

I got this error also when pushing policy to a new IP530 (R55 FP3) and the console of IP530 showed an error msg:

Sep 10 08:48:48 IP530-TEST-FW [LOG_CRIT] kernel: FW-1: fwurl_patternfilter_load: Failed to compile patterns to regular expressions.

Solution I applied:
SmartDashboard > SmartDefense Tab >
Web Intelligence > Malicious Code > General HTTP Worm Catcher
Select Default_Protection and click on "Deactivate All"

Push the policy again.

Hope this help.
Good Luck.