SPLAT NGx R65 with HFA_30 and memory consumption

[cciesec2006]

I have SPLAT NGx R65 with HFA_30 standalone (Enforcement module and
SmartCenter) running a Dell Poweredege 2650 (dual processors 2.8Ghz
and 2GB RAM).

Ever since I ugraded the SPLAT box to HFA_30, I see that the memory
consumption on this box goes from 500MB to almost 1.9MB even though
there are virtually NO traffics going across this firewall. As you
can see from the output, memory consumption is almost 1.9GB.

Can anyone tell why it is this way?

Thanks in advance.


23:35:49 up 5 days, 11:26, 1 user, load average: 0.01, 0.02, 0.00
232 processs: 231 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.6% 0.0% 0.6% 0.0% 0.2% 0.2% 197.8%
cpu00 0.2% 0.0% 0.0% 0.0% 0.0% 0.2% 99.4%
cpu01 0.5% 0.0% 0.7% 0.0% 0.2% 0.0% 98.4%
Mem: 2055308k av, 1882548k used, 172760k free, 0k shrd, 114456k buff
443692k actv, 960984k in_d, 30144k in_c
Swap: 4192956k av, 0k used, 4192956k free 1392644k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1327 root 15 0 55188 53M 8284 S 199.9 2.6 0:10 0 cpd
1620 root 15 0 16220 15M 3720 S 33.1 0.7 0:01 1 in.msd
1561 root 15 0 16220 15M 3720 S 31.3 0.7 0:01 0 in.msd
1486 root 15 0 16220 15M 3720 S 30.1 0.7 0:01 0 in.msd
1549 root 15 0 16220 15M 3720 S 29.8 0.7 0:01 0 in.msd
1573 root 15 0 16220 15M 3720 S 29.8 0.7 0:01 1 in.msd
1572 root 15 0 16220 15M 3720 S 29.6 0.7 0:01 1 in.msd
1585 root 15 0 16220 15M 3720 S 29.3 0.7 0:01 0 in.msd
1547 root 15 0 16220 15M 3720 S 28.3 0.7 0:01 1 in.msd
1499 root 15 0 16220 15M 3720 S 27.8 0.7 0:01 1 in.msd
1602 root 15 0 16220 15M 3720 S 27.8 0.7 0:01 0 in.msd
1496 root 15 0 16220 15M 3720 S 27.6 0.7 0:01 0 in.msd
1541 root 15 0 16220 15M 3720 S 27.6 0.7 0:01 0 in.msd
1612 root 15 0 16220 15M 3720 S 27.6 0.7 0:01 1 in.msd
1490 root 15 0 16220 15M 3720 S 27.3 0.7 0:01 0 in.msd
1520 root 15 0 16220 15M 3720 S 27.3 0.7 0:01 0 in.msd
1592 root 15 0 16220 15M 3720 S 26.8 0.7 0:01 0 in.msd
1610 root 15 0 16220 15M 3720 S 26.8 0.7 0:01 1 in.msd
1509 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 0 in.msd
1536 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 0 in.msd
1554 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 1 in.msd
1563 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 0 in.msd
1571 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 1 in.msd
1633 root 15 0 16220 15M 3720 S 26.6 0.7 0:01 0 in.msd
1553 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 1 in.msd
1560 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 0 in.msd
1564 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 0 in.msd
1565 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 1 in.msd
1582 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 1 in.msd
1588 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 1 in.msd
1601 root 15 0 16220 15M 3720 S 26.3 0.7 0:01 1 in.msd
1504 root 15 0 16220 15M 3720 S 26.1 0.7 0:01 1 in.msd
1491 root 15 0 16220 15M 3720 S 25.8 0.7 0:01 0 in.msd
1529 root 15 0 16220 15M 3720 S 25.8 0.7 0:01 1 in.msd
1533 root 15 0 16220 15M 3720 S 25.8 0.7 0:01 0 in.msd

[Expert@NEO-labgw]# vmstat 2 10
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
1 0 0 172944 114456 1392640 0 0 1 45 45 4 0 0 0 8
0 0 0 172944 114456 1392640 0 0 0 70 150 36 0 0 0 100
0 0 0 172944 114456 1392640 0 0 0 0 191 31 0 0 0 100
0 0 0 172952 114456 1392640 0 0 0 22 143 32 0 0 0 100
0 0 0 172952 114456 1392640 0 0 0 0 146 35 0 0 0 100
0 0 0 172952 114456 1392640 0 0 0 0 137 106 0 0 0 100
0 0 0 172960 114456 1392640 0 0 0 94 154 35 0 0 0 100
0 0 0 172960 114456 1392644 0 0 0 32 208 70 0 0 0 100
0 0 0 172960 114456 1392644 0 0 0 46 147 36 0 0 0 100
0 0 0 172960 114456 1392644 0 0 0 0 139 27 0 0 0 100
[Expert@NEO-labgw]#
[Expert@NEO-labgw]# more /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 2104635392 1927409664 177225728 0 117202944 1426075648
Swap: 4293586944 0 4293586944
MemTotal: 2055308 kB
MemFree: 173072 kB
MemShared: 0 kB
Buffers: 114456 kB
Cached: 1392652 kB
SwapCached: 0 kB
Active: 443412 kB
ActiveAnon: 181440 kB
ActiveCache: 261972 kB
Inact_dirty: 960992 kB
Inact_laundry: 303596 kB
Inact_clean: 30144 kB
Inact_target: 347628 kB
HighTotal: 131008 kB
HighFree: 17572 kB
LowTotal: 1924300 kB
LowFree: 155500 kB
SwapTotal: 4192956 kB
SwapFree: 4192956 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
[Expert@NEO-labgw]#

[melipla]

Hi. Starting up top and showing threads I see now see a lot of the same processes: in.msd. This is new behavior which I did not see before I applied hfa 30. The file itself is a link to fwssd which is Check Point's Security Server Daemon. I only see some Security Server changes in the R65 HFA1 release notes, so its unclear to me why this would now use additional memory (or need to start so many processes).

[abusharif]

Is Messaging security activated?

I've recently done upgrade of R60 HFA04 -> R65 HFA30 on splat, but without messaging mumbojumbo. Only one in.msd is active and idle.

The box is Dell as well 2gb ram and about 1.8gb is "used" (1.3gb cached)

This unit has traffic going through it, vpns etc.

Generally high amount of "used" memory on Linux isnt a problem imho unless ofc all memory is out and it starts using disk.
Only possible issue from what you posted is excessive(?) amount of in.msd started.

[cciesec2006]

Originally Posted by abusharif

Is Messaging security activated?

I've recently done upgrade of R60 HFA04 -> R65 HFA30 on splat, but without messaging mumbojumbo. Only one in.msd is active and idle.

The box is Dell as well 2gb ram and about 1.8gb is "used" (1.3gb cached)

This unit has traffic going through it, vpns etc.

Generally high amount of "used" memory on Linux isnt a problem imho unless ofc all memory is out and it starts using disk.
Only possible issue from what you posted is excessive(?) amount of in.msd started.

I do NOT use messanging security. Furthermore, this box has virtually NO
traffics going across.

I am beginning to believe that Checkpoint has NOT tested this HFA_30
extensively. Full of bugs, again.

[cciesec2006]

Anyone else seeing the same issue I am seeing with HFA_30? I just
installed HFA_30 on another SPLAT box and I am seeing the same thing,
memory utilization goes from 200MB to 1.9GB on a system with 2GB RAM.

[melipla]

I'm seeing varying usage:

1849.82/2006.86 active cluster
881.82/2006.86 standby

653.54/1001.03 active cluster
645.17/1001.03 standby

716.61/1000.94

I don't see all of the in.msd processes active like you do. Its only when I switch to "show all threads" that I see the long list briefly. Debugging the process could help to identify the cause of why you see so many processes active at once, question is, is that possible?

[ccie16798]

hello

i didn't dare to try HFA_30, but i would try to disable all useless dameons in $FWDIR/conf/fwauthd.conf
that's common best practice, fwd tends to load a lot of crappy and useless processes. I'm glad to see checkpoint added a new buggy one ;)

Etienne

Originally Posted by cciesec2006

Anyone else seeing the same issue I am seeing with HFA_30? I just
installed HFA_30 on another SPLAT box and I am seeing the same thing,
memory utilization goes from 200MB to 1.9GB on a system with 2GB RAM.

[cciesec2006]

Originally Posted by ccie16798

hello

i didn't dare to try HFA_30, but i would try to disable all useless dameons in $FWDIR/conf/fwauthd.conf
that's common best practice, fwd tends to load a lot of crappy and useless processes. I'm glad to see checkpoint added a new buggy one ;)

Etienne

Funny you said that because that was the first thing I did whenever I build
CP firewall. I commented out almost everything ins the
$FWDIR/conf/fwauthd.conf file.

That doesn't help either.

[melipla]

Originally Posted by cciesec2006

Can anyone tell why it is this way?

Since installing HFA 30 I've been getting errors about being unable to install the policy because "Policy installation is in progress.". This happens if I need to install two policies on two different gateways. I'll install the first, load up the other policy and when I try to push I get the error message.

The problem started about a month after I installed HFA 30, but has crept up at about two week intervals since then. I've verified that the first policy installation finishes and that the gateway does have the latest policy. If I wait 10 minutes then I am able to install the policy again, or if I cpstop / cpstart / reboot the smartcenter server then I am ok [for two weeks].

I think it may be related to a memory leak, because the third time the problem crept up I noticed the smartcenter memory usage went from 35% -> 100% (it took a couple of hours to use 100%). This was something I didn't check the first two times but since you had posted these messages lately it was something I checked. So when I spotted the 35% memory usage, the smartcenter server had been up for a month and used constantly, meaning if all the RAM were to be "used" it should've hit 100% before the problem hit. Now the smartcenter server is at 100% utilization of 8 gigs of RAM (I haven't cpstopped or rebooted since the last incident) and I can replicate the problem at will.

I haven't been too successful with identifying the culprit of the memory usage, but thought it might be relevant to what you're seeing. Hopefully next week I can spend time to resolve the policy installation is in progress error...

[melipla]

Unfortunately I had to cpstop/cpstart so progress on my SR is delayed while I wait for the problem to happen again. One thing the tech said was that cpd memory consumption was abnormal. I found an SK article today which lists all the known issues in R65, fixed and outstanding, sk36267. One of which is a memory leak in cpd for SmartCenter. Unfortunately I was on a gateway today & saw that cpd was consuming 33% of the memory, so I'm thinking the memory leak is not only isolated to the SmartCenter...