Too many open files on Solaris, but not a rlim_fd_max issue

[moergi]

We are running a R55 cluster on Solaris 8. The fwd daemon opens an unsually high amount of file descriptors. When the 256th file descriptors is returned by open(), fwd refuses it with "too many open files".

rlim_fd_max and rlim_fd_cur are set to 1024. The system call open() returns 256, not an error.

So I think fwd is simply not supposed to open that many files.

When looking at what files fwd opens, it looks like this:

# pfiles `pgrep -fx fwd`
465: fwd
Current rlimit: 1024 file descriptors
0: S_IFCHR mode:0666 dev:32,0 ino:112666 uid:0 gid:3 rdev:13,2
O_RDONLY|O_LARGEFILE
1: S_IFCHR mode:0666 dev:32,0 ino:112666 uid:0 gid:3 rdev:13,2
O_WRONLY|O_LARGEFILE
2: S_IFCHR mode:0666 dev:32,0 ino:112666 uid:0 gid:3 rdev:13,2
O_WRONLY|O_LARGEFILE
[snip]
252: S_IFREG mode:0775 dev:32,0 ino:1588454 uid:0 gid:0 size:0
O_WRONLY
253: S_IFREG mode:0660 dev:32,0 ino:1447536 uid:0 gid:0 size:0
O_WRONLY
254: S_IFREG mode:0775 dev:32,0 ino:1588455 uid:0 gid:0 size:0
O_WRONLY
255: S_IFREG mode:0660 dev:32,0 ino:1447536 uid:0 gid:0 size:0
O_WRONLY

Or with lsof:

[root@juliette] /opt/CPfw1-R55/log # lsof -p `pgrep -fx fwd`
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
fw 465 root cwd VDIR 32,0 1024 2 /
fw 465 root txt VREG 32,0 457768 1126537 /opt/CPfw1-R55/lib/libCPLogKlogUnify.so
fw 465 root txt VREG 32,0 55456 157814 /opt/CPshrd-R55/lib/libcpP11Modules.so
fw 465 root txt VREG 32,0 134452 157850 /opt/CPshrd-R55/lib/libfwadb.so
[snip]
fw 465 root 247w VREG 32,0 0 1447536 / (/dev/dsk/c1t0d0s0)
fw 465 root 248w VREG 32,0 0 1588452 / (/dev/dsk/c1t0d0s0)
fw 465 root 249w VREG 32,0 0 1447536 / (/dev/dsk/c1t0d0s0)
fw 465 root 250w VREG 32,0 0 1588453 / (/dev/dsk/c1t0d0s0)
fw 465 root 251w VREG 32,0 0 1447536 / (/dev/dsk/c1t0d0s0)
fw 465 root 252w VREG 32,0 0 1588454 / (/dev/dsk/c1t0d0s0)
fw 465 root 253w VREG 32,0 0 1447536 / (/dev/dsk/c1t0d0s0)
fw 465 root 254w VREG 32,0 0 1588455 / (/dev/dsk/c1t0d0s0)
fw 465 root 255w VREG 32,0 0 1447536 / (/dev/dsk/c1t0d0s0)

Here the open() system call:

465: open("/opt/CPfw1-R55/tmp/ether.tmp", O_RDONLY) = 256
465: close(256)

And here the message in fwd.elg:

Failed to open file 'ether.tmp': Too many open files

Does anybody have seen this before?

Thanks,
Marc

[yowieWithin]

If you have just changed some firewall objects, or imported the policy & objects you may have firewall objects clashing e.g. with a shared address or name.
YW

[moergi]

Thanks YW for your input.

I verified each IP on each member. No clashing. BTW, it happens only on the active node.

Cheers,
Marc

[bvanniekerk]

Hi

You didn't perhaps get a weird message "rand_something something_entropy: Failed for all sources" from the GUI, did you?

rgrds
b