SPLAT High CPU utilization in standby cluster member

[omahrez]

hI,
Can you tell me what is wrong with my clusterxl; when i disable cluster membership of the standby firewall, the cpu load is normal, but as soon as I enable the membership, the CPU load is almost 100%.
Any idea ? thanks.

below the trace of top command, followed by the trace of cphaprob .



6:00pm up 21 min, 1 user, load average: 1.38, 1.30, 0.94
60 processes: 56 sleeping, 4 running, 0 zombie, 0 stopped
CPU states: 93.7% user, 0.7% system, 0.0% nice, 5.4% idle
Mem: 3090600K av, 176360K used, 2914240K free, 4K shrd, 13004K buff
Swap: 6291400K av, 0K used, 6291400K free 55840K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
826 root 14 0 15800 15M 9172 R 99.9 0.5 19:45 fw
1 root 9 0 504 504 436 S 0.0 0.0 0:04 init
2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 19 19 0 0 0 RWN 0.0 0.0 0:00 ksoftirqd_CPU0
4 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd
5 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd
6 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
7 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
13 root 9 0 0 0 0 SW 0.0 0.0 0:00 scsi_eh_0
16 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
82 root 9 0 0 0 0 SW 0.0 0.0 0:00 khubd
148 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
158 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
159 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
160 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
161 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
476 root 9 0 628 628 520 S 0.0 0.0 0:00 syslogd
482 root 9 0 1152 1152 452 S 0.0 0.0 0:00 klogd
575 root 9 0 1304 1304 1164 S 0.0 0.0 0:00 sshd
645 root 9 0 644 644 556 S 0.0 0.0 0:00 crond
673 root 9 0 936 936 788 S 0.0 0.0 0:00 cp_http_server_
677 root 9 0 900 900 764 S 0.0 0.0 0:00 cpwmd_wd
686 root 9 0 1164 1164 852 S 0.0 0.0 0:00 cpwmd
697 root 9 0 2764 2764 2072 S 0.0 0.0 0:00 cprid
701 nobody 9 0 3712 3712 2728 S 0.0 0.1 0:00 cp_http_server
724 root 9 0 1600 1600 972 S 0.0 0.0 0:00 cpwd
736 root 9 0 15504 15M 8224 S 0.0 0.5 0:00 cpd
738 root 9 0 15504 15M 8224 S 0.0 0.5 0:00 cpd
805 root 9 0 15504 15M 8224 S 0.0 0.5 0:00 cpd
806 root 9 0 15504 15M 8224 S 0.0 0.5 0:00 cpd
822 root 9 0 6808 6808 3120 S 0.0 0.2 0:00 cphamcset
823 root 9 0 6808 6808 3120 S 0.0 0.2 0:00 cphamcset
824 root 9 0 6808 6808 3120 S 0.0 0.2 0:00 cphamcset
835 root 9 0 15800 15M 9172 S 0.0 0.5 0:00 fw
836 root 9 0 15800 15M 9172 S 0.0 0.5 0:00 fw
837 root 9 0 15800 15M 9172 S 0.0 0.5 0:00 fw
855 root 9 0 11860 11M 7000 S 0.0 0.3 0:00 in.asessiond
856 root 9 0 11760 11M 6976 S 0.0 0.3 0:00 in.aufpd
857 root 9 0 14840 14M 7368 S 0.0 0.4 0:00 vpnd
858 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
866 root 9 0 11860 11M 7000 S 0.0 0.3 0:00 in.asessiond
867 root 9 0 11760 11M 6976 S 0.0 0.3 0:00 in.aufpd
868 root 9 0 11860 11M 7000 S 0.0 0.3 0:00 in.asessiond
869 root 9 0 11760 11M 6976 S 0.0 0.3 0:00 in.aufpd
870 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
871 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
873 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
874 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
875 root 9 0 12192 11M 7072 S 0.0 0.3 0:00 mdq
876 root 9 0 14840 14M 7368 S 0.0 0.4 0:00 vpnd












fwone# fw ctl pstat

Hash kernel memory (hmem) statistics:
Total memory allocated: 6291456 bytes in 1534 4KB blocks using 2 pools
Total memory bytes used: 396812 unused: 5894644 (93.69%) peak: 399712
Total memory blocks used: 125 unused: 1409 (91%) peak: 125
Allocations: 6956 alloc, 0 failed alloc, 598 free

System kernel memory (smem) statistics:
Total memory bytes used: 16593088 peak: 16595376
Blocking memory bytes used: 1642084 peak: 1644372
Non-Blocking memory bytes used: 14951004 peak: 14951004
Allocations: 54925 alloc, 0 failed alloc, 54613 free, 0 failed free

Kernel memory (kmem) statistics:
Total memory bytes used: 10686560 peak: 10689460
Allocations: 7152 alloc, 0 failed alloc, 655 free, 0 failed free

Kernel stacks:
0 bytes total, 0 bytes stack size, 0 stacks,
0 peak used, 0 max stack bytes used, 0 min stack bytes used,
0 failed stack calls

INSPECT:
46 packets, 15611 operations, 348 lookups,
0 record, 3530 extract

Cookies:
6392 total, 0 alloc, 0 free,
58 dup, 1121 get, 140 put,
6612 len, 55 cached len, 0 chain alloc,
0 chain free

Connections:
19 total, 6 TCP, 0 UDP, 3 ICMP,
10 other, 0 anticipated, 0 recovered, 4 concurrent,
18 peak concurrent, 836 lookups

Fragments:
0 fragments, 0 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
0/0 forw, 0/0 bckw, 0 tcpudp,
0 icmp, 7-6 alloc

Sync:
Version: new
Status:
Unable to send sync packets
Unable to receive sync packets
Sync packets sent:
total : 0, retransmitted : 0, retrans reqs : 0, acks : 0
Sync packets received:
total : 0, were queued : 0, dropped by net : 0
retrans reqs : 0, received 0 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0