CoreXL for 4 core system

[Raj909]

We have several 4800 clusters running VSX, with 2 virtual systems (1 for firewall and 1 for VPN). Since it is a 4 core box, I’d like to ask what the optimum setup is.

They are running R77.30 and our SE recommended the following:
• CoreXL is disabled on VS0
• Assign 2 cores to Firewall VS and 1 core to VPN VS, leaving 1 core for interfaces
• 2 virtual system instances assigned to Firewall VS
• 1 virtual system instances assigned to VPN VS

According to top and cpview, Core 0 is getting hammered with high (si). This happens intermittently and when it does there is high latency. From what I have read, CoreXL instances and SND should not share a core. Separating the internal and external interfaces on separate cores makes sense, but how to allocate enough for FWK? What is max virtual instances we should define per VS?

Thanks

[varera]

You have 4 cores in your disposal, so it makes sense to use 1 SND and 3 FWK instances. The recommendation from your SE is reasonable. However, it is unclear what Core0 is doing in your case. Is it SND or not? Do you have any fragmentation issues in the network?

[cciesec2006]

You have 4 cores in your disposal, so it makes sense to use 1 SND and 3 FWK instances. The recommendation from your SE is reasonable. However, it is unclear what Core0 is doing in your case. Is it SND or not? Do you have any fragmentation issues in the network?

Having deal with high CPU issue on the firewall on both the Power-1 appliances and open servers, both 805 licenses (8 cores and 5 blades but I am only using FW blade), I can offer my opinion below:

1- fragmentation might be an issue but unlikely because your application must be intentionally doing it so it is unlikely the cause,

2- the traffics traversing the firewall is under some kind of checkpoint enforcement like SQL'net or Microsoft DFS. those are CPU killers.

3- if you're going to disable CoreXL, I think it will make the situation worse.

my 2c.

[varera]

Having deal with high CPU issue on the firewall on both the Power-1 appliances and open servers, both 805 licenses (8 cores and 5 blades but I am only using FW blade), I can offer my opinion below:

1- fragmentation might be an issue but unlikely because your application must be intentionally doing it so it is unlikely the cause,

2- the traffics traversing the firewall is under some kind of checkpoint enforcement like SQL'net or Microsoft DFS. those are CPU killers.

3- if you're going to disable CoreXL, I think it will make the situation worse.

my 2c.

That may only be partially correct.

Software Interrupts are FW kernel operations or SecureXL processing. Most of FW tasks today are outsourced to FWK instances, but de-fragmentation is not. Deep inspection is done by FWK and would look like user mode CPU utilization.

Yet, without sim affinity info we are just guessing here. Let's wait till TS posts sim affinity details

[Raj909]

Not planning to disable CoreXL, wondering if a 2/2 split will be fine (2 SND, 2 FWK)

[Expert@fw-:0]# fw ctl affinity -l
eth1: CPU 0
eth7: CPU 0
Mgmt: CPU 0
VS_0 fwk: CPU 1 2 3
VS_1 fwk: CPU 1 2 3
VS_2 fwk: CPU 1 2 3
VS_3 fwk: CPU 1 2 3

[Expert@fw:0]# sim affinity -l
Mgmt : 0
eth1 : 0
eth2 : 0
eth3 : 0
eth4 : 0
eth5 : 0
eth6 : 0
eth7 : 0

top - 17:24:28 up 124 days, 21:42, 3 users, load average: 1.55, 1.31, 1.18
Tasks: 183 total, 2 running, 181 sleeping, 0 stopped, 0 zombie
Cpu0 : 32.7%us, 4.3%sy, 0.0%ni, 14.7%id, 0.0%wa, 5.7%hi, 42.7%si, 0.0%st
Cpu1 : 24.3%us, 9.3%sy, 0.0%ni, 65.1%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st
Cpu2 : 17.2%us, 3.6%sy, 1.9%ni, 75.1%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st
Cpu3 : 26.6%us, 6.2%sy, 0.0%ni, 62.7%id, 0.0%wa, 0.0%hi, 4.5%si, 0.0%st
Mem: 8003244k total, 7556768k used, 446476k free, 643060k buffers
Swap: 18900464k total, 600k used, 18899864k free, 2485788k cached

[varera]

Okay, it seems there is an issue with SXL core. The issue is, we do not know what is that, yet. Try getting SXL statistics, look for any acceleration errors, traffic spikes, etc

[cciesec2006]

Okay, it seems there is an issue with SXL core. The issue is, we do not know what is that, yet. Try getting SXL statistics, look for any acceleration errors, traffic spikes, etc

In my situation, see below:

top - 13:38:58 up 16 days, 16:14, 5 users, load average: 3.02, 1.11, 0.50
Tasks: 168 total, 6 running, 162 sleeping, 0 stopped, 0 zombie
Cpu0 : 12.0%us, 2.0%sy, 0.0%ni, 54.0%id, 0.0%wa, 0.0%hi, 32.0%si, 0.0%st
Cpu1 : 1.9%us, 1.9%sy, 0.0%ni, 86.5%id, 0.0%wa, 0.0%hi, 9.6%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 32.0%id, 0.0%wa, 0.0%hi, 68.0%si, 0.0%st
Cpu3 : 0.0%us, 2.0%sy, 0.0%ni, 34.0%id, 0.0%wa, 0.0%hi, 64.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni, 34.7%id, 0.0%wa, 0.0%hi, 65.3%si, 0.0%st
Cpu5 : 0.0%us, 2.0%sy, 0.0%ni, 62.0%id, 0.0%wa, 0.0%hi, 36.0%si, 0.0%st
Cpu6 : 0.0%us, 2.0%sy, 0.0%ni, 60.8%id, 0.0%wa, 0.0%hi, 37.3%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni, 9.6%id, 0.0%wa, 0.0%hi, 89.4%si, 0.0%st
Mem: 5960440k total, 3927864k used, 2032576k free, 289404k buffers
Swap: 14707496k total, 0k used, 14707496k free, 1710600k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5297 admin 15 0 0 0 0 R 89 0.0 7857:58 fw_worker_0
5302 admin 15 0 0 0 0 R 67 0.0 8181:43 fw_worker_5
5300 admin 15 0 0 0 0 S 65 0.0 1565:45 fw_worker_3
5301 admin 15 0 0 0 0 R 65 0.0 583:03.26 fw_worker_4
5298 admin 15 0 0 0 0 R 38 0.0 387:51.57 fw_worker_1
5299 admin 15 0 0 0 0 R 38 0.0 1709:09 fw_worker_2

Firewall is passing about 800Mbits/sec of SQL'net traffics and I can see latencies with other traffics. "fwaccel stats -s" shows that 99% of the traffics go through F2F which causes the CPU spikes, no packet fragmentation and I also have DD enable on the gateways, and that my SQL'net is at the very top of the rule base:

Every 2.0s: fwaccel stats -s Mon Apr 17 13:40:59 2017

Accelerated conns/Total conns : 11/31 (35%)
Accelerated pkts/Total pkts : 4104/32792559 (0%)
F2Fed pkts/Total pkts : 32788455/32792559 (99%)
PXL pkts/Total pkts : 0/32792559 (0%)
QXL pkts/Total pkts : 0/32792559 (0%)

fw ctl affinity -l -r -v -a
CPU 0: Sync (irq 139) Exp2-2 (irq 91) Lan1 (irq 235) Exp1-1 (irq 107)
CPU 1: Exp2-1 (irq 75) Lan2 (irq 211) Exp1-2 (irq 123)
CPU 2: fw_5
CPU 3: fw_4
CPU 4: fw_3
CPU 5: fw_2
CPU 6: fw_1
CPU 7: fw_0
All: mpdaemon rtmd fwd cprid cpd

[varera]

In my situation, see below:

It seems to me you are trying to highjack this thread. Just open a new one, would you?

[Raj909]

Varera, here you go:

[Expert@fw:1]# fwaccel stats -s
Accelerated conns/Total conns : 332/11550 (2%)
Delayed conns/(Accelerated conns + PXL conns) : 169/6029 (2%)
Accelerated pkts/Total pkts : 2921044/202601942 (1%)
F2Fed pkts/Total pkts : 19855139/202601942 (9%)
PXL pkts/Total pkts : 179825759/202601942 (88%)
QXL pkts/Total pkts : 0/202601942 (0%)

[Expert@fw:1]# fwaccel stats
Name Value Name Value
-------------------- --------------- -------------------- ---------------

Accelerated Path
------------------------------------------------------------------------------
accel packets 2919885 accel bytes 1674697195
conns created 1282897 conns deleted 826982
C total conns 11691 C templates 376
C TCP conns 10134 C delayed TCP conns 175
C non TCP conns 1557 C delayed nonTCP con 0
conns from templates 292198 temporary conns 39539
nat conns 732999 dropped packets 38147
dropped bytes 51419194 nat templates 0
port alloc templates 0 conns from nat tmpl 0
port alloc conns 0 conns auto expired 681966

Accelerated VPN Path
------------------------------------------------------------------------------
C crypt conns 0 enc bytes 0
dec bytes 0 ESP enc pkts 0
ESP enc err 0 ESP dec pkts 0
ESP dec err 0 ESP other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0

Medium Path
------------------------------------------------------------------------------
PXL packets 179560871 PXL async packets 179844554
PXL bytes 122105304015 C PXL conns 5804
C PXL templates 331 PXL FF conns 0
PXL FF packets 0 PXL FF bytes 0
PXL FF acks 0

Accelerated QoS Path
------------------------------------------------------------------------------
QXL packets 0 QXL async packets 0
QXL bytes 0 C QXL conns 0
C QXL templates 0

Firewall Path
------------------------------------------------------------------------------
F2F packets 19837892 F2F bytes 11548151109
C F2F conns 5546 TCP violations 101711
C partial conns 0 C anticipated conns 0
port alloc f2f 0

GTP
------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
------------------------------------------------------------------------------
memory used 0 free memory 0
C used templates 235 pxl tmpl conns 240070
C conns from tmpl 1751 C non TCP F2F conns 1170
C tcp handshake conn 178 C tcp established co 9356
C tcp closed conns 600 C tcp f2f handshake 112
C tcp f2f establishe 4220 C tcp f2f closed con 44
C tcp pxl handshake 62 C tcp pxl establishe 4893
C tcp pxl closed con 463 outbound packets 2919885
outbound pxl packets 179554673 outbound f2f packets 18331611
outbound bytes 1721164081 outbound pxl bytes 124872806273
outbound f2f bytes 10315699211

[varera]

one time look does not help. You need to check which stats are changing when you are experiencing higher latency.

[ba3113]

VPN always use CPU0 so assign different CPU to SND.

[ba3113]

VWhen CoreXL is enabled, VPN traffic inspection occurs only in global CoreXL FW instance #0 (fw_worker_0). By design, global CoreXL FW instance #0 (fw_worker_0) always runs on the CPU core with highest ID (as allowed by the current CoreXL license) so so assign different CPU to SND.

[ba3113]

Can you also share fwaccel stat output which will give idea where securexl is breaking.

[cciesec2006]

Can you also share fwaccel stat output which will give idea where securexl is breaking.

Either "watch -d -n 1 fwaccel stats -p" or "watch -d -n 1 fwaccel stats -s"

Every 1.0s: fwaccel stats -s Thu Apr 27 09:50:23 2017

Accelerated conns/Total conns : 11/22 (50%)
Accelerated pkts/Total pkts : 100135/34355268 (0%) --- secureXL is accelerating traffics GOOD !!!!! (but none for me)
F2Fed pkts/Total pkts : 34246331/34355268 (99%) ---- it is going through the F2F means that SecureXL can not accelerate your traffics BAD !!!!!
PXL pkts/Total pkts : 8802/34355268 (0%)
QXL pkts/Total pkts : 0/34355268 (0%)


Every 2.0s: fwaccel stats -p Thu Apr 27 09:51:52 2017

F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 2615975 pkt has IP options 0
ICMP miss conn 456178 TCP-SYN miss conn 1248
TCP-other miss conn 622 UDP miss conn 27379154 this is BAD !!!!!
other miss conn 20130 VPN returned F2F 0
ICMP conn is F2Fed 0 TCP conn is F2Fed 9940779 ---- this is BAD !!!!
UDP conn is F2Fed 863 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 1
TCP state viol 127 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 0 cluster forward 0
chain forwarding 0 general reason 0

[ba3113]

Can you please share fwaccel stat not the fwaccel stats.