Appliances 5900 R80.10 and bonding interfaces limited throughput

[juvann]

Hi guys,

I have a clusterxl with two appliances 5900 and Gaia OS R80.10.
Network is configured with 2 bond interfaces 802.3ad layer 3+4, each bond has 4 ethernet aggregated, each bond has more vlan interfaces configured, about 10 in internal bond and 3 on external bond. All ethernet interfaces are connected to cisco switch with LACP configured, 2 groups.
No QoS is configured, no IPS is configured only security policies with about 50 rules.
My test is simple I trying to transfer data between vlan on internal bond and my speed is about 3Gbps and somethings more, the same test on external bond I got 3Gbps.
When I trying to transfer data between vlan on internal bond and vlan on external bond, I reach the limit of ethernet interface 1Gbps. Why?
In my test I have used different source address and destination address so also TCP ports all different
While my test is executing I can see in cpview the throughput on each interfaces, I can see the traffic is balanced between they but the sum don't go over 1Gpbs when I use vlans on both bond interfaces, but when the vlan is on same bond interface I see more interfaces reach the limit of 1Gbps so the sum on bond interface is over the 1Gbps and I got 3Gbps and more of throughput.

Suggestions?
Thanks

[cciesec2006]

Hi guys,

I have a clusterxl with two appliances 5900 and Gaia OS R80.10.
Network is configured with 2 bond interfaces 802.3ad layer 3+4, each bond has 4 ethernet aggregated, each bond has more vlan interfaces configured, about 10 in internal bond and 3 on external bond. All ethernet interfaces are connected to cisco switch with LACP configured, 2 groups.
No QoS is configured, no IPS is configured only security policies with about 50 rules.
My test is simple I trying to transfer data between vlan on internal bond and my speed is about 3Gbps and somethings more, the same test on external bond I got 3Gbps.
When I trying to transfer data between vlan on internal bond and vlan on external bond, I reach the limit of ethernet interface 1Gbps. Why?
In my test I have used different source address and destination address so also TCP ports all different
While my test is executing I can see in cpview the throughput on each interfaces, I can see the traffic is balanced between they but the sum don't go over 1Gpbs when I use vlans on both bond interfaces, but when the vlan is on same bond interface I see more interfaces reach the limit of 1Gbps so the sum on bond interface is over the 1Gbps and I got 3Gbps and more of throughput.

Suggestions?
Thanks

A few questions for you:

1- Is SecureXL enabled?
2- How do you perform the test? Are you using Iperf to do this test?


My guess is that you might have a sim affinity issue based on the symptom you described.

[juvann]

Yes SecureXL is enabled and yes I have used iperf for test.
With cpview I can't see none CPU with 100% usage.

[ShadowPeak.com]

Yes SecureXL is enabled and yes I have used iperf for test.
With cpview I can't see none CPU with 100% usage.

Please provide output of the following commands run on the firewall from expert mode:

fwaccel stat
fwaccel stats -s
grep -c ^processor /proc/cpuinfo
/sbin/cpuinfo
fw ctl affinity -l -r
sim affinity -l
netstat -ni
fw ctl multik stat
fw ctl multik get_mode
cpstat os -f multi_cpu -o 1
free -m
enabled_blades

[cciesec2006]

Yes SecureXL is enabled and yes I have used iperf for test.
With cpview I can't see none CPU with 100% usage.

I assume that you also have this command on your Cisco switches:

port-channel load-balance src-dst-ip

On Cisco newer switches, you also see this:

port-channel load-balance vlan-src-dst-mixed-ip-port

[juvann]

Thank you to all for suggestions! I solved my issue.
I enabled multi queue on ethernet in bond interfaces, and now I got all 2Gpbs with multi connections.

I enabled 4 queue for driver igb
cpmq set rx_num igb 4

Enabled queue for interfaces
cpmq set -f

and after reboot I have this situation:

# cpmq get -v

Active igb interfaces:
Mgmt [Off]
Sync [Off]
eth1 [On]
eth1-01 [On]
eth1-05 [On]
eth5 [On]
eth8 [Off]

The rx_num for igb is: 4

multi-queue affinity for igb interfaces:
CPU | TX | Vector                      | RX Bytes
-------------------------------------------------------------
0   | 0  | eth1-01-TxRx-0 (186)        | 30661660675
    |    | eth1-05-TxRx-0 (91)         |
    |    | eth1-TxRx-0 (179)           |
    |    | eth5-TxRx-0 (227)           |
1   | 2  | eth1-01-TxRx-2 (202)        | 19214905571
    |    | eth1-05-TxRx-2 (107)        |
    |    | eth1-TxRx-2 (195)           |
    |    | eth5-TxRx-2 (52)            |
2   | 0  |                             |
3   | 2  |                             |
4   | 0  |                             |
5   | 2  |                             |
6   | 0  |                             |
7   | 2  |                             |
8   | 1  | eth1-01-TxRx-1 (194)        | 26115216389
    |    | eth1-05-TxRx-1 (99)         |
    |    | eth1-TxRx-1 (187)           |
    |    | eth5-TxRx-1 (235)           |
9   | 3  | eth1-01-TxRx-3 (210)        | 683483583031
    |    | eth1-05-TxRx-3 (115)        |
    |    | eth1-TxRx-3 (203)           |
    |    | eth5-TxRx-3 (60)            |
10  | 1  |                             |
11  | 3  |                             |
12  | 1  |                             |
13  | 3  |                             |
14  | 1  |                             |
15  | 3  |                             |

[ShadowPeak.com]

Thank you to all for suggestions! I solved my issue.
I enabled multi queue on ethernet in bond interfaces, and now I got all 2Gpbs with multi connections.

I enabled 4 queue for driver igb
cpmq set rx_num igb 4

Enabled queue for interfaces
cpmq set -f

and after reboot I have this situation:

# cpmq get -v

Active igb interfaces:
Mgmt [Off]
Sync [Off]
eth1 [On]
eth1-01 [On]
eth1-05 [On]
eth5 [On]
eth8 [Off]

The rx_num for igb is: 4

multi-queue affinity for igb interfaces:
CPU | TX | Vector                      | RX Bytes
-------------------------------------------------------------
0   | 0  | eth1-01-TxRx-0 (186)        | 30661660675
    |    | eth1-05-TxRx-0 (91)         |
    |    | eth1-TxRx-0 (179)           |
    |    | eth5-TxRx-0 (227)           |
1   | 2  | eth1-01-TxRx-2 (202)        | 19214905571
    |    | eth1-05-TxRx-2 (107)        |
    |    | eth1-TxRx-2 (195)           |
    |    | eth5-TxRx-2 (52)            |
2   | 0  |                             |
3   | 2  |                             |
4   | 0  |                             |
5   | 2  |                             |
6   | 0  |                             |
7   | 2  |                             |
8   | 1  | eth1-01-TxRx-1 (194)        | 26115216389
    |    | eth1-05-TxRx-1 (99)         |
    |    | eth1-TxRx-1 (187)           |
    |    | eth5-TxRx-1 (235)           |
9   | 3  | eth1-01-TxRx-3 (210)        | 683483583031
    |    | eth1-05-TxRx-3 (115)        |
    |    | eth1-TxRx-3 (203)           |
    |    | eth5-TxRx-3 (60)            |
10  | 1  |                             |
11  | 3  |                             |
12  | 1  |                             |
13  | 3  |                             |
14  | 1  |                             |
15  | 3  |                             |

Great, thanks for the follow-up.