CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 7 of 7

Thread: Appliances 5900 R80.10 and bonding interfaces limited throughput

  1. #1
    Join Date
    2018-10-06
    Posts
    6
    Rep Power
    0

    Default Appliances 5900 R80.10 and bonding interfaces limited throughput

    Hi guys,

    I have a clusterxl with two appliances 5900 and Gaia OS R80.10.
    Network is configured with 2 bond interfaces 802.3ad layer 3+4, each bond has 4 ethernet aggregated, each bond has more vlan interfaces configured, about 10 in internal bond and 3 on external bond. All ethernet interfaces are connected to cisco switch with LACP configured, 2 groups.
    No QoS is configured, no IPS is configured only security policies with about 50 rules.
    My test is simple I trying to transfer data between vlan on internal bond and my speed is about 3Gbps and somethings more, the same test on external bond I got 3Gbps.
    When I trying to transfer data between vlan on internal bond and vlan on external bond, I reach the limit of ethernet interface 1Gbps. Why?
    In my test I have used different source address and destination address so also TCP ports all different
    While my test is executing I can see in cpview the throughput on each interfaces, I can see the traffic is balanced between they but the sum don't go over 1Gpbs when I use vlans on both bond interfaces, but when the vlan is on same bond interface I see more interfaces reach the limit of 1Gbps so the sum on bond interface is over the 1Gbps and I got 3Gbps and more of throughput.

    Suggestions?
    Thanks

  2. #2
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Quote Originally Posted by juvann View Post
    Hi guys,

    I have a clusterxl with two appliances 5900 and Gaia OS R80.10.
    Network is configured with 2 bond interfaces 802.3ad layer 3+4, each bond has 4 ethernet aggregated, each bond has more vlan interfaces configured, about 10 in internal bond and 3 on external bond. All ethernet interfaces are connected to cisco switch with LACP configured, 2 groups.
    No QoS is configured, no IPS is configured only security policies with about 50 rules.
    My test is simple I trying to transfer data between vlan on internal bond and my speed is about 3Gbps and somethings more, the same test on external bond I got 3Gbps.
    When I trying to transfer data between vlan on internal bond and vlan on external bond, I reach the limit of ethernet interface 1Gbps. Why?
    In my test I have used different source address and destination address so also TCP ports all different
    While my test is executing I can see in cpview the throughput on each interfaces, I can see the traffic is balanced between they but the sum don't go over 1Gpbs when I use vlans on both bond interfaces, but when the vlan is on same bond interface I see more interfaces reach the limit of 1Gbps so the sum on bond interface is over the 1Gbps and I got 3Gbps and more of throughput.

    Suggestions?
    Thanks
    A few questions for you:

    1- Is SecureXL enabled?
    2- How do you perform the test? Are you using Iperf to do this test?


    My guess is that you might have a sim affinity issue based on the symptom you described.

  3. #3
    Join Date
    2018-10-06
    Posts
    6
    Rep Power
    0

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Yes SecureXL is enabled and yes I have used iperf for test.
    With cpview I can't see none CPU with 100% usage.

  4. #4
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Quote Originally Posted by juvann View Post
    Yes SecureXL is enabled and yes I have used iperf for test.
    With cpview I can't see none CPU with 100% usage.
    Please provide output of the following commands run on the firewall from expert mode:

    fwaccel stat
    fwaccel stats -s
    grep -c ^processor /proc/cpuinfo
    /sbin/cpuinfo
    fw ctl affinity -l -r
    sim affinity -l
    netstat -ni
    fw ctl multik stat
    fw ctl multik get_mode
    cpstat os -f multi_cpu -o 1
    free -m
    enabled_blades
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  5. #5
    Join Date
    2006-09-26
    Posts
    3,193
    Rep Power
    16

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Quote Originally Posted by juvann View Post
    Yes SecureXL is enabled and yes I have used iperf for test.
    With cpview I can't see none CPU with 100% usage.
    I assume that you also have this command on your Cisco switches:

    port-channel load-balance src-dst-ip

    On Cisco newer switches, you also see this:

    port-channel load-balance vlan-src-dst-mixed-ip-port

  6. #6
    Join Date
    2018-10-06
    Posts
    6
    Rep Power
    0

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Thank you to all for suggestions! I solved my issue.
    I enabled multi queue on ethernet in bond interfaces, and now I got all 2Gpbs with multi connections.

    I enabled 4 queue for driver igb
    cpmq set rx_num igb 4

    Enabled queue for interfaces
    cpmq set -f

    and after reboot I have this situation:

    Code:
    # cpmq get -v
    
    Active igb interfaces:
    Mgmt [Off]
    Sync [Off]
    eth1 [On]
    eth1-01 [On]
    eth1-05 [On]
    eth5 [On]
    eth8 [Off]
    
    The rx_num for igb is: 4
    
    multi-queue affinity for igb interfaces:
    CPU | TX | Vector                      | RX Bytes
    -------------------------------------------------------------
    0   | 0  | eth1-01-TxRx-0 (186)        | 30661660675
        |    | eth1-05-TxRx-0 (91)         |
        |    | eth1-TxRx-0 (179)           |
        |    | eth5-TxRx-0 (227)           |
    1   | 2  | eth1-01-TxRx-2 (202)        | 19214905571
        |    | eth1-05-TxRx-2 (107)        |
        |    | eth1-TxRx-2 (195)           |
        |    | eth5-TxRx-2 (52)            |
    2   | 0  |                             |
    3   | 2  |                             |
    4   | 0  |                             |
    5   | 2  |                             |
    6   | 0  |                             |
    7   | 2  |                             |
    8   | 1  | eth1-01-TxRx-1 (194)        | 26115216389
        |    | eth1-05-TxRx-1 (99)         |
        |    | eth1-TxRx-1 (187)           |
        |    | eth5-TxRx-1 (235)           |
    9   | 3  | eth1-01-TxRx-3 (210)        | 683483583031
        |    | eth1-05-TxRx-3 (115)        |
        |    | eth1-TxRx-3 (203)           |
        |    | eth5-TxRx-3 (60)            |
    10  | 1  |                             |
    11  | 3  |                             |
    12  | 1  |                             |
    13  | 3  |                             |
    14  | 1  |                             |
    15  | 3  |                             |

  7. #7
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,248
    Rep Power
    14

    Default Re: Appliances 5900 R80.10 and bonding interfaces limited throughput

    Quote Originally Posted by juvann View Post
    Thank you to all for suggestions! I solved my issue.
    I enabled multi queue on ethernet in bond interfaces, and now I got all 2Gpbs with multi connections.

    I enabled 4 queue for driver igb
    cpmq set rx_num igb 4

    Enabled queue for interfaces
    cpmq set -f

    and after reboot I have this situation:

    Code:
    # cpmq get -v
    
    Active igb interfaces:
    Mgmt [Off]
    Sync [Off]
    eth1 [On]
    eth1-01 [On]
    eth1-05 [On]
    eth5 [On]
    eth8 [Off]
    
    The rx_num for igb is: 4
    
    multi-queue affinity for igb interfaces:
    CPU | TX | Vector                      | RX Bytes
    -------------------------------------------------------------
    0   | 0  | eth1-01-TxRx-0 (186)        | 30661660675
        |    | eth1-05-TxRx-0 (91)         |
        |    | eth1-TxRx-0 (179)           |
        |    | eth5-TxRx-0 (227)           |
    1   | 2  | eth1-01-TxRx-2 (202)        | 19214905571
        |    | eth1-05-TxRx-2 (107)        |
        |    | eth1-TxRx-2 (195)           |
        |    | eth5-TxRx-2 (52)            |
    2   | 0  |                             |
    3   | 2  |                             |
    4   | 0  |                             |
    5   | 2  |                             |
    6   | 0  |                             |
    7   | 2  |                             |
    8   | 1  | eth1-01-TxRx-1 (194)        | 26115216389
        |    | eth1-05-TxRx-1 (99)         |
        |    | eth1-TxRx-1 (187)           |
        |    | eth5-TxRx-1 (235)           |
    9   | 3  | eth1-01-TxRx-3 (210)        | 683483583031
        |    | eth1-05-TxRx-3 (115)        |
        |    | eth1-TxRx-3 (203)           |
        |    | eth5-TxRx-3 (60)            |
    10  | 1  |                             |
    11  | 3  |                             |
    12  | 1  |                             |
    13  | 3  |                             |
    14  | 1  |                             |
    15  | 3  |                             |
    Great, thanks for the follow-up.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

Similar Threads

  1. How many CPU cores 5900 has?
    By blason in forum Firewall Blade
    Replies: 3
    Last Post: 2018-05-11, 09:05
  2. Bonding interfaces with 2 Switch HP 5406ZL
    By networkingkool in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 2
    Last Post: 2013-06-20, 02:38
  3. using bonding interfaces in Virtual Systems
    By zenitt in forum Check Point VSX/VSX-1 Appliances
    Replies: 1
    Last Post: 2012-10-04, 05:53
  4. Interface Bonding on UTM Appliances not supported!
    By Yasushi Kono in forum Check Point UTM-1 Appliances
    Replies: 12
    Last Post: 2012-09-08, 08:28
  5. Power-1 11067 2x10Gb interfaces loadbalancing (8021ad) bonding
    By pekka.kovesjarvi in forum Check Point Power-1 Appliances
    Replies: 4
    Last Post: 2012-03-27, 09:30

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •