CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 12 of 12

Thread: Amalgamating / Joining Bonds

  1. #1
    Join Date
    2008-07-07
    Posts
    97
    Rep Power
    11

    Default Amalgamating / Joining Bonds

    Hi All

    I currently have bonds configured on a HA pair of firewalls running R77.10. I am needing to join two bonds together as both terminate on the same Nexus 6K so makes more sense to provide a single 6 Gbps link instead of a 4 Gbps (Server) and 2 Gbps (DMZ)

    Bond 2 carries server networks
    Bond 3 carries DMZ traffic

    Bond 3 has only 3 sub interfaces ( Bond3.60 etc) so it will be easier to move over to Bond 2 but unsure how best to achieve this without breaking everything? Any ideas / hints?

    Regards

    Neil

  2. #2
    Join Date
    2006-09-26
    Posts
    3,163
    Rep Power
    16

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by Neilharrison_253 View Post
    Hi All

    I currently have bonds configured on a HA pair of firewalls running R77.10. I am needing to join two bonds together as both terminate on the same Nexus 6K so makes more sense to provide a single 6 Gbps link instead of a 4 Gbps (Server) and 2 Gbps (DMZ)

    Bond 2 carries server networks
    Bond 3 carries DMZ traffic

    Bond 3 has only 3 sub interfaces ( Bond3.60 etc) so it will be easier to move over to Bond 2 but unsure how best to achieve this without breaking everything? Any ideas / hints?

    Regards

    Neil
    The way Ether-Channel work is 1, 2, 4 and 8 (power of 2). You can not have odd numbers on Ether-Channel, like 3, 5, 6 or 7.

    This is especially with Cisco. It might work; however, when you have issues, Cisco will not support it
    Last edited by cciesec2006; 2015-07-06 at 21:14.

  3. #3
    Join Date
    2008-07-07
    Posts
    97
    Rep Power
    11

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by cciesec2006 View Post
    The way Ether-Channel work is 1, 2, 4 and 8 (power of 2). You can not have odd numbers on Ether-Channel, like 3, 5, 6 or 7.

    This is especially with Cisco. It might work; however, when you have issues, Cisco will not support it
    Hi

    Thanks for the response. I have looked further into this as there seems to be a precedence around how the traffic is split over the interfaces where odd number interfaces will not have an even split of traffic so have decided to re-evaluate the change. Thanks for the heads up

    Regards

  4. #4
    Join Date
    2008-07-07
    Posts
    97
    Rep Power
    11

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by cciesec2006 View Post
    The way Ether-Channel work is 1, 2, 4 and 8 (power of 2). You can not have odd numbers on Ether-Channel, like 3, 5, 6 or 7.

    This is especially with Cisco. It might work; however, when you have issues, Cisco will not support it
    Another question :)

    We have an 8 port copper card / 4 port copper card / 4 port fibre card. Is there a best practice around how interfaces and bonded together on the same line card / different line cards?

    Thanks

  5. #5
    Join Date
    2006-09-26
    Posts
    3,163
    Rep Power
    16

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by Neilharrison_253 View Post
    Another question :)

    We have an 8 port copper card / 4 port copper card / 4 port fibre card. Is there a best practice around how interfaces and bonded together on the same line card / different line cards?

    Thanks
    I usually setup bonding on interfaces from different card. That way, you can increase performance due to the bus on the motherboard and reduce the probability that the whole card might die. In case the whole card die, you still have redundancy.

  6. #6
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,229
    Rep Power
    13

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by cciesec2006 View Post
    I usually setup bonding on interfaces from different card. That way, you can increase performance due to the bus on the motherboard and reduce the probability that the whole card might die. In case the whole card die, you still have redundancy.
    Apparently there was an unofficial recommendation floating around that advised always using an even number of physical interfaces when creating a bond under Gaia. Using an odd number would result in degraded performance under heavy load (probably due to suboptimal balancing of traffic amongst the physical interfaces). That was all the information I was able to get about this issue. The good news however is that this was a Intel driver issue that was fixed in R77.30 when the driver was updated.

    While researching my book I also attempted to figure out if it was better from a performance perspective to have physical interfaces of the same bond on the same NIC card or on different NIC cards. Couldn't ever find a firm answer to this other than "it depends" (even from Check Point) so cciesec2006's recommendation to have physical interfaces of the same bond spread across different NIC cards to improve reliability is as good as any. Would be curious if anyone else has real-world insights into this.
    Last edited by ShadowPeak.com; 2015-07-16 at 09:33.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  7. #7
    Join Date
    2006-09-26
    Posts
    3,163
    Rep Power
    16

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by ShadowPeak.com View Post
    cciesec2006's recommendation to have physical interfaces of the same bond spread across different NIC cards to improve reliability is as good as any.
    by having physical interfaces on the same bond spread across different NIC cards not only improve reliability, it also increases the performance.

    Even on the Checkpoint power-1 appliances, in order to achieve the desired throughput, you can only use a single 10Gig interface of a dual 10 Gig NIC card. Not me saying it but checkpoint saying it. By bonding two different 10Gig interfaces on the same NIC card, you essentially cut down the throughput in half. I've tested and validated it in the lab.

  8. #8
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,229
    Rep Power
    13

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by cciesec2006 View Post
    by having physical interfaces on the same bond spread across different NIC cards not only improve reliability, it also increases the performance.

    Even on the Checkpoint power-1 appliances, in order to achieve the desired throughput, you can only use a single 10Gig interface of a dual 10 Gig NIC card. Not me saying it but checkpoint saying it. By bonding two different 10Gig interfaces on the same NIC card, you essentially cut down the throughput in half. I've tested and validated it in the lab.
    I can definitely see that limitation coming into play at 10Gbps speeds; just because the line speed of 2 individual ports on a single NIC is 10Gbps does not mean the NIC (or its connection to the system bus/backplane) can move 40Gbps of traffic in that case assuming full duplex operation.

    Have you noticeably encountered this limitation at 1Gbps speeds? I imagine it is still possible, guess it would highly depend on the density of ports per NIC so you'd be far more likely to hit the wall with a quad card than a dual-port NIC or a single-port NIC. I've seen NIC ports start to run out of gas at 1 Gbps line speeds once about 925-950 Mbps is reached.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  9. #9
    Join Date
    2006-09-26
    Posts
    3,163
    Rep Power
    16

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by ShadowPeak.com View Post
    Have you noticeably encountered this limitation at 1Gbps speeds? I imagine it is still possible, guess it would highly depend on the density of ports per NIC so you'd be far more likely to hit the wall with a quad card than a dual-port NIC or a single-port NIC. I've seen NIC ports start to run out of gas at 1 Gbps line speeds once about 925-950 Mbps is reached.
    On older servers, you will run into limitations as well. Based on experience with IBM x3650 generation 1, the bus speed can only handle a maximum of 2.5Gbps. Therefore, if you have a quad-port NIC card, you will not be able to push 4Gbps because the bus can only 2.5Gbps.

    yes, the 1Gig port will max out at 950Mbps...

  10. #10
    Join Date
    2008-07-07
    Posts
    97
    Rep Power
    11

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by ShadowPeak.com View Post
    Apparently there was an unofficial recommendation floating around that advised always using an even number of physical interfaces when creating a bond under Gaia. Using an odd number would result in degraded performance under heavy load (probably due to suboptimal balancing of traffic amongst the physical interfaces). That was all the information I was able to get about this issue. The good news however is that this was a Intel driver issue that was fixed in R77.30 when the driver was updated.

    While researching my book I also attempted to figure out if it was better from a performance perspective to have physical interfaces of the same bond on the same NIC card or on different NIC cards. Couldn't ever find a firm answer to this other than "it depends" (even from Check Point) so cciesec2006's recommendation to have physical interfaces of the same bond spread across different NIC cards to improve reliability is as good as any. Would be curious if anyone else has real-world insights into this.
    Hi ShadowPeak

    Quick Plug, purchased your book recently and very impressed. Has made troubleshooting and performance changes much easier to understand :) Appreciated the time and effort that went into this

    However, I have a question around this original topic. I have recently made changes to a Bond interface off the back of this conversation and increased a bond from 3 to 4 interfaces however it's not going well. I am getting a continuous and significant growth on RX-ERR on the newly added bond interface and run out of things to test / change. To make things more complicated when I fail over to the second firewall the same RX-ERR on the new interface starts to climb. Bond 1 comprises of eth1-01/02/03/04

    1. Both SFP's on the Cisco 6880 have been changed twice. First time with a used one and the second a brand new SFP (Same as on the other 3 interfaces on the 6880)
    2. Cable has also been changed twice
    3. Buffer size is the same on all interfaces on the bond. Was 256 and recently increased to 1024
    4. Speed / Duplex is set to Auto on both firewall and Cisco 6880 and connecting at 1000 / Full Duplex
    5. No errors on the Cisco interface side
    6. Both firewalls have been rebooted
    7. Multiple policy updates have taken place

    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    bond1 1500 0 1045996878 139265 0 0 1180130410 0 0 0 BMmRU
    eth1-01 1500 0 282165137 5 0 0 328352105 0 0 0 BMsRU
    eth1-02 1500 0 242261638 0 0 0 221606436 0 0 0 BMsRU
    eth1-03 1500 0 184464062 0 0 0 218884569 0 0 0 BMsRU
    eth1-04 1500 0 337106041 139260 0 0 411287300 0 0 0 BMsRU

    Do you have any additional suggestion?

    Regards

    Neil

  11. #11
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,229
    Rep Power
    13

    Default Re: Amalgamating / Joining Bonds

    Quote Originally Posted by Neilharrison_253 View Post
    Hi ShadowPeak

    Quick Plug, purchased your book recently and very impressed. Has made troubleshooting and performance changes much easier to understand :) Appreciated the time and effort that went into this
    Thanks for the plug, I responded to your interface error question in another thread. Probably best to keep the discussion of that issue there.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  12. #12
    Join Date
    2012-03-13
    Posts
    6
    Rep Power
    0

    Default Re: Amalgamating / Joining Bonds

    It is generally recommended to have 2/4/8 links in a bond. It ensures (in a perfect world) that the links are equally loaded because this is how Etherchannel works: it takes a hash from some packet fields (s-mac, d-mac, ip, etc, can be tuned) and produces a value of 1 to 8. Those possible 8 values are distributed among the physical links, hash value determines the link that traffic should be transmitted from.
    Only 2, 4 or 8 links would give us equal distribution of 8 possible "buckets". If we had, for example, 5 physical links, distribution would be (1,2) - (3,4) - (5,6) - (7) - (8) , meaning the first 3 interfaces would transmit twice as much traffic as the last 2. Bonding 10 links would give us 1-2-3-4-5-6-7-8-0-0 distribution, the last 2 links would not have anything to transmit.
    It also means that no link bonding will ever help with an "elephant" flow as it's packets will always generate the same hash and hence select the same physical link ignoring all the others.
    Last edited by Relicto; 2015-07-30 at 11:48.

Similar Threads

  1. Pb joining public IP of VPN peers
    By caro06 in forum IPsec VPN Blade (Virtual Private Networks)
    Replies: 4
    Last Post: 2008-10-06, 12:17
  2. ip: joining multicasts failed (3) on ce1 interface
    By pop_alex in forum Miscellaneous
    Replies: 0
    Last Post: 2006-04-14, 05:01

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •