大网数据广告活动
服务器运维问答

nic bonding very slow

· 52CE 我爱测

I have 2 dl380 g9 servers which feature 2 nics with 4 ports each running debian buster. I have connected 7x 0.50m cat6e patch cables directly server-to-server and set the /etc/network/interfaces on both servers(with minor differe…

问题原文

原帖链接 · Meni · CC BY-SA 4.0

I have 2 dl380 g9 servers which feature 2 nics with 4 ports each running debian buster. I have connected 7x 0.50m cat6e patch cables directly server-to-server and set the /etc/network/interfaces on both servers(with minor difference in the ip addresses) as:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

auto eno49
iface eno49 inet manual

auto eno50
iface eno50 inet manual

auto eno51
iface eno51 inet manual

auto eno52
iface eno52 inet manual

auto bond0
iface bond0 inet static
        address 10.10.10.11/24
        bond-slaves eno2 eno3 eno4 eno49 eno50 eno51 eno52
        bond-miimon 100
        bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.234/16
        gateway 10.0.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

the plan was to use the bonded connection for zfs replication between the nodes. The issue is that the thourhgput i get, is limited to 2.25gbps(value taken from nload). the wierd part of the situation is that if i bond 3 ports insetad of 7.......i get again 2.25gbps. it's like the 4 ports are not utilised. any idea how can i diagnose the issue?

已采纳解答

原帖链接 · suprjami · CC BY-SA 4.0

Don't use round-robin: bond-mode balance-rr

This mode will cause a lot of out-of-order TCP traffic and limit throughput like you are experiencing.

Change the bond to bond-mode balance-xor which balances a single flow out each member of the bond. So a single stream will go at 1x NIC speed (1 Gbps) but you can run many flows and achieve full speed of all NICs (7 Gbps).

If the traffic is within the same subnet, then the default bond-xmit-hash-policy layer2 may be fine, this performs the load balancing based on MAC addresses.

if all your traffic goes through a default gateway, then look at setting bond-xmit-hash-policy layer2+3 or bond-xmit-hash-policy layer3+4 which balance on IP address and TCP/UDP port respectively. This will allow you to balance flows to multiple hosts, or multiple sessions to the same host, or just improve the balancing algorithm.

I presume your switch is correctly configured with a port-channel or EtherChannel or other sort of Link Aggregation Group. This is needed for your existing balance-rr mode anyway. The switch will have its own load balancing policy back in (similar to layer 2/3/4 above), so make sure the switch is configured with a useful policy too.

Make sure you are running irqbalance so multiple CPUs can receive traffic streams at the same time. If you don't spread IRQs, then all traffic will be handled by CPU core 0 which becomes a bottleneck.

However, if your aim is to achieve a single 7Gbps TCP stream, this is not a feature that either bonding or teaming offer, and is not how link aggregation works. If you want one single faster stream then upgrade to 10 Gbps or faster.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-04-26)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。