大网数据广告活动
服务器运维问答

Infiniband drivers : OFED or distro included?

· 52CE 我爱测

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome ! We are currently using Mellanox OFED drivers, but our infiniband cards are old and not recognize…

问题原文

原帖链接 · nirnaeth · CC BY-SA 4.0

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome !

We are currently using Mellanox OFED drivers, but our infiniband cards are old and not recognized by the latest MOFED drivers. So I'm wondering why not to use distribution shipped drivers (running CentOS7).

What difference will that make to use one or another ? Should I expect any performance decrease ?

thx

已采纳解答

原帖链接 · Vinícius Ferrão · CC BY-SA 4.0

By not using the vendor OFED distribution, in this case Mellanox OFED you should expect not only a performance penalty but lack of features and a lot of stability issues.

Infiniband is not rock solid as Ethernet is, the main goal of Infiniband is to provide a low latency fabric, not only a high throughput network as everybody usually think.

The inbox driver (that's how Mellanox calls the OFED distribution shipped on the distribution) is unreliable at best, and if you're running cards older than Connect-X4 you'll have a bad time when running IPoIB if needed, just keeping it enabled will eventually lead to kernel panics. Performance is just bad and the network is unreliable.

There are some alternatives, first of all there's the MLNX OFED 4.9 which is an LTS release that support older cards like the Connect-X3. I would stick with it since it's supported and will be supported for a long time.

The difference is the support for the following hardware and technology:

  • ConnectX-3 Pro
  • ConnectX-3
  • Connect-IB
  • RDMA experimental verbs library (mlnx_lib)

Download it from here: https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed

If the LTS version of Mellanox OFED does not fit you, another solution is moving to Oracle Linux, adopt UEK (Unbreakable Enterprise Kernel) and consume its RDMA distribution. At least Oracle test this OFED release, their Exadata product uses it. There's documentation available here: https://docs.oracle.com/en/operating-systems/uek/6/relnotes6.2/ol_instav.html#uek6_install_rdma

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-01-06)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。