大网数据广告活动
服务器运维问答

How to access files from a local server with infiniband without using IP

· 52CE 我爱测

I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me. I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are all connected to a Voltaire gri…

问题原文

原帖链接 · tot18 · CC BY-SA 4.0

I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me.

I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are all connected to a Voltaire grid director 4036 infiniband switch thats running the subnet manager. For simplicity's sake I will call the computers s1-s4. I want s2, s3 and s4 to be able to access and download files from s1 without using IP since I've read that removing the use of IP's can improve performance by up to 20%(correct me on this if I'm wrong). Is it even possible to make file transfers without the use of IP?

Also, since I'm new to this site feel free to tell me if I should have given more information or was unclear about something. If you think I have misunderstood something then you can also feel free to correct me on it.

Thanks

已采纳解答

原帖链接 · Vinícius Ferrão · CC BY-SA 4.0

tl;dr: look and stick with NFSoRDMA.

That's some points that needs to be clear. Considering that you're using RHEL 7 as basis you may want to serve files over NFS from s1 to s2, s3 and s4.

Your right about running IP over Infiniband is problem due to performance and latency issues, not to mention the high CPU load, since basically there's no IP offload on Infiniband, everything is done in software. When running pure IPoIB workload you'll probably have a server with 99% wasted CPU cycles on ksoftirq.

With that in mind, there's another issue: NFS, and also other file services, relies on IP networks to properly function.

What you want is an RDMA enabled protocol for file sharing. Usually they still depends on IP, but only for initial handshake, after the connection is stablished, they got upgraded to RDMA, and in your case, using Infiniband. There's also other fabrics that implements RDMA: RoCE and Omni-Path for instance.

For NFS specifically you should look for NFSoRDMA (NFS over RDMA). You didn't say which hardware do you have, but if it's Mellanox you'll need to install MLNX OFED (probably the LTS 4.9 version to support old hardware) and enable NFSoRDMA. I'm not sure if NFSoRDMA is available on the INBOX RHEL 7 OFED implementation. On EL8 is available AFAIK.

You'll still need an up and running IPoIB interface with an IP address, usually ib0, but with proper configuration NFS will only transfer control messages over IP while the bulk traffic will be over RDMA.

Other filesystems, for example, BeeGFS operates the same way. The initialize over IP and upgrades to RDMA. Gluster was an option in the past but RDMA has been removed from it.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-03-31)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。