大网数据广告活动
服务器运维问答

Small writes to SMB network share are slow on Windows, fast over CIFS Linux mount

· 52CE 我爱测

I have been struggling to fix a performance problem with a SMB/CIFS share when performing small writes. First, let me describe my current network setup: Server Synology DS215j (with SMB3 support enabled) Clients (same computer du…

问题原文

原帖链接 · mevatron · CC BY-SA 3.0

I have been struggling to fix a performance problem with a SMB/CIFS share when performing small writes.

First, let me describe my current network setup:

Server

  • Synology DS215j (with SMB3 support enabled)

Clients (same computer dual-booted wired Gig-E)

  • Ubuntu 14.04.5 LTS, Trusty Tahr
  • Windows 8.1

smb.conf

[global]
    printcap name=cups
    winbind enum groups=yes
    include=/var/tmp/nginx/smb.netbios.aliases.conf
    socket options=TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
    security=user
    local master=no
    realm=*
    passdb backend=smbpasswd
    printing=cups
    max protocol=SMB3
    winbind enum users=yes
    load printers=yes
    workgroup=WORKGROUP

I'm currently testing the small write performance with the following program written in C++ (on GitHub here):

#include <iostream>
#include <fstream>
#include <sstream>

using namespace std;

int main(int argc, char* argv[])
{
    ofstream outFile(argv[1]);
    for(int i = 0; i < 1000000; i++)
    {
        outFile << "Line #" << i << endl;   
    }

    outFile.flush();
    outFile.close();
    return 0;
}

Linux mount configuration:

//192.168.1.10/nas-main on /mnt/nas-main type cifs (rw,noexec,nodev)

Program run-time on Linux (peaks network output at ~100Mbps):

$ time ./nas-write-test /mnt/nas-main/home/will/test.txt

real    0m0.965s
user    0m0.148s
sys 0m0.672s

PCAP snapshot showing chunking of many lines into a single TCP packet:

Program run-time on Windows as measured by PowerShell:

> Measure-Command {start-process .\nas-write-test.exe -argumentlist "Z:\home\will\test-win.txt" -wait}


Days              : 0
Hours             : 0
Minutes           : 9
Seconds           : 29
Milliseconds      : 316
Ticks             : 5693166949
TotalDays         : 0.00658931359837963
TotalHours        : 0.158143526361111
TotalMinutes      : 9.48861158166667
TotalSeconds      : 569.3166949
TotalMilliseconds : 569316.6949

PCAP snapshot on Windows showing single line per SMB Write Request:

This same program takes about 10 minutes (~2.3Mbps) on Windows. Obviously, the Windows PCAP shows a very noisy SMB conversation with very low payload efficiency.

Are there any settings on Windows that can improve small write performance? It seems from looking at packet captures that Windows doesn't buffer the writes properly and immediately sends out the data one line at a time. Whereas, on Linux, the data is heavily buffered and thus has far superior performance. Let me know if the PCAP files would be helpful, and I can find a way to upload them.

Update 10/27/16:

As mentioned by @sehafoc, I reduced the Samba servers max protocol setting to SMB1 with the following:

max protocol=NT1

The above setting resulted in the exact same behavior.

I also removed the variable of Samba by creating a share on another Windows 10 machine, and it also exhibits the same behavior as the Samba server, so I'm beginning to believe this is a write caching bug with Windows clients in general.

Update: 10/06/17:

Full Linux packet capture (14MB)

Full Windows packet capture (375MB)

Update: 10/12/17:

I also setup an NFS share and Windows does write with no buffering for this as well. So, it's definitely an underlying Windows client issue as far as I can tell, which is definitely unfortunate :-/

Any help would be appreciated!

已采纳解答

原帖链接 · Doug · CC BY-SA 3.0

The C++ endl is defined to output '\n' followed by a flush. flush() is an expensive operation, so you should generally avoid using endl as your default end of line as it can create exactly the performance issue you are seeing (and not just with SMB, but with any ofstream with an expensive flush including local spinning rust or even the latest NVMe at some ridiculously high rate of output).

Replacing endl with "\n" will fix the performance above by allowing the system to buffer as intended. Except some libraries may flush on "\n", in which case you have more headaches (see https://stackoverflow.com/questions/21129162/tell-endl-not-to-flush for a solution overriding the sync() method).

Now to complicate things, flush() is only defined for what happens within the library buffers. The effect of flush on operating system, disk, and other external buffers is not defined. For Microsoft.NET "When you call the FileStream.Flush method, the operating system I/O buffer is also flushed." (https://msdn.microsoft.com/en-us/library/2bw4h516(v=vs.110).aspx) This makes flush particularly expensive for Visual Studio C++ as it will round-trip the write all the way out to the physical media at the far end of your remote server as you are seeing. GCC on the other hand says "A last reminder: there are usually more buffers involved than just those at the language/library level. Kernel buffers, disk buffers, and the like will also have an effect. Inspecting and changing those are system-dependent." (https://gcc.gnu.org/onlinedocs/libstdc++/manual/streambufs.html) Your Ubuntu traces would seem to indicate that the operating system / network buffers are not flushed by the library flush(). System dependant behaviour would be all the more reason to avoid endl and flushing excessively. If you are using VC++ you might try switching to a Windows GCC derivative to see how the system dependant behaviours react, or alternatively using Wine to run the Windows executable on Ubuntu.

More generally you need to think about your requirements to determine if flushing every line is appropriate or not. endl is generally suitable for interactive streams such as the display (we need the user to actually see our output, and not in bursts), but generally not suitable for other types of streams including files where the flushing overhead can be significant. I've seen apps flush on every 1 and 2 and 4 and 8 byte writes... it's not pretty to see the OS grind millions of IOs to write a 1MB file.

As an example a log file may need flushing every line if you are debugging a crash because you need to flush the ofstream before the crash occurs; while another log file may not need flushing every line if it is just producing verbose informational logging that is expected to flush automatically before the application terminates. It need not be either/or as you could derive a class with a more sophisticated flush algorithm to suit specific requirements.

Compare your case with the contrasting case of people who need to ensure their data is completely persisted to disk and not vulnerable in an operating system buffer (https://stackoverflow.com/questions/7522479/how-do-i-ensure-data-is-written-to-disk-before-closing-fstream).

Note that as written, outFile.flush() is superfluous as it flushes an already flushed ofstream. To be pedantic, you should have used endl alone or preferably "\n" with outFile.flush() but not both.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-10-12)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。