大网数据广告活动
网站运营与故障问答

Can file encoding change when FTP is used?

· 52CE 我爱测

I'm developing a web application which downloads a number of web pages using PHP curl. It then uses diff to compare the files as they change each day. I reported a problem a few weeks back where seemingly identical files were bei…

问题原文

原帖链接 · Andy · CC BY-SA 3.0

I'm developing a web application which downloads a number of web pages using PHP curl. It then uses diff to compare the files as they change each day.

I reported a problem a few weeks back where seemingly identical files were being flagged by diff as being different: https://stackoverflow.com/questions/42552239/different-versions-of-diff-giving-mixed-results-when-comparing-2-identical-fil

The answer to the above was that if diff was used with the -w flag it ignores whitespace.

However, I've now noticed a separate problem. If I download one of the files I'm comparing, and re-upload (overwrite) it through an FTP client, the output changes.

For example: Compare file1.html against file2.html with diff file1.html file2.html it may give output such as

12159,12161c12159,12161
<   
< 
< 
---
>   
> 
> 
12163,12172c12163,12172
< 
< 
< 
< 
< 
< 
< 
< 
< 
< 
---

However, if I download file2.html to my desktop and re-upload it through FTP, diff without the -w flag reports there being no differences at all i.e. it's now saying the files are identical.

I've tried to check the encoding of the file using file -bi file2.html but it's reported the same before and after upload through FTP. The encoding is text/html; charset=us-ascii

If the encoding is no different and the file contents have not been modified, how is re-uploading the file through FTP changing anything?? I've tried it using FileZilla and also through Netbeans.

I'm using macOS Sierra locally and the remote server is Apache 2/PHP 7/centOS.

已采纳解答

原帖链接 · MrWhite · CC BY-SA 3.0

You are probably seeing a difference in line-endings. When transferring a file in ASCII/Text mode (as opposed to "Binary mode") then most FTP clients will convert/normalise line-endings to the OS being transferred to.

On Classic Mac OS (9.x and earlier) the line-ending char is simply \r (ASCII 13), on Mac OS X this changed to \n (ASCII 10), on Linux it is \n (ASCII 10). And Windows is \r\n or ASCII 13+10. (Thanks @8bittree for the Mac correction.)

So, when downloading from one OS to another all line-endings are silently converted. The conversion is reversed when uploaded. (However, as noted in @Joshua's answer this can result in corruption, depending on the file's character encoding and specific characters contained in the file.) If there is a mishmash of line-endings then it's possible the FTP software is normalising/fixing the line-endings. This would explain why downloading and then uploading the file results in a "different" file to what was originally on the server (ie. it is "fixed"). Or it is reverting a previously miss-converted file? However, the EOL-conversion may not be so intelligent and you can just end up with either double spaced lines or missing line breaks altogether (ie. mildly corrupted).

By default, most FTP clients are set to "Auto" transfer mode and have a list of known file types to transfer in ASCII/Text mode. Other file types are transferred in "Binary" mode. If you are transferring between the same OS, or you wish to transfer with no conversion, then you should use "Binary" mode only.

Ordinarily, the FTP software will not change the character encoding of the transferred file unless the source/target operating systems use a very different character encoding with which to represent text files. As @KeithDavies noted in comments, one such example is when downloading from a mainframe, that uses EBCDIC, to a local Windows machine. EBCDIC is not supported natively by Windows, so a conversion is required to convert this to ASCII. Again, transferring in "Binary mode" avoids any such conversion. (Thanks to @KeithDavies for the note regarding character encoding.)

The answer to the above was that if diff was used with the -w flag it ignores whitespace.

Yes, line-endings (whitespace) are ignored in the comparison.

If I download one of the files I'm comparing, and re-upload (overwrite) it through an FTP client, the output changes.

If there was a mixture of line-endings in the original file then downloading and re-uploading in ASCII mode could well "fix" the inconsistent line-endings. So, the files are now "the same".

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2017-03-16)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。