大网数据广告活动
服务器运维问答

Can I validate a large file download piecemeal over http

· 52CE 我爱测

I'm downloading a large file over http via wget , 1.2TB. The download takes about a week and has contained corruptions twice now (failed md5 check, which takes days to run by itself). Is there a good way to validate the file piec…

问题原文

原帖链接 · davidparks21 · CC BY-SA 3.0

I'm downloading a large file over http via wget, 1.2TB. The download takes about a week and has contained corruptions twice now (failed md5 check, which takes days to run by itself).

Is there a good way to validate the file piecemeal over http using say curl? Or to break it into separate blocks such that I could identify a specific bad block and redownload just that section?

The file is a tar archive, so I believe corruptions per block could be identified sequentially during unpacking.

已采纳解答

原帖链接 · ThoriumBR · CC BY-SA 3.0

On the server side, you can use dd and md5sum to checksum each chunk of the file:

#!/bin/bash
FILENAME="$1"
FILESIZE=`stat --printf="%s" $FILENAME`
CHUNKSIZE=536870912 # 512MB
CHUNKNUM=0
while ! grep -q 'cannot skip' hash.log 2> /dev/null ; do
    dd if=$FILENAME bs=$CHUNKSIZE skip=$CHUNKNUM count=1 2> hash.log | md5sum >> $FILENAME.md5
    CHUNKNUM=$(( CHUNKNUM + 1 ))
done
rm hash.log

You will be left with a single $FILENAME.md5 file with all chunk hashes.

You can now download that large file and the checksums, run this script on the file and compare the hashes. If any piece gets a mismatched hash, you can use curl to download only part of the file (if the server supports RANGE) and patch the file with dd.

For example, if the chunk 2 gets a hash mismatch:

curl -s -r 536870912-1073741824 | dd of=somelargetarfile.tar seek=536870912 conv=notrunc

This will download the chunk 2, and patch the large tar file with it.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2017-10-24)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。