问题原文
原帖链接 · davidparks21 · CC BY-SA 3.0
I'm downloading a large file over http via wget, 1.2TB. The download takes about a week and has contained corruptions twice now (failed md5 check, which takes days to run by itself).
Is there a good way to validate the file piecemeal over http using say curl? Or to break it into separate blocks such that I could identify a specific bad block and redownload just that section?
The file is a tar archive, so I believe corruptions per block could be identified sequentially during unpacking.
已采纳解答
原帖链接 · ThoriumBR · CC BY-SA 3.0
On the server side, you can use dd and md5sum to checksum each chunk of the file:
#!/bin/bash
FILENAME="$1"
FILESIZE=`stat --printf="%s" $FILENAME`
CHUNKSIZE=536870912 # 512MB
CHUNKNUM=0
while ! grep -q 'cannot skip' hash.log 2> /dev/null ; do
dd if=$FILENAME bs=$CHUNKSIZE skip=$CHUNKNUM count=1 2> hash.log | md5sum >> $FILENAME.md5
CHUNKNUM=$(( CHUNKNUM + 1 ))
done
rm hash.log
You will be left with a single $FILENAME.md5 file with all chunk hashes.
You can now download that large file and the checksums, run this script on the file and compare the hashes. If any piece gets a mismatched hash, you can use curl to download only part of the file (if the server supports RANGE) and patch the file with dd.
For example, if the chunk 2 gets a hash mismatch:
curl -s -r 536870912-1073741824 | dd of=somelargetarfile.tar seek=536870912 conv=notrunc
This will download the chunk 2, and patch the large tar file with it.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
