大网数据广告活动
服务器运维问答

Which EC2 Instance to hash a large amount of 25gb files

· 52CE 我爱测

I have an EBS (elastic block storage) on AWS with large amount of 25GB files (1000). I would like to calculate a SHA256 sum for each file. Which EC2 instance would be the best for such task (cpu optimized? many cores? storage opt…

问题原文

原帖链接 · ECII · CC BY-SA 4.0

I have an EBS (elastic block storage) on AWS with large amount of 25GB files (1000). I would like to calculate a SHA256 sum for each file.

Which EC2 instance would be the best for such task (cpu optimized? many cores? storage optimized?). Can I somehow hash the files in parallel? How can I optimize performance?

已采纳解答

原帖链接 · MLu · CC BY-SA 4.0

You will be struggling with EBS throughput.

Smaller m5 / m5a /m6g instances have up to 4,750 Mbps EBS throughput = ca 600 MB/s max. Larger instances like m5.24xlarge can go up to 19,000 Mbps or 2.4 GB/s. But only if your EBS volume can handle it, ie it will probably have to be IOPS-optimised volume (io1 type) to sustain this throughput.

That means your 25TB of data (1000 files x 25 GB each) can be read from the EBS in somewhere between 3 hours and 12 hours in ideal conditions. In reality it will probably be slower. And that's just reading the files.

The CPU doesn't really matter - any CPU can do sha256 at this speed so use a cheaper architecture like M6g (ARM based) and if it comes with multiple cores (e.g. m6g.xlarge with 4 CPU cores) you can hash 4 files in parallel. Though that may not reduce the time 4x as you would expect because of the EBS throughput bottleneck.

I would buy m6g.4xlarge as a Spot Instance and expect it will take a day to hash the files. If your EBS is in a region where M6g (ARM) is not available use M5a (AMD) - it's cheaper than the M5 (Intel) and still powerful enough for the hashing.

Hope that helps :)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-08-12)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。