大网数据广告活动
服务器运维问答

GNU parallel doesn't fully utilize my CPUs

· 52CE 我爱测

I'm running a command like this on my 36 core server (EC2 c4.8xlarge/Amazon Linux). find . -type f | parallel -j 36 mycommand The number of files to process is ~1,000,000, and it takes dozens of minutes. It should run 36 processe…

问题原文

原帖链接 · aosho235 · CC BY-SA 3.0

I'm running a command like this on my 36 core server (EC2 c4.8xlarge/Amazon Linux).

find . -type f | parallel -j 36 mycommand

The number of files to process is ~1,000,000, and it takes dozens of minutes. It should run 36 processes simultaneously. However, from the result of top, there are about 10 processes at most, and 70% is idle. ps shows more processes, but most of them are defunct.

I guessed it was because each mycommand finished so quickly, parallel could not catch up spawning new processes. So I tried parallel --nice 20 to allocate more CPU time to parallel itself, but this didn't work.

Does anyone have an idea to improve this?

$ parallel --version GNU parallel 20151022

已采纳解答

原帖链接 · Ole Tange · CC BY-SA 3.0

The number of files to process is ~1,000,000, and it takes dozens of minutes.

So you are running around 600 jobs per second. The overhead for a single GNU Parallel job is in the order of 2-5 ms, so when you are getting more than 200 jobs per second, GNU Parallel will not perform better without tweaking.

The tweak is to have more parallels spawining jobs in parallel. From https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Running-more-than-250-jobs-workaround

cat myinput | parallel --pipe -N 100 --round-robin -j50 parallel -j100 your_prg

This way you will have 50 GNU Parallel that can each spawn 100 jobs per second.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-05-31)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。