大网数据广告活动
服务器运维问答

"Too many open files" Apache error even with increased ulimit and sysctl

· 52CE 我爱测

I'm load testing an Amazon Linux EC2 instance running Apache (event MPM) and PHP-FPM using Locust. When I run my load test with 200 users (~28 requests per second), everything is fine. When I boost the number of users to 300 (~43…

问题原文

原帖链接 · Monkey34 · CC BY-SA 4.0

I'm load testing an Amazon Linux EC2 instance running Apache (event MPM) and PHP-FPM using Locust. When I run my load test with 200 users (~28 requests per second), everything is fine. When I boost the number of users to 300 (~43 requests per second), I start seeing these errors in the Locust logs:

ConnectionError(MaxRetryError("HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x....>: Failed to establish a new connection: [Errno 24] Too many open files'))"))

Researching online, I decided to bump up the available number of open file descriptors to see if I could get around this issue. I edited /etc/security/limits.conf and set the following values (possibly exaggerated but I'm just trying to see if something sticks):

*               soft    nofile          65000
*               hard    nofile          65000
*               soft    nproc           10240
*               hard    nproc           10240

Afterwards, I restarted both Apache and PHP-FPM:

sudo service httpd restart
sudo service php-fpm restart

I also looked at the processes to verify the new limits and make sure they were sticking. One of Apache's child processes:

$ cat /proc/22725/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             14745                14745                processes
Max open files            170666               170666               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       14745                14745                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

And one of PHP-FPM's child processes:

$ cat /proc/22963/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             10240                10240                processes
Max open files            10240                10240                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       14745                14745                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

I've also upped the max open files at the kernel level in /etc/sysctl.conf:

fs.file-max = 512000

Then I persisted the values with sysctl -p. Again, this is probably egregious but I saw the same results with a value of 65000.

Under load, I'm only seeing ~4,200 open files, which is puzzling given the overall limits I've provided:

$ lsof | wc -l
4178

During all of this, my CPU usage never goes above 20%, and my server still has around 3GB of free memory.

Any ideas?

社区高分解答

原帖链接 · Monkey34 · CC BY-SA 4.0

After sleeping on this issue, I realized the problem might not be on the server side at all, but instead on the client side (i.e., my laptop running Locust). Indeed, checking ulimit -a here gave these results (running macOS 10.14.6):

➜ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                256

Bumping the file descriptors up to 2048 (ulimit -n 2048) and re-running Locust in the same shell made the errors go away.

Sorry for the quick question-and-answer, but I thought I'd keep this up rather than remove the question in case someone else runs into this issue.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-08-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。