大网数据广告活动
服务器运维问答

minikube/k8s/kubectl "failed to watch file [ ... ]: no space left on device"

· 52CE 我爱测

I'm fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running kubectl logs -f <POD NAME> I get this, sometimes, and unpredictably, on pods in my own namespace, and i…

问题原文

原帖链接 · Ben · CC BY-SA 4.0

I'm fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running kubectl logs -f <POD NAME>

I get this, sometimes, and unpredictably, on pods in my own namespace, and in kube-system, e.g. when trying to tail the ingress controller logs. There is no apparent error in the pods themselves, and the output of the kubectl logs command contains normal-looking log lines right up to, e.g. failed to watch file "/var/log/pods/kube-system_nginx-ingress-controller-586cdc477c-95pgh_4b8310a4-5f9b-11e9-9666-0800270e7244/nginx-ingress-controller/1.log": no space left on device$

What's the ultimate source of this, and how do I resolve it?

已采纳解答

原帖链接 · Michael Hampton · CC BY-SA 4.0

It sounds like the node has run out of inotify watches.

You can check or set the number of available watches using the sysctl fs.inotify.max_user_watches on each Kubernetes node.

You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.

[root@small ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

[root@large ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 1048576

Whatever the amount is currently, double it until the problem stops occurring.

(Use minikube ssh to access a Minikube node.)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-04-17)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。