大网数据广告活动
服务器运维问答

kube-apiserver started to die on start-up after a few months of good work - how can I check its logs?

· 52CE 我爱测

The stage: a control plane machine, kubernetes 1.24.3 on a baremetal Ubuntu 22.04, installed with kubeadm, there is also one worker node. The whole set-up worked like a charm for 4 months until some unknown silent kaboom yesterda…

问题原文

原帖链接 · Mikha · CC BY-SA 4.0

The stage: a control plane machine, kubernetes 1.24.3 on a baremetal Ubuntu 22.04, installed with kubeadm, there is also one worker node. The whole set-up worked like a charm for 4 months until some unknown silent kaboom yesterday (I actually don't exclude a sudden hardware issue)

The problem: port 6443 is listed by netstat for the first few minutes after the control plane machine start-up, and then disappears. While it's on, the apiserver is irresponsive anyway - any connection attempt to it is reset by the peer. I.e. there should be some serious problems on the kube-apiserver side, but I can't figure out what it's unhappy with.

I checked some obvious things - ip address didn't change, enough disk space, k8s' certificates are not expired. So I need to check kube-apiserver logs somehow.

As for the logs, the official page says:

On systemd-based systems, you may need to use journalctl instead of examining log files.

But... what component should I run journalctl for?? If I run it for kubelet (journalctl -u kubelet), I don't see much of the logs related to apiserver apart from "can't connect to :6443"

And I don't see any service named kube-apiserver or alike when I run e.g. just systemctl... Also, there are no logs in /var/log/ (not surprising since it's a systemd-based system, but I checked nevertheless)

I wonder is there a way to check the apiserver's logs, or is there some gotcha that I'm missing? Would appreciate any help on this subject!

已采纳解答

原帖链接 · larsks · CC BY-SA 4.0

On my clusters, kube-apiserver runs in a pod (and logs are available via kubectl logs).

If the apiserver was down, I would log into a node directly and use the container runtime to examine the container logs (e.g., crictl logs or docker logs or whatever is appropriate for your system).

For example:

# Find the kube-apiserver pod
root@infra-control-plane:/# crictl pods | grep kube-apiserver
ca01411d447a6       3 days ago          Ready               kube-apiserver-infra-control-plane                  kube-system          0                   (default)
# Find container in that pod
root@infra-control-plane:/# crictl ps --pod ca01411d447a6
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
44da6bf198244       400e6a4878256       3 days ago          Running             kube-apiserver      0                   ca01411d447a6       kube-apiserver-infra-control-plane
# Look at logs for that container
root@infra-control-plane:/# crictl logs 44da6bf198244 |& tail -2
Trace[1944612718]: ---"Writing http response done" 1254ms (21:59:45.444)
Trace[1944612718]: [1.255113794s] [1.255113794s] END

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2022-10-22)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。