大网数据广告活动
服务器运维问答

EKS ARM Node stuck in NotReady status - runtime network not ready cni config uninitialized

· 52CE 我爱测

I have an EKS Cluster (AWS) named cluster-main running on Kubernetes version: 1.16 Platform version: eks.4 CNI version v1.6.1 There are two node groups in the cluster Cluster Name Instance Type AMI Type generic-node-group t3a.med…

问题原文

原帖链接 · argyrodagdileli · CC BY-SA 4.0

I have an EKS Cluster (AWS) named cluster-main running on

  • Kubernetes version: 1.16
  • Platform version: eks.4
  • CNI version v1.6.1

There are two node groups in the cluster

Cluster Name Instance Type AMI Type
generic-node-group t3a.medium AL2_x86_64
memory-node-group r5a.large AL2_x86_64

The nodes in these groups work fine.

I am trying to add a new node group that consists of ARM instances

Cluster Name Instance Type AMI Type
cpu-node-group c6g.xlarge AL2_ARM_64

However, the nodes of this group are stuck in Not Ready status and the node group fails to get created due to the issue below

Conditions:

Type Status LastHeartbeatTime LastTransitionTime Reason Message
Ready False Mon, 31 May 2021 08:40:22 -0400 Mon, 31 May 2021 08:38:21 -0400 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  • All node groups have Node IAM Role ARN
  • All node groups are AWS-managed groups.
  • All node groups are deployed under two specific subnets (private)

When I SSH into the EC2 instance I get the following logs under /var/log/message

1430 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
1430 kubelet.go:2193] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

I've confirmed that the /etc/cni/net.d directory is indeed empty

I have another EKS cluster with similar characteristics where the ARM node group is initialized without any issue. However, I have found two differences. The test cluster uses:

  • Platform version: eks.5
  • CNI version 1.7.5
    • amazon-k8s-cni-init:v1.7.5-eksbuild.1
    • amazon-k8s-cni:v1.7.5-eksbuild.1

Any ideas?

已采纳解答

原帖链接 · argyrodagdileli · CC BY-SA 4.0

Ok - as @thomas suggested the issue was related to the EKS addons.

For context and as I said in my comment, the cluster was initially created at 1.14 version and was later upgraded to 1.16.

However, the aws-node, kube-proxy, and coredns add-ons were never upgraded. Followed the instructions here but the issue remained.

What I did notice though was that the aws-node was still using the same CNI image (v1.6.3)

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

After further investigation I had to manually upgrade the CNI version following the instructions here

Lastly, I noticed that an aws-node pod was created for my arm64 node - which previously it didn't. However, the liveness probe for the pod was failing and the node was still stuck in NotReady status. So I had to edit the configuration for the kube-proxy daemon set as described in step (3) of this guide.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-05-31)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。