问题原文
原帖链接 · bayindirh · CC BY-SA 4.0
I'm managing a standalone kubernetes (v 1.17.2) cluster installed on CentOS 7 with single API server and two worker nodes for pods.
One of the services is configured as a node port service however, I cannot reach the service from other nodes. I'm not very experienced in Kubernetes but, here is what I know
kube-proxyis doing its job. Related port (31505) is open in every node (checked withlost -i).The service is up, I can access it just fine from the nodeport where the pod is running.
- Docker is configured with
no-iptablesoption, and all firewall rules are managed by Kubernetes itself.- Stopping
kubeletanddocker, flushing all IPTables rules and forcing kubernetes to re-create them didn't solve the problem.
- Stopping
- All other services inside the Kubernetes cluster is working as expected.
For service details kubectl get svc -o wide --namespace docker-registry docker-registry-deployment is as follows:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
docker-registry-deployment NodePort 10.104.194.201 <none> 5000:31505/TCP 67d app=docker-registry
Output of kubectl describe svc docker-registry-deployment --namespace registry is as follows:
Name: docker-registry-deployment
Namespace: registry
Labels: app=docker-registry
Annotations: <none>
Selector: app=docker-registry
Type: NodePort
IP: 10.104.194.201
Port: <unset> 5000/TCP
TargetPort: 5000/TCP
NodePort: <unset> 31505/TCP
Endpoints: 10.244.1.65:5000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
At the current state, the pod cannot be accessed via its Cluster IP:5000 from any node, but the Endpoint:5000 is accessible and its responding as it should.
Update (2020-04-14)
There's no namespace problem for the service & the pods. They are in the same namespace.
I've updated the installations completely (incl. Docker and Kubernetes), and the problem has morphed. Now, other nodes are also responding but, exactly after 63s.
Will try to debug the problem and report back here.
已采纳解答
原帖链接 · bayindirh · CC BY-SA 4.0
I've found the problem at last.
It's a combination of flannel, complicated cni network structure and virtualization.
flannel is implementing its node to node routing network with vxlan and it's enabling offloading features on the vxlan device.
When offloading is enabled on a flannel device, NodePort accesses across nodes experiences a 63 second routing delay. The delay goes away when offloading is turned off with
ethtool --offloading tx off rx off flannel.1
My cluster is working on a Proxmox environment and the network cards are virtio devices. I don't think the problem is on the virtualization layer because I have another cluster on the same Proxmox node and everything is working fine. Since it's a production cluster, it's not updated this frequently.
So this one is marked as a flannel bug. Now I'll try to update flannel and see what happens.
Update: flannel is updated on this particular cluster and it's working like charm.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
