大网数据广告活动
服务器运维问答

k8s nginx ingress returns randomly 502 error on load

· 52CE 我爱测

we are using an nginx (1.15.8.1) as ingress controller on our k8s cluster (v.1.17), managed by rancher (2.5.7). This worked pretty fine so far, but now we set up a custom API pod that can be externally accessed via ingress. Now, …

问题原文

原帖链接 · michafn · CC BY-SA 4.0

we are using an nginx (1.15.8.1) as ingress controller on our k8s cluster (v.1.17), managed by rancher (2.5.7).

This worked pretty fine so far, but now we set up a custom API pod that can be externally accessed via ingress.

Now, doing some load testing of the API by firing request every second at the API randomly returns "502 Bad Gateway" every couple of requests. However not periodically.

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

The respective log on ingress controller:

2021/04/11 11:02:48 [error] 25430#25430: *55805583 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx, server: xxx, request: "POST /api/v1/xxx HTTP/2.0", upstream: "http://xxx/api/v1/xxx", host: "xxx"

We ran the very same container on a non K8s environment as docker container, where we never experienced this issue before, so at the moment I would assume this is not a problem of the container/API implementation.

My thoughts were:

1- the service definition that abstracts the API pod cannot route the traffic to a pod and therefore replies with a 502

-> I changed the ingress to directly go to the workdload for testing to understand if this is the issue. I get the same errors then, so the service definition seems not to be the problem

2- timeout in ingress resource (b/c the post replies with latency) might cause the routing to fail

-> i set the timeouts to

    nginx.ingress.kubernetes.io/proxy-connect-timeout: 1d
    nginx.ingress.kubernetes.io/proxy-read-timeout: 1d
    nginx.ingress.kubernetes.io/proxy-send-timeout: 1d

but ran in the same error again. so this seem not to be the problem either

3- error in API container (most unlikely) which sends back errors

-> checking the container logs reveals that the missing request (which returned the 502 or anything similar which is then evaluated on ingress level as 502) was never received by the API container. Furthermore this doesn't make sense at all from a traffic routing perspective.

4- As a final step i scaled up the pod to two containers and everything works perfectly well.

Summary: with the results above i sum up that 1- service definition cannot be the problem (test 1) 2- ingress timeouts cannot be the problem 3- the api itself cannot be the problem

Thinking about test 4 it points in a direction that the container availability in the pods could be a problem in terms that the API drops requests (resulting in a 502) when the container is in a specific state (this would explain why it works with 2 containers scaling) OR the traffic routing times out somewhere on the way.

Currently i don't have any further ideas where to go from here. Any hint is appreciated.

社区高分解答

原帖链接 · Tin Pham · CC BY-SA 4.0

if you use the nginx ingress reverse proxy, it maybe the cause. Try to check in Configmap for proxy-next-upstream settings, and extend it to handle the http_502 case.
Also enable the retry-non-idempotent if the request that got 502 is POST, LOCK, PATCH, if it is safe for your app to do so.

My guess: when the backend pod reach its load limit (or pod recycling), it "rejects" new request and nginx reverse proxy is more sensitive with this rejection. Without nginx-ingress, k8s handling the load balancing and it can handle everything better, queuing instead of rejecting.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-04-11)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。