大网数据广告活动
服务器运维问答

nginx use-proxy-protocol with Kubernetes cluster

· 52CE 我爱测

I have a Kubernetes cluster with an external load balancer on a self hosted server running NGINX. I tried to activate the proxy_protocol in order to get the real_ip of clients but NGINX logs are 2020/05/11 14:57:54 [error] 29614#…

问题原文

原帖链接 · MHogge · CC BY-SA 4.0

I have a Kubernetes cluster with an external load balancer on a self hosted server running NGINX. I tried to activate the proxy_protocol in order to get the real_ip of clients but NGINX logs are

2020/05/11 14:57:54 [error] 29614#29614: *1325 broken header: "▒▒▒▒▒▒▒Ωߑa"5▒li<c▒*▒ ▒▒▒s▒       ▒6▒▒▒▒▒X▒▒o▒▒▒E▒▒i▒{ ▒/▒0▒+▒,̨̩▒▒ ▒▒
▒▒/5▒" while reading PROXY protocol, client: 51.178.168.233, server: 0.0.0.0:443

Here is my NGINX configuration file:

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {

    upstream rancher_servers_http {
        least_conn;
        server <IP_NODE_1>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_protocol on;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     443 ssl proxy_protocol;
        ssl_certificate /certs/fullchain.pem;
        ssl_certificate_key /certs/privkey.pem;
        proxy_pass rancher_servers_https;
        proxy_protocol on;
    }
}

Here is my configmap for the ingress-controller:

apiVersion: v1
data:
  compute-full-forwarded-for: "true"
  proxy-body-size: 500M
  proxy-protocol: "true"
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","data":null,"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}}'
  creationTimestamp: "2019-12-09T13:26:59Z"
  labels:
    app: ingress-nginx
  name: nginx-configuration
  namespace: ingress-nginx

Everything was working fine before I add the proxy_protocol directive but now I got all these broken headers errors and I can't reach any services behind ingresses without getting a connection reset error.

What could be wrong with my config ?

Should I use an http reverse proxy instead of a tcp reverse proxy ?

Thank you.


Edit:

I should also say that I doesn't have any service of type LoadBalancer in my cluster. Should I have one ? I'm thinking of Metallb but I'm not sure what it will add to my configuration as I'm already load balancing to nodes with nginx.

社区高分解答

原帖链接 · Dirbaio · CC BY-SA 4.0

Nginx allows you to specify whether to use proxy_protocol in incoming or outgoing requests, and you're confusing the two.

To use proxy_protocol in incoming connections, you have to add proxy_protocol to the listen line like this:

listen     443 ssl proxy_protocol;

To use proxy_protocol in outgoing connections, you have to use the standalone proxy_protocol directive, like this:

proxy_protocol on;

They are not the same. In a load balancer, incoming connections come from browsers, which do not speak the proxy protocol. You want proxy protocol only in your outgoing requests, to the nginx-ingress in your kubernetes cluster.

Therefore, remove the proxy_protocol argument from the listen directive, and it should work.

Additionally, you want use-forwarded-headers: "false" in your nginx-ingress config. That controls whether to use the X-Forwarded-For & co. headers in incoming connections (from the point of view of the nginx-ingress, ie outgoing from your load balancer), and you're using proxy protocol in these instead of the headers. With it enabled, your users may be able to spoof IPs by specifying X-Forwarded-For, which can be a security issue. (only if nginx-ingress gives priority to the headers over the proxy protocol, which I'm not sure)

An aside: nginx-ingress itself already load-balances traffic between all pods. With your architecture, you're running two "layers" of load balancers, which is probably unnecessary. If you want to simplify, force nginx-ingress to run on a single node (with nodeSelector for example) and simply send all your traffic to that node. If you want to keep the load balancer on a dedicated machine, you can join the 4th machine to the cluster and make sure it just runs nginx-ingress (with taints and tolerations).

Also, make sure you're running nginx-ingress with hostNetwork: true, otherwise you may be having yet another layer of balancing (kube-proxy, the kubernetes service proxy)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-05-11)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。