问题原文
原帖链接 · MHogge · CC BY-SA 4.0
I have a Kubernetes cluster with an external load balancer on a self hosted server running NGINX.
I tried to activate the proxy_protocol in order to get the real_ip of clients but NGINX logs are
2020/05/11 14:57:54 [error] 29614#29614: *1325 broken header: "▒▒▒▒▒▒▒Ωߑa"5▒li<c▒*▒ ▒▒▒s▒ ▒6▒▒▒▒▒X▒▒o▒▒▒E▒▒i▒{ ▒/▒0▒+▒,̨̩▒▒ ▒▒
▒▒/5▒" while reading PROXY protocol, client: 51.178.168.233, server: 0.0.0.0:443
Here is my NGINX configuration file:
worker_processes 4;
worker_rlimit_nofile 40000;
events {
worker_connections 8192;
}
stream {
upstream rancher_servers_http {
least_conn;
server <IP_NODE_1>:80 max_fails=3 fail_timeout=5s;
server <IP_NODE_2>:80 max_fails=3 fail_timeout=5s;
server <IP_NODE_3>:80 max_fails=3 fail_timeout=5s;
}
server {
listen 80;
proxy_protocol on;
proxy_pass rancher_servers_http;
}
upstream rancher_servers_https {
least_conn;
server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
}
server {
listen 443 ssl proxy_protocol;
ssl_certificate /certs/fullchain.pem;
ssl_certificate_key /certs/privkey.pem;
proxy_pass rancher_servers_https;
proxy_protocol on;
}
}
Here is my configmap for the ingress-controller:
apiVersion: v1
data:
compute-full-forwarded-for: "true"
proxy-body-size: 500M
proxy-protocol: "true"
use-forwarded-headers: "true"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","data":null,"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}}'
creationTimestamp: "2019-12-09T13:26:59Z"
labels:
app: ingress-nginx
name: nginx-configuration
namespace: ingress-nginx
Everything was working fine before I add the proxy_protocol directive but now I got all these broken headers errors and I can't reach any services behind ingresses without getting a connection reset error.
What could be wrong with my config ?
Should I use an http reverse proxy instead of a tcp reverse proxy ?
Thank you.
Edit:
I should also say that I doesn't have any service of type LoadBalancer in my cluster. Should I have one ? I'm thinking of Metallb but I'm not sure what
it will add to my configuration as I'm already load balancing to nodes with nginx.
社区高分解答
原帖链接 · Dirbaio · CC BY-SA 4.0
Nginx allows you to specify whether to use proxy_protocol in incoming or outgoing requests, and you're confusing the two.
To use proxy_protocol in incoming connections, you have to add proxy_protocol to the listen line like this:
listen 443 ssl proxy_protocol;
To use proxy_protocol in outgoing connections, you have to use the standalone proxy_protocol directive, like this:
proxy_protocol on;
They are not the same. In a load balancer, incoming connections come from browsers, which do not speak the proxy protocol. You want proxy protocol only in your outgoing requests, to the nginx-ingress in your kubernetes cluster.
Therefore, remove the proxy_protocol argument from the listen directive, and it should work.
Additionally, you want use-forwarded-headers: "false" in your nginx-ingress config. That controls whether to use the X-Forwarded-For & co. headers in incoming connections (from the point of view of the nginx-ingress, ie outgoing from your load balancer), and you're using proxy protocol in these instead of the headers. With it enabled, your users may be able to spoof IPs by specifying X-Forwarded-For, which can be a security issue. (only if nginx-ingress gives priority to the headers over the proxy protocol, which I'm not sure)
An aside: nginx-ingress itself already load-balances traffic between all pods. With your architecture, you're running two "layers" of load balancers, which is probably unnecessary. If you want to simplify, force nginx-ingress to run on a single node (with nodeSelector for example) and simply send all your traffic to that node. If you want to keep the load balancer on a dedicated machine, you can join the 4th machine to the cluster and make sure it just runs nginx-ingress (with taints and tolerations).
Also, make sure you're running nginx-ingress with hostNetwork: true, otherwise you may be having yet another layer of balancing (kube-proxy, the kubernetes service proxy)
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
