问题原文
原帖链接 · biscuit314 · CC BY-SA 3.0
We are using nginx as a load balancer/failover for a pair of upstream servers. We started 11/1.
This chart shows the picture. Where there is no dot, there were no 502s on that day:
In the first few days, the logs show a small number of 502 response codes, likely due to tweaking or other activity as we stabilized our nginx configuration. Then we ran for 12 days with no 502s (except one blip 11/13 - again maybe a tweak).
On 11/20, we moved SSL termination from the upstream servers to the edge. From then on we see 502s daily, and the number seems to be growing (as a percentage of all requests)
Yesterday, for the first time since 11/1, we started receiving client complaints.
Though a low percentage (never reaching 1%) of all traffic (~½ million requests per day), they usually bunch up and span ~10-15 seconds. During this time, many users experience degraded functionality or loss of access.
nginx.conf
worker_processes auto;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 70;
keepalive_requests 100000;
tcp_nopush on;
tcp_nodelay on;
open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
gzip on;
gzip_min_length 1000;
gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
gzip_disable "MSIE [1-6]\.";
log_format main '$time_iso8601\t$status\t$remote_addr\t$upstream_addr\t$upstream_status\t$scheme\t$request\t$request_time\t$upstream_response_time\t$body_bytes_sent';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log error;
# error_log /var/log/nginx/error_debug.log
upstream example {
server 192.168.1.40:80;
server 192.168.1.41:80;
}
server {
listen 80;
listen 443 default ssl;
server_name example.com;
# ssl on;
ssl_certificate ssl/example.com.crt;
ssl_certificate_key ssl/example.com.key;
ssl_trusted_certificate ssl/example.com.pem;
location / {
proxy_read_timeout 180;
proxy_pass http://example;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
When a 502 hits, the access log provides these values:
$status: 502
$upstream_addr: 192.168.1.40, example
$upstream_status: 500, 502
or
$status: 502
$upstream_addr: example
$upstream_status: 502
or variations like those.
The error log says:
[error] 21293#21293: *2441745 no live upstreams while connecting to upstream
Installation details:
- Ubuntu server 16.04.3 LTS
- nginx version: nginx/1.12.2
- 2 CPU cores @ 3.00GHz
- 8 GB RAM
- 2x10 Gbe NIC
- 500 GB HDD
My questions:
- How did moving the cert to the edge increase the instances of 502 errors and how do we fix it?
- Why is the rate increasing? Actual load is fairly flat. Is this a leak of some kind?
EDIT TO ADD:
- adding the keepalive (thanks @Owen Garret) did not eliminate the 502s. We will check tonight if it perhaps reduced, then we can adjust the keepalive value accordingly
- in the meantime, we reverted to terminating SSL on the web server (passthrough). This, so far, has eliminated the 502s.
已采纳解答
原帖链接 · Owen Garrett · CC BY-SA 3.0
NGINX is generating the 502 error because it cannot make an http connection when needed to the upstreams (your 'proxy_pass http://example;' config).
The first place to check would be your upstream servers. Check the server error logs and the syslog to look for indications about why they may be failing.
The problem became greater when you changed from proxying the SSL connection using TCP (stream) load balancing, to terminating SSL and making http connections to the upstream? If so, then one effect of this change is that the upstreams may handle more frequent TCP connects:
- When proxying the SSL connection using TCP (stream) load balancing, all of the requests in the connection are sent to the upstream in the same proxied connection.
- When terminating the connection and then making new requests to the upstream, NGINX will by default create a new TCP connection for each request.
You can encourage NGINX to hold TCP connections open and reuse them for future requests, by configuring NGINX to use keepalive connections to the upstreams as described. This change may reduce the number of 502 errors.
Add the following to your location block, alongside the proxy_pass directive:
proxy_http_version 1.1;
proxy_set_header Connection "";
Add the following to your upstream group configuration:
keepalive 20;
See here for more details: https://www.nginx.com/blog/load-balancing-with-nginx-plus-part2/#keepalive
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
