大网数据广告活动
服务器运维问答

POST request is repeated with nginx loadbalanced server (status 499)

· 52CE 我爱测

double uploads Ever since we went from a simple Apache instance to a loadbalanced environment, there are sometimes issues with POST requests getting repeated. We are running nginx as reverse proxy. Static content is being served …

问题原文

原帖链接 · Nanne · CC BY-SA 3.0

double uploads

Ever since we went from a simple Apache instance to a loadbalanced environment, there are sometimes issues with POST requests getting repeated. We are running nginx as reverse proxy. Static content is being served by nginx itself, and dynamic content is served from two Apache backends.

I have checked that it is not an interface/user error. A small for-instance: a simple image-upload will result in the image getting uploaded twice. The request/POST isn't send twice by double clicking or user-failure. I have not found any evidence that the browser is sending the request twice, so my suspicion is on the server-side. (Notice that this is suspicion only.) Most of these requests are internal, meaning that they are from employees, so I can verify how they come about.

The only thing 'wrong' I can find is that nginx will log a 499 error in these cases. I'm not sure however if this is the cause or just a (side)effect of the problem. (I'm aware that 499 isn't a default http status, it's a nginx status meaning "client has closed connection")

requests

The repeated POST-requests are almost all requests that can take a while. The one I'm showing here as an example is a simple image upload, but the script does some stuff in the background (the image must be converted into different formats/sizes, and should be distributed to both servers, etc).

logs

An example is the uploading of an image. nginx will log one '499', and one 200 request, but Apache is receiving (and handling!) two requests.

Apache

[17:17:37 +0200] "POST ***URL** HTTP/1. 0" 200 9045   
[17:17:47 +0200] "POST ***URL** HTTP/1. 0" 200 20687

nginx

[17:17:47 +0200] "POST ***URL** HTTP/1.1" 499 0 
[17:17:52 +0200] "POST ***URL** HTTP/1.1" 200 5641

Suspicions

It seems to me that bigger/slower uploads suffer from this more, so I suspect a timeout. I have tried to read up on the 499 error: conclusions seem to be that it is "client closed connection". That could be the case in the background, but I'm not sure how this would mean that a second request should be issued and there certainly isn't something like "user closed browser" going on.

Currently it seems to help to break up slower POST requests (if there are multiple things to do, just make the user choose 1 and POST a second time for the other), but this could be just lowering the chance that it occurs. Not sure.

This is obviously a temporary solution. If it is a timeout, I need to find out where and increase the corresponding numbers, but I'm not sure why a timeout would cause this behaviour:I'd suspect a "well, that went wrong" message, not a repeat.

Questions

I'm looking to find out what process/situation can cause a POST to be repeated. Of course, any "not sure why, but it will be fixed by increasing this timeout" is great as well.

nginx configurations

NGINX.conf

user  nginx;
worker_processes  2;
worker_rlimit_nofile 10240;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;


events {
    multi_accept on;
    worker_connections  4096;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    tcp_nodelay     off;    
    client_max_body_size    30m;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}

conf.d

I have removed some IP-specific lines in the geo parts, as well as the SSL variations, to keep it simple. If needed I can replace them, but it boils down to an extra geo part for the ssl backends, and corresponding upstreams and conf files.

geo $backend {
    default apache-backend;
}

upstream apache-backend {
    ip_hash;
    server SERVER1 max_fails=3 fail_timeout=30s weight=2;
    server SERVER2 max_fails=3 fail_timeout=30s weight=3;
}

conf.d/somestring.conf

limit_conn_zone $binary_remote_addr zone=somestring:10m;
server {
    listen ip1:80;
    listen ip2:80;
    server_name name.tld www.name.tld;

    root PATH

    access_log PATH/name.log main;

    location / {
            proxy_pass              http://$backend;
    }

            //*some more locations**//

    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
}

conf.d/proxy.conf

proxy_set_header        Accept-Encoding "";
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        Host $http_host;

proxy_buffering         on;
proxy_read_timeout      90;
proxy_buffer_size       32k;
proxy_buffers           8 32k;
proxy_busy_buffers_size    32k;
proxy_temp_file_write_size 32k;

已采纳解答

原帖链接 · Nanne · CC BY-SA 3.0

From this forum topic we learned that the culprint might be SPDY. For that user it seems a solution to disable it, and we have not had double posts since disabling it either.

The exact problem, other then "SPDY did it", is unknown at this point, side-effects of the proposed solution (disable SPDY) are obviously "no more SPDY", but we can live with that.

Until the bug jumps up again I call this a 'fix' (or at least: a solution to the problem).

edit: We have not (25-02-2014) seen this problem come up anymore, so this seems like a lasting solution indeed.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2013-07-15)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。