大网数据广告活动
服务器运维问答

Why am I not receiving a response when the request is sent through a load balancer and the response is too big to fit into a single packet?

· 52CE 我爱测

There's a certain condition where I'm unable to receive a response from an HTTP request to a particular service: the request is sent through a load balancer and the response is too big to fit into a single packet. Under those con…

问题原文

原帖链接 · Justin York · CC BY-SA 4.0

There's a certain condition where I'm unable to receive a response from an HTTP request to a particular service: the request is sent through a load balancer and the response is too big to fit into a single packet. Under those conditions, a response is never received. Neither is an error. The server sends a TCP ACK in response to the packet containing the HTTP request, but nothing comes after that. The connection just hangs waiting for something to happen. The behavior is observed whether the request is sent via CURL, node's HTTP library, or Postman.

What could cause that behavior? How can I debug what is causing the issue?

  • I have verified that the service behind the load balancer is receiving and responding to the HTTP request, but the response is getting lost somewhere.
  • We have other services with a similar setup that don't exhibit this issue.
  • We tried setting a different instance of the load balancer but it had the same problem.

In tshark on the server, I see a handful of [TCP Retransmission] entries after it receives one of the problematic requests through the load balancer (this doesn't happen when the same request is received directly). This seems to indicate that something between the client and the server is misdirecting TCP traffic (highly implicates the load balancer) but I can't understand what would cause that to occur only when HTTP responses are split into multiple packets, and only by this particular service.

Conditions that cause the failure:

Direct Loadbalancer
Single Packet ✅ ✅
Multiple Packets ✅

Network topology:

Client -> Load Balancer -> Service

已采纳解答

原帖链接 · Justin York · CC BY-SA 4.0

It turns out that I discovered a PMTUD black hole.

I first suspected a fragmentation problem due to an article from Cloudflare about packet fragmentation. I confirmed the issue using ping to send different packet sizes: ping -s SIZE.

First I tried 1500 which is the default MTU for ethernet.

> ping 172.16.5.1 -s 1500

But nothing happened. It just hung and printed nothing. If there's a problem, you're supposed to get an error like this:

ping: sendto: Message too long
Request timeout for icmp_seq 0

Eventually, I found an article that described the issue in more detail and recommended a fix: setting /proc/sys/net/ipv4/tcp_mtu_probing to 1 to enable black hole discovery (which is not enabled by default in Linux ‍♂️). I wasn't actually able to modify that file, so I followed the recommendation to set it in /etc/sysctl.conf and then reload settings via sysctl --system. After all that, I was able to receive responses from the server.

MTU probing adds overhead to the response (it was noticeably delayed) so I still intend to figure out why ICMP packets are not being received, but at least I now understand the problem and have a workable solution.

Update: Turns out a LAG had an MTU of 1500 so it was dropping the packets.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-03-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。