大网数据广告活动
服务器运维问答

Linux: slow SSL handshake due to delayed client hello

· 52CE 我爱测

While investigating an issue afflicting a cluster of proxies which are all affected simultaneously, I've found weird behaviour in the establishment of SSL connections. The symptom is that outgoing HTTPS requests are slower than u…

问题原文

原帖链接 · André Fernandes · CC BY-SA 3.0

While investigating an issue afflicting a cluster of proxies which are all affected simultaneously, I've found weird behaviour in the establishment of SSL connections.

The symptom is that outgoing HTTPS requests are slower than usual when the impact occurs, I've pinned it down to slowness in finishing the SSL handshake. HTTP requests/connections are not affected in the same way.

The issue appears to be caused on outbound connections by a delay between the end of the TCP 3-way handshake and the sending of the Client Hello by the proxies. After that the handshake completes normally without delays.

Here are some examples from traffic captures:

To api.twitter.com (2.4 seconds delay):

To graph.facebook.com (28.4 seconds delay):

Even with the retransmissions in the second example, the Client Hello packet should not have taken that long to go out.

Some facts/considerations:

  • The issue occurs temporarily at specific times of the day (around 1000h and 1700h), affects all hosts and disappears about 30min. later, simultaneously
  • this would indicate an external cause (network, perhaps), but the tcpdump output seems to put the blame on the local server
  • CPU, load, memory, and all other monitored performance indicators are normal at that time
  • it affects all SSL remote hosts
  • affects connections randomly, some behave normally but many are extremely slow
  • throughput (after the handshake) doesn't seem to be affected
  • once the issue passes, SSL connections to the same remote hosts are consistently fast
  • tests were done with curl and openssl s_client connect, with the same results

What needs clarification:

  1. What could be causing such a delay on the local side?
  2. Could wireshark be fooling me?
  3. what other performance indicators / stats / commands could I look into to further troubleshoot the causes for the delay?
  4. is there any networking factor (MTU, receive buffers, fragmentation) which could justify such behaviour?
  5. how can I find evidence to clarify if it's a networking issue external to my servers?

Software versions:

  • Red Hat Enterprise Linux Server release 5.11 (Tikanga)
  • OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
  • kernel 2.6.18-416.el5 #1 SMP Wed Oct 26 12:04:18 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

Edit: strace information

Did a few straces as recommended by an answer below, caught these slow calls:

strace -T -o output.strace openssl s_client -connect 104.244.42.66:443 </dev/null

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.244.42.66")}, 16) = 0 <2.266597>
poll([{fd=4, events=POLLIN}], 1, 5000)  = 1 ([{fd=4, revents=POLLIN}]) <2.387366>
write(3, "\26\3\1\0S\1\0\0O\3\1X\342\24\3556c\354\270T\302\225[\236\317\327\305\205r\177\t/"..., 88) = 88 <0.000034>
read(3, "\26\3\1\0001\2\0", 7)          = 7 <2.556229>
read(3, "\0-\3\1\332\37\254+\240\320\236qA\375\275L\23l\340\355\205x\264\274\273\213\377\323&\345\307O"..., 47) = 47 <0.000011>
read(3, "\26\3\1\v\273", 5)             = 5 <0.000007>
(...)
read(3, "\24\3\1\0\1", 5)               = 5 <2.223115>

The poll() call is a reverse DNS lookup, it's doing:

sendto(4, "\3623\1\0\0\1\0\0\0\0\0\0\00266\00242\003244\003104\7in-ad"..., 44, MSG_NOSIGNAL, NULL, 0) = 44 <0.000157>

Other such poll() calls in the same trace are quick.

已采纳解答

原帖链接 · Nils · CC BY-SA 3.0

You could try running your curl commands with strace to see on which system call they "hang". I've found that these things are sometimes related to DNS lookups (or reverse DNS lookups)

strace curl https://[...]

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2017-03-24)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。