大网数据广告活动
服务器运维问答

IP forwarding with IPTables for lvs nat mode

· 52CE 我爱测

I am learning about the L3 loadbalancers and now working on the keepalived NAT mode.The packet flow should be Client ------> L3(lvs DNAT)------> real servers and the return flow should be exact opposite to request flow.But the lv…

问题原文

原帖链接 · chellathurai · CC BY-SA 4.0

I am learning about the L3 loadbalancers and now working on the keepalived NAT mode.The packet flow should be Client ------> L3(lvs DNAT)------> real servers and the return flow should be exact opposite to request flow.But the lvs DNAT changes only the destination NAT,means only the the destination IP of the packet is changed ie) [source=client ip,destination=real server ip].So after the real server process the request it forwards it to default gateway instead of the L3 server and the packet gets dropped.I dont want to set L3 server as the default gateway of the real server,instead I need an IP table related solution to forward the traffic from Real Server to the L3 server when it comes to the port 8443 and 8080.

IP tables provide the way to forward the traffic before processing the packet via PREROUTING but what I want is forward traffic to L3 after the server process the request. I tried with the OUTPUT chain still with no luck sudo iptables -t nat -A OUTPUT -p tcp --match multiport --sports 8443,8080 -j DNAT --to 172.24.248.201

172.24.248.201 is the ip of the L3 server.

Thanks in Advance

已采纳解答

原帖链接 · A.B · CC BY-SA 4.0

This is a routing problem. iptables doesn't route, but routing can be affected by the addresses that it can change and that's why iptables is often mistakenly assumed to do routing. Because of this iptables can't be the solution to solve the problem (but can sometimes be part of such solution).

Moreover any attempt to alter a reply packet using Netfilter/iptables' NAT is ignored: the nat table is only triggered for the first packet of a flow: not for a reply: iptables -t nat -A OUTPUT -p tcp --match multiport --sports 8443,8080 -j DNAT --to 172.24.248.201 will never match. And if it could be made to match, this wouldn't work, because the destination of the reply has to be the source of the initial query seen by the real server: the actual remote client IP address, not the L3 LB.

Below is a simple method to do it correctly on the real server: with policy routing.


In this specific problem and answer I'll assume Linux kernel >= 4.17 for "Extends fib rule match support to include sport, dport and ip proto match", which avoids a complex use of iptables to mark packets (but would still require policy routing anyway).

One has to ensure real server's dedicated ports can't overlap the real server's dynamic port range (used when the server is a client, like for DNS queries or system upgrade):

# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999

Ensure that the lower port is above the real server's highest port used for a service. Here: 32768 > 8443 is good enough.

Add a routing table for specific traffic intended to use the L3 LB as gateway. Table 248201 is an arbitrarily chosen value. I'm assuming the real server's interface is named eth0, please adapt:

ip route add default via 172.24.248.201 dev eth0 table 248201

Add the selectors: one per port range. It could be a single wide range (as long as there's no interaction with dynamic ports, it doesn't matter) or X rules for X distinct ports. For a single range:

ip rule add iif lo ipproto tcp sport 8000-8443 lookup 248201

where iif lo is the special syntax for locally initiated outgoing traffic and is not really about the lo interface.

Now all intended return traffic will be routed using the L3 LB as gateway. Any other traffic will use the usual default gateway if needed.


I don't think SRPF should be an issue if enabled unless the server's usual default gateway uses an other interface. If this were to be an issue, it can be set to Loose mode instead.

To be done only if it doesn't work without it (and even then, eth0 may be enough instead of all below):

sysctl -w net.ipv4.conf.all.rp_filter=2

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2022-05-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。