layer 3 bridge - hanyong/note GitHub Wiki
3 层网桥
两个网桥
- 网桥 nat0 做 NAT, ip
192.168.1.2/24 - 网桥 vmbr0 连接虚拟机
- 宿主机 han-yoga 设置
net.ipv4.ip_forward=1
/etc/network/interfaces 配置如下:
auto nat0
iface nat0 inet static
bridge_ports none
bridge_maxwait 1
address 192.168.1.2/24
up iptables -t nat -A POSTROUTING -s 192.168.0.0/16 ! -d 192.168.0.0/16 -j MASQUERADE
down iptables -t nat -D POSTROUTING -s 192.168.0.0/16 ! -d 192.168.0.0/16 -j MASQUERADE
dns-nameserver 192.168.1.1
auto vmbr0
iface vmbr0 inet manual
bridge_ports none
bridge_maxwait 1
宿主机 han-yoga 网络状态:
hanyong@han-yoga:~$ ip -d link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
link/ether 34:e6:ad:d1:12:a3 brd ff:ff:ff:ff:ff:ff promiscuity 0
3: nat0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 62:95:f9:7a:9d:68 brd ff:ff:ff:ff:ff:ff promiscuity 0
bridge
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether c6:13:ea:23:29:66 brd ff:ff:ff:ff:ff:ff promiscuity 0
bridge
hanyong@han-yoga:~$ ip route
default via 192.168.100.1 dev wlan0 proto static
169.254.0.0/16 dev nat0 scope link metric 1000
192.168.1.0/24 dev nat0 proto kernel scope link src 192.168.1.2
192.168.100.0/24 dev wlan0 proto kernel scope link src 192.168.100.56 metric 9
hanyong@han-yoga:~$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
- 虚拟机 node1n1 连接 nat0
- node1n1 作为网关, IP
192.168.1.1/24, 设置net.ipv4.ip_forward=1, 提供 DNS 和 DHCP 服务. - 虚拟机 node2n2 连接 vmbr0, IP 未配置
宿主机网络状态:
hanyong@han-yoga:~$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
link/ether 34:e6:ad:d1:12:a3 brd ff:ff:ff:ff:ff:ff
3: nat0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether fe:54:00:1a:3c:41 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether fe:54:00:e7:9a:9b brd ff:ff:ff:ff:ff:ff
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nat0 state UNKNOWN mode DEFAULT group default qlen 500
link/ether fe:54:00:1a:3c:41 brd ff:ff:ff:ff:ff:ff
6: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 500
link/ether fe:54:00:e7:9a:9b brd ff:ff:ff:ff:ff:ff
hanyong@han-yoga:~$ brctl show
bridge name bridge id STP enabled interfaces
nat0 8000.fe54001a3c41 no vnet0
vmbr0 8000.fe5400e79a9b no vnet1
node2n2 网络状态:
hanyong@node2n2:~$ ip -d link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:e7:9a:9b brd ff:ff:ff:ff:ff:ff promiscuity 0
hanyong@node2n2:~$ ip route
hanyong@node2n2:~$ arp -n
node2n2 添加路由:
hanyong@node2n2:~$ sudo ip route add default dev eth0
RTNETLINK answers: Network is down
hanyong@node2n2:~$ sudo ip link set eth0 up
hanyong@node2n2:~$ sudo ip route add default dev eth0
hanyong@node2n2:~$ ip route
default dev eth0 scope link
开两个终端监听 nat0 和 vmbr0 网桥:
hanyong@han-yoga:~$ sudo tcpdump -i nat0 -nn -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on nat0, link-type EN10MB (Ethernet), capture size 65535 bytes
hanyong@han-yoga:~$ sudo tcpdump -i vmbr0 -nn -e
tcpdump: WARNING: vmbr0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), capture size 65535 bytes
node2n2 ping nat0:
hanyong@node2n2:~$ ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
--- 192.168.1.2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
nat0 无显示, vmbr0 显示如下:
17:30:21.595793 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.2 tell 0.0.0.0, length 28
17:30:21.595831 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.2 is-at fe:54:00:e7:9a:9b, length 28
17:30:21.595984 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 0.0.0.0 > 192.168.1.2: ICMP echo request, id 881, seq 1, length 64
- node2n2 询问
192.168.1.2, vmbr0 用自己的地址作了应答 (因为 vmbr0 在宿主机上, 宿主机上的所有 IP 他都应答). - 但 node2n2 发出去的 ping 没有响应, 100% 丢包.
node2n2 ping node1n1:
hanyong@node2n2:~$ ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 127.0.0.1 icmp_seq=1 Destination Host Unreachable
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 显示如下:
17:36:40.636299 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 0.0.0.0, length 28
17:36:41.634165 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 0.0.0.0, length 28
17:36:42.634157 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 0.0.0.0, length 28
node2n2 询问了 3 次谁是 192.168.1.1, 没有应答, 100% 丢包.
设置 proxy_arp
vmbr0 设置 proxy_arp=1 (参考 https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt):
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.vmbr0.proxy_arp
net.ipv4.conf.vmbr0.proxy_arp = 0
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.vmbr0.proxy_arp=1
net.ipv4.conf.vmbr0.proxy_arp = 1
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.vmbr0.proxy_arp
net.ipv4.conf.vmbr0.proxy_arp = 1
node2n2 再 ping node1n1, 还是一样寻址了 3 次没有应答, 100% 丢包.
nat0 设置 proxy_arp=1:
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.nat0.proxy_arp
net.ipv4.conf.nat0.proxy_arp = 0
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.nat0.proxy_arp=1
net.ipv4.conf.nat0.proxy_arp = 1
node2n2 再 ping node1n1, 结果还是一样.
根据 https://wiki.debian.org/BridgeNetworkConnectionsProxyArp 的说明,
proxy_arp 让一个设备对不在该网络上的地址进行应答, -但只会应答存在 /32 主机路由的地址-.
测试设置 proxy_arp 时的状态, nat0, vmbr0 取消 proxy_arp, 添加到 node1n1 的主机路由:
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.nat0.proxy_arp=0
net.ipv4.conf.nat0.proxy_arp = 0
hanyong@han-yoga:~$ sudo sysctl net.ipv4.conf.vmbr0.proxy_arp=0
net.ipv4.conf.vmbr0.proxy_arp = 0
hanyong@han-yoga:~$ sudo ip route add 192.168.1.1/32 dev nat0
hanyong@han-yoga:~$ ip route
default via 192.168.100.1 dev wlan0 proto static
169.254.0.0/16 dev nat0 scope link metric 1000
192.168.1.0/24 dev nat0 proto kernel scope link src 192.168.1.2
192.168.1.1 dev nat0 scope link
192.168.100.0/24 dev wlan0 proto kernel scope link src 192.168.100.56 metric 9
node2n2 再 ping node1n1, 还是一样不通.
vmbr0 设置 proxy_arp=1, node2n2 再 ping node1n1, 还是一样不通.
nat0 设置 proxy_arp=1, node2n2 再 ping node1n1, 还是一样不通.
给 vmbr0 分配 IP
给 vmbr0 分配 ip 地址 192.168.1.5/32, 路由表无变化:
hanyong@han-yoga:~$ sudo ip addr add 192.168.1.5/32 dev vmbr0
hanyong@han-yoga:~$ ip route
default via 192.168.100.1 dev wlan0 proto static
169.254.0.0/16 dev nat0 scope link metric 1000
192.168.1.0/24 dev nat0 proto kernel scope link src 192.168.1.2
192.168.1.1 dev nat0 scope link
192.168.100.0/24 dev wlan0 proto kernel scope link src 192.168.100.56 metric 9
node2n2 ping vmbr0:
hanyong@node2n2:~$ ping -c 1 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
--- 192.168.1.5 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 看到:
18:27:46.334189 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.5 tell 0.0.0.0, length 28
18:27:46.334264 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.5 is-at fe:54:00:e7:9a:9b, length 28
18:27:46.334367 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 0.0.0.0 > 192.168.1.5: ICMP echo request, id 904, seq 1, length 64
node2n2 询问 192.168.1.5, vmbr0 用自己的 mac 地址做了应答 (理所当然).
但对 ping 请求没有响应.
node2n2 没有 IP, vmbr0 不知道如何响应它.
node2n2 ping nat0:
18:31:26.463196 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 0.0.0.0 > 192.168.1.2: ICMP echo request, id 906, seq 1, length 64
18:31:31.474151 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.2 tell 0.0.0.0, length 28
18:31:31.474214 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.2 is-at fe:54:00:e7:9a:9b, length 28
跟之前一样不通, node2n2 问谁是 192.168.1.2, vmbr0 又返回了自己.
node2n2 ping node1n1 (192.168.1.1), 跟之前一样不通, ARP 也没有响应.
给 node2n2 分配 ip
给 vmbr0 分配 ip 地址 192.168.1.6/24:
hanyong@node2n2:~$ sudo ip addr add 192.168.1.6/24 dev eth0
hanyong@node2n2:~$ ip route
default dev eth0 scope link
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.6
node2n2 再 ping vmbr0:
hanyong@node2n2:~$ ping -c 1 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
--- 192.168.1.5 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 (incomplete) eth0
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 看到:
18:37:20.334611 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.5: ICMP echo request, id 914, seq 1, length 64
18:37:25.346181 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.5 tell 192.168.1.6, length 28
18:37:26.346173 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.5 tell 192.168.1.6, length 28
18:37:27.346179 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.5 tell 192.168.1.6, length 28
node2n2 请求 vmbr0, 没有应答.
node2n2 询问谁是 192.168.1.5, 依然没有应答.
node2n2 失去了 vmbr0 的地址.
宿主机 han-yoga 添加到 node2n2 的主机路由:
hanyong@han-yoga:~$ sudo ip route add 192.168.1.6/32 dev vmbr0
hanyong@han-yoga:~$ ip route
default via 192.168.100.1 dev wlan0 proto static
169.254.0.0/16 dev nat0 scope link metric 1000
192.168.1.0/24 dev nat0 proto kernel scope link src 192.168.1.2
192.168.1.1 dev nat0 scope link
192.168.1.6 dev vmbr0 scope link
192.168.100.0/24 dev wlan0 proto kernel scope link src 192.168.100.56 metric 9
node2n2 再 ping vmbr0:
hanyong@node2n2:~$ ping -c 1 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=0.184 ms
--- 192.168.1.5 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.184/0.184/0.184/0.000 ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 看到:
18:43:49.157480 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.5 tell 192.168.1.6, length 28
18:43:49.157512 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.5 is-at fe:54:00:e7:9a:9b, length 28
18:43:49.157580 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.5: ICMP echo request, id 916, seq 1, length 64
18:43:49.157615 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.5 > 192.168.1.6: ICMP echo reply, id 916, seq 1, length 64
18:43:54.166279 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.6 tell 192.168.1.5, length 28
18:43:54.166619 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.6 is-at 52:54:00:e7:9a:9b, length 28
ping 通, node2n2 和 vmbr0 相互得到对方的地址.
可见, 当来源地址不可达时, vmbr0 连 ARP 请求都不会响应. 但来源地址是 0.0.0.0 时却会响应.
node2n2 ping nat0 也能 ping 通:
hanyong@node2n2:~$ ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.376 ms
--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.376/0.376/0.376/0.000 ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 看到:
20:05:35.386022 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.2: ICMP echo request, id 921, seq 1, length 64
20:05:35.386132 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.2 > 192.168.1.6: ICMP echo reply, id 921, seq 1, length 64
nat0 还是无任何显示. 对同一机器的 ping 请求 vmbr0 也代为应答了(?).
非 ping 请求呢? 用 nc 做个测试:
hanyong@han-yoga:~$ echo hello | nc -l 192.168.1.2 8080
hanyong@node2n2:~$ nc 192.168.1.2 8080
hello
vmbr0 看到:
20:25:18.339061 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 74: 192.168.1.6.55934 > 192.168.1.2.8080: Flags [S], seq 1695649492, win 29200, options [mss 1460,sackOK,TS val 2832724 ecr 0,nop,wscale 7], length 0
20:25:18.339124 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 74: 192.168.1.2.8080 > 192.168.1.6.55934: Flags [S.], seq 2639203414, ack 1695649493, win 28960, options [mss 1460,sackOK,TS val 3211871 ecr 2832724,nop,wscale 7], length 0
20:25:18.339279 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 66: 192.168.1.6.55934 > 192.168.1.2.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 2832724 ecr 3211871], length 0
20:25:18.339403 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 72: 192.168.1.2.8080 > 192.168.1.6.55934: Flags [P.], seq 1:7, ack 1, win 227, options [nop,nop,TS val 3211871 ecr 2832724], length 6
20:25:18.339435 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 66: 192.168.1.2.8080 > 192.168.1.6.55934: Flags [F.], seq 7, ack 1, win 227, options [nop,nop,TS val 3211871 ecr 2832724], length 0
20:25:18.339914 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 66: 192.168.1.6.55934 > 192.168.1.2.8080: Flags [.], ack 7, win 229, options [nop,nop,TS val 2832724 ecr 3211871], length 0
20:25:18.340108 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 66: 192.168.1.6.55934 > 192.168.1.2.8080: Flags [F.], seq 1, ack 8, win 229, options [nop,nop,TS val 2832724 ecr 3211871], length 0
20:25:18.340141 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 66: 192.168.1.2.8080 > 192.168.1.6.55934: Flags [.], ack 2, win 227, options [nop,nop,TS val 3211871 ecr 2832724], length 0
nat0 还是无输出. 可见, 虽然 IP 是挂在设备上, 同一主机的所有 IP 是共享的, 任一设备都可以处理对所有 IP 的请求(?).
取消 proxy_arp
nat0 和 vmbr0 设置 proxy_arp=0, 再试 node2n2 到 nat0 的 ping 和 nc, 还是一样的连通.
proxy_arp 不影响 node2n2 到宿主机的测试.
node2n2 ping node1n1:
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 (incomplete) eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
hanyong@node2n2:~$ ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.6 icmp_seq=1 Destination Host Unreachable
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
vmbr0 显示:
20:37:56.323708 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.6, length 28
20:37:57.322171 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.6, length 28
20:37:58.322193 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.6, length 28
询问 3 次没有应答.
再次设置 proxy_arp
vmbr0 设置 proxy_arp=1, node2n2 再 ping node1n1:
hanyong@node2n2:~$ ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 ether fe:54:00:e7:9a:9b C eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
vmbr0 显示:
20:40:31.161070 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.6, length 28
20:40:31.826340 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 192.168.1.1 is-at fe:54:00:e7:9a:9b, length 28
20:40:31.826579 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.1: ICMP echo request, id 933, seq 1, length 64
nat0 显示:
20:40:31.826657 fe:54:00:1a:3c:41 > 52:54:00:1a:3c:41, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.1: ICMP echo request, id 933, seq 1, length 64
20:40:31.826912 52:54:00:1a:3c:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.6 tell 192.168.1.1, length 28
20:40:32.826124 52:54:00:1a:3c:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.6 tell 192.168.1.1, length 28
20:40:33.826123 52:54:00:1a:3c:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.6 tell 192.168.1.1, length 28
vmbr0 设置 proxy_arp=1 后, node2n2 询问 192.168.1.1, vmbr0 应答了自己的 mac 地址.
node2n2 发出 ping 请求, vmbr0 和 nat0 成功将其转发到 node1n1 (3 层转发).
但随后 node1n1 询问 192.168.1.6 时没有应答, node1n1 无法发送响应, ping 失败.
nat0 也设置 proxy_arp=1, 再次 ping:
hanyong@node2n2:~$ ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=83.5 ms
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 83.542/83.542/83.542/0.000 ms
vmbr0 看到:
20:56:23.763177 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.1: ICMP echo request, id 936, seq 1, length 64
20:56:23.846532 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 936, seq 1, length 64
nat0 看到:
20:56:23.763253 fe:54:00:1a:3c:41 > 52:54:00:1a:3c:41, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 192.168.1.1: ICMP echo request, id 936, seq 1, length 64
20:56:23.763449 52:54:00:1a:3c:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.6 tell 192.168.1.1, length 28
20:56:23.846235 fe:54:00:1a:3c:41 > 52:54:00:1a:3c:41, ethertype ARP (0x0806), length 42: Reply 192.168.1.6 is-at fe:54:00:1a:3c:41, length 28
20:56:23.846486 52:54:00:1a:3c:41 > fe:54:00:1a:3c:41, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 936, seq 1, length 64
ping 成功. nc 通信也测试成功.
由此可见, 设置 proxy_arp 加 3 层路由可实现类似网桥效果 (3 层网桥).
删除 nat0 上 /32 主机路由
IP 段 192.168.1.0/24 的机器默认在 nat0 网桥上.
对 nat0 上的单个机器设置 /32 主机路由不合理, 而且数量会很多.
删除 192.168.1.1/32 路由:
hanyong@han-yoga:~$ sudo ip route del 192.168.1.1
hanyong@han-yoga:~$ ip route
default via 192.168.100.1 dev wlan0 proto static
169.254.0.0/16 dev nat0 scope link metric 1000
192.168.1.0/24 dev nat0 proto kernel scope link src 192.168.1.2
192.168.1.6 dev vmbr0 scope link
192.168.100.0/24 dev wlan0 proto kernel scope link src 192.168.100.56 metric 9
再次 ping 还是 ok 的.
nat0 网桥上增加一台机器 node1n3, 与 node2n2 也能互相 ping 通. node2n2 看到的 mac 地址始终是 vmbr0 的 mac 地址.
hanyong@node2n2:~$ ping -c 1 192.168.1.80
PING 192.168.1.80 (192.168.1.80) 56(84) bytes of data.
64 bytes from 192.168.1.80: icmp_seq=1 ttl=63 time=0.469 ms
--- 192.168.1.80 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.469/0.469/0.469/0.000 ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
192.168.1.1 ether fe:54:00:e7:9a:9b C eth0
192.168.1.2 ether fe:54:00:e7:9a:9b C eth0
192.168.1.80 ether fe:54:00:e7:9a:9b C eth0
可见只要是能路由到的地址都会应答,
会收到 ARP 请求, 说明是在一个局域网网段内.
可见 proxy_arp 可配合 3 层路由将多个小局域网拼成一个大局域网.
node2n2 设置直连全网 ping 外网 IP 测试, 此时 wifi 网关为 192.168.43.1.
hanyong@node2n2:~$ sudo ip addr flush dev eth0
hanyong@node2n2:~$ sudo ip addr add 192.168.1.6/0 dev eth0
hanyong@node2n2:~$ ip route
hanyong@node2n2:~$ sudo ip route add default dev eth0
hanyong@node2n2:~$ ip route
default dev eth0 scope link
hanyong@node2n2:~$ ping -c 1 180.149.132.47
PING 180.149.132.47 (180.149.132.47) 56(84) bytes of data.
64 bytes from 180.149.132.47: icmp_seq=1 ttl=46 time=666 ms
--- 180.149.132.47 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 666.609/666.609/666.609/0.000 ms
hanyong@node2n2:~$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.5 ether fe:54:00:e7:9a:9b C eth0
180.149.132.47 ether fe:54:00:e7:9a:9b C eth0
vmbr0 看到:
23:29:11.747840 52:54:00:e7:9a:9b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 180.149.132.47 tell 192.168.1.6, length 28
23:29:12.110337 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype ARP (0x0806), length 42: Reply 180.149.132.47 is-at fe:54:00:e7:9a:9b, length 28
23:29:12.110568 52:54:00:e7:9a:9b > fe:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 192.168.1.6 > 180.149.132.47: ICMP echo request, id 1012, seq 1, length 64
23:29:12.414119 fe:54:00:e7:9a:9b > 52:54:00:e7:9a:9b, ethertype IPv4 (0x0800), length 98: 180.149.132.47 > 192.168.1.6: ICMP echo reply, id 1012, seq 1, length 64
wlan0 看到:
23:29:12.110647 34:e6:ad:d1:12:a3 > 20:82:c0:6f:47:46, ethertype IPv4 (0x0800), length 98: 192.168.43.108 > 180.149.132.47: ICMP echo request, id 1012, seq 1, length 64
23:29:12.414057 20:82:c0:6f:47:46 > 34:e6:ad:d1:12:a3, ethertype IPv4 (0x0800), length 98: 180.149.132.47 > 192.168.43.108: ICMP echo reply, id 1012, seq 1, length 64
node2n2 询问外网 IP, vmbr0 依然直接返回自己的 mac 地址.
宿主机上 NAT 转换后访问外网, 并返回结果.
可见设置了 proxy_arp=1 的设备, 实际上成了一个透明网关.