flannel服务生成一个虚拟的网络，负责分配给docker和pod地址。因此docker所在的宿主机和k8s产生的pod的地址都可以互相ping通。出现网络不通的问题，会影响pod之间的通信。

1. 排查

1.1 主机之间排查，master node1 node2 flannle地址是否通。

主机之间先排查通信情况

 [root@master ~]# ping node1
PING node1 (192.168.0.72) 56(84) bytes of data.
64 bytes from node1 (192.168.0.72): icmp_seq=1 ttl=64 time=0.851 ms
64 bytes from node1 (192.168.0.72): icmp_seq=2 ttl=64 time=0.420 ms

node1上docker0地址是10.0.58.1

[root@node1 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1472
        inet 10.0.58.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::42:55ff:fe9d:b2da  prefixlen 64  scopeid 0x20<link>
        ether 02:42:55:9d:b2:da  txqueuelen 0  (Ethernet)
        RX packets 54  bytes 3784 (3.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 1916 (1.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 10.0.58.0  netmask 255.255.0.0  destination 10.0.58.0
        inet6 fe80::43d1:10c7:a30b:bf88  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 15  bytes 1260 (1.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1152 (1.1 KiB)

[root@master ~]# ping 10.0.58.1
PING 10.0.58.1 (10.0.58.1) 56(84) bytes of data.
64 bytes from 10.0.58.1: icmp_seq=1 ttl=62 time=1.04 ms
64 bytes from 10.0.58.1: icmp_seq=2 ttl=62 time=0.627 ms

1.2 宿主机和pod之间通信的排查

[root@master ~]# kubectl get pod -o wide
NAME            READY     STATUS    RESTARTS   AGE       IP          NODE
httpdrc-2zv0t   1/1       Running   0          19m       10.0.82.3   node2
httpdrc-4k1bv   1/1       Running   0          19m       10.0.82.2   node2
httpdrc-4zrgv   1/1       Running   0          19m       10.0.58.2   node1

任意选择一个pod地址，ping

[root@master ~]# ping 10.0.82.3 
PING 10.0.82.3 (10.0.82.3) 56(84) bytes of data.
64 bytes from 10.0.82.3: icmp_seq=1 ttl=61 time=3.49 ms
64 bytes from 10.0.82.3: icmp_seq=2 ttl=61 time=0.475 ms

2. ping pod不通

2.1 检查flanneld是否启动

如果不启动flanneld服务，执行ifconfig命令没有flannel虚拟网络，

[root@master ~]# systemctl status flanneld.service 
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-05-15 15:19:24 CST; 44min ago
 Main PID: 10049 (flanneld)
   CGroup: /system.slice/flanneld.service
           └─10049 /usr/bin/flanneld -etcd-endpoints=http://master:2379 -etcd-prefix=/coreos.com/network -iface=en

2.2 在master etcd检查pod子网分配情况

[root@master ~]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.21.0-24
/coreos.com/network/subnets/10.0.82.0-24
/coreos.com/network/subnets/10.0.58.0-24

2.3 检查路由

分别在master和node上检查route，如果存在flannel子网，就没有问题

[root@master ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 ens33
10.0.0.0        0.0.0.0         255.255.0.0     U     0      0        0 flannel0
10.0.21.0       0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33
[root@node1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 ens33
10.0.0.0        0.0.0.0         255.255.0.0     U     0      0        0 flannel0
10.0.58.0       0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33

如果不存在，填加路由

 [root@node1 ~]# route add -net 10.0.0.0 netmask  255.255.0.0 dev flannel0

添加路由后，要重新启动flannal服务。

2.4 填加临时转发

master和node节点

 [root@node1 ~]# echo "1" > /proc/sys/net/ipv4/ip_forward

2.5 iptables 放行转发

master和node节点

 [root@node1 ~]#iptables -P FORWARD ACCEPT
 

————Blueicex 2020/05/15 16:15 blueice1980@126.com