环境准备：
三台centos7.4主机，每台主机配置两块网卡：仅主机模式（192.168.147.0/24）和桥接模式（10.10.20.0/24）

master	192.168.147.132
node1	192.168.147.133
node2	192.168.147.134

组件版本
docker 19.03.8
kubeadm 1.18.2
kubelet 1.18.2
kubectl 1.18.2
实验前准备工作：
1、关闭防火墙、关闭selinux

[root@centos7 ~]# setenforce 0
setenforce: SELinux is disabled
[root@centos7 ~]# systemctl stop firewalld

2、同步时间，这一步非常重要，如果集群时间不同步后期会无法使用。
这里使用阿里云的时间同步服务器

[root@centos7 ~]# vim /etc/ntp.conf 
...
#server 192.168.147.1 iburst
server ntp.aliyun.com
...

[root@centos7 ~]# systemctl start ntpd
[root@centos7 ~]# systemctl enable ntpd

3、设置永久主机名称，然后重新登录

[root@centos7 ~]# hostnamectl set-hostname master
[root@centos7 ~]#  hostnamectl set-hostname node1
[root@centos7 ~]# hostnamectl set-hostname node2

修改 /etc/hosts 文件，添加主机名和 IP 的对应关系

[root@node1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 
192.168.147.132 master
192.168.147.133 node1
192.168.147.134 node2

4、 关闭 swap 分区
如果开启了 swap 分区，kubelet 会启动失败(可以通过将参数 --fail-swap-on 设置为false 来忽略 swap on)，故需要在每台机器上关闭 swap 分区：

$ sudo swapoff -a

为了防止开机自动挂载 swap 分区，可以注释 /etc/fstab 中相应的条目：

$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

5、配置安装源
5.1、配置kubernetes仓库
清华大学镜像源

[root@master ~]# cat /etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0

阿里云镜像

[root@node2 ~]# cd /etc/yum.repos.d/
[root@master yum.repos.d]# vim kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
enable=1

5.2、配置docker-ce 源
（1）添加docker-ce 源信息

[root@master ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo

（2）修改docker-ce 源，改为清华大学的仓库

[root@master ~]# sed -i 's@download.docker.com@mirrors.tuna.tsinghua.edu.cn/docker-ce@g' /etc/yum.repos.d/docker-ce.repo

一、安装docker-ce 、kubelet、kubeadm、kubectl

以下操作在3个服务器上，都要执行！

kubelet：负责管理pods和它们上面的容器，维护容器的生命周期
kubeadm：安装K8S工具
kubectl：K8S命令行工具

1、安装

建议安装时指定版本号，默认是会安装最新版本。

[root@master ~]# yum install docker-ce-19.03.8

[root@master ~]# yum install kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

注意
安装docker-ce时会提示错误

解决办法：
（1）下载container-selinux：

wget https://mirrors.aliyun.com/centos/7/extras/x86_64/Packages/container-selinux-2.107-3.el7.noarch.rpm

（2）安装container-selinux（使用–nodeps --force跳过依赖检查）：

 rpm -ivh container-selinux-2.107-3.el7.noarch.rpm --nodeps --force

2、配置启动docker服务
（1）启动服务

[root@centos7 app]# systemctl start docker
[root@centos7 app]# systemctl enable docker

查看版本信息

[root@centos7 app]# docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

（2）配置docker加速器

[root@centos7 ~]# vim /etc/docker/daemon.json

{
      "registry-mirrors" : ["https://registry.docker-cn.com"]
}

（3）配置docker的service文件，添加环境变量，让它帮我们代理访问k8s来拉取镜像
踩坑点（注意：此步骤后导致后面无法拉取镜像，因为国内无法访问，因此后面还得去掉）

[root@master ~]# vim /usr/lib/systemd/system/docker.service
...
[Service]
Environment="HTTPS_PROXY=http://www.ik8s.io:10080"
Environment="NO_PROXY=127.0.0.0/8,192.168.147.0/24"
...

由于修改的service文件，因此需要systemd重载一下

[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart docker

docker info查看信息

[root@master ~]# docker info
...
 HTTPS Proxy: http://www.ik8s.io:10080
 No Proxy: 127.0.0.0/8,192.168.147.0/24
...

（4）打开iptables内生的桥接相关功能，已经默认开启了，没开启的自行手动开启

[root@master ~]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables 
1
[root@master ~]# cat /proc/sys/net/bridge/bridge-nf-call-iptables 
1

3、配置启动kubelet服务
（1）修改配置文件

[root@master ~]# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
KUBE_PROXY=MODE=ipvs   ##指明kube_proxy使用ipvs模式
　　

（2）先设为开机自启

[root@master ~]# systemctl enable kubelet.service

因为K8S集群还未初始化，所以kubelet 服务启动不成功，下面初始化完成，再启动即可。

二、初始化kubernetes master节点

在master服务器上执行，完成以下所有操作
1、使用kubeadm init初始化

kubeadm init --kubernetes-version=v1.18.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap

初始化过程报错：
提示不能拉取镜像，连接拒绝。

解决方法
这可能跟“科学上网”有关，由于kubeadm 默认从官网k8s.grc.io下载所需镜像，而国内无法访问。因此需要把上面在docker的service文件中设置的代理注释或删除掉，同时需要通过–image-repository指定阿里云镜像仓库地址。
（1）把docker的service文件中设置的代理注释掉

[root@master ~]# vim  /usr/lib/systemd/system/docker.service 
[Service]

#Environment="HTTPS_PROXY=http://www.ik8s.io:10080"
#Environment="NO_PROXY=127.0.0.0/8,192.168.147.0/24"

（2）初始化时通过–image-repository指定阿里云镜像仓库地址

[root@master ~]# kubeadm init --kubernetes-version=v1.18.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap --image-repository registry.aliyuncs.com/google_containers

以下是集群初始化时的信息：

W0521 15:14:11.969194   20427 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.10.20.207]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [10.10.20.207 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [10.10.20.207 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0521 15:22:09.789434   20427 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0521 15:22:09.790837   20427 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 59.514238 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: nkxbv5.ojapt3936s6o76fk
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.20.207:6443 --token nkxbv5.ojapt3936s6o76fk \
    --discovery-token-ca-cert-hash sha256:38be88afcafafe027512e1f8e7dcb669a990ee32c069acce37badef2b019e393

记录生成的最后部分内容，此内容需要在其它节点加入Kubernetes集群时执行。最好复制保存到某个文件中。

然后根据初始化时的提示信息在用户的家目录下创建一个隐藏文件.kube，并复制配置文件到该目录下，由于我们使用的是root用户，所以简化一下步骤。

[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

2、 验证
使用docker image ls列出拉取下来的镜像

[root@master ~]# docker image ls
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy                v1.18.2             0d40868643c6        4 weeks ago         117MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.18.2             6ed75ad404bd        4 weeks ago         173MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.18.2             ace0a8c17ba9        4 weeks ago         162MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.18.2             a3099161e137        4 weeks ago         95.3MB
registry.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        3 months ago        683kB
registry.aliyuncs.com/google_containers/coredns                   1.6.7               67da37a9a360        3 months ago        43.8MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        6 months ago        288MB

验证kube-apiserver 的6443端口

[root@master ~]# ss -ntl | grep 6443
LISTEN     0      128         :::6443                    :::*      

使用kubectl命令查询集群信息

查询组件状态信息

[root@master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
etcd-0               Healthy   {"health":"true"}   
scheduler            Healthy   ok                  
controller-manager   Healthy   ok      

查询集群节点信息（因为还没有部署好flannel，所以节点显示为NotReady）

[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   18h   v1.18.2

查询名称空间

[root@master ~]# kubectl get ns
NAME              STATUS   AGE
default           Active   18h
kube-node-lease   Active   18h
kube-public       Active   18h
kube-system       Active   18h

三、部署网络插件flannel

1、直接使用kubectl 执行github上的flannel 部署文件

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

但是直接使用kubectl apply -f声明式部署flannel时，显示无法连接，使用wget下载也是显示拒绝连接。

解决方法：
经排查发现是DNS服务的问题，之前默认的DNS服务无法解析raw.githubusercontent.com，因此这里重新指定了DNS服务

[root@master ~]# vim /etc/resolv.conf 

# Generated by NetworkManager
#nameserver 10.10.100.3
#nameserver 10.10.100.5
#nameserver 202.101.172.35
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
#nameserver 202.96.96.236
nameserver 114.114.114.114
[root@master ~]# dig @114.114.114.114 raw.githubusercontent.com

; <<>> DiG 9.9.4-RedHat-9.9.4-50.el7 <<>> @114.114.114.114 raw.githubusercontent.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24452
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;raw.githubusercontent.com.	IN	A

;; ANSWER SECTION:
raw.githubusercontent.com. 38	IN	CNAME	github.map.fastly.net.
github.map.fastly.net.	38	IN	A	151.101.108.133

;; Query time: 12 msec
;; SERVER: 114.114.114.114#53(114.114.114.114)
;; WHEN: 五 5月 22 13:51:59 CST 2020
;; MSG SIZE  rcvd: 105

再次使用kubectl apply -f部署

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

会看到下载好的flannel 的镜像

[root@master ~]#  docker image ls |grep flannel
quay.io/coreos/flannel                                            v0.12.0-amd64       4e9f801d2217        2 months ago        52.8MB

验证
master 节点已经Ready

[root@master ~]# kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   22h   v1.18.2

查询kube-system名称空间下

[root@master ~]#  kubectl get pods -n kube-system |grep flannel
kube-flannel-ds-amd64-h6pnc      1/1     Running   0          12m

四、初始化kubernetes node节点

在2个node 服务器上执行，完成以下所有操作

1、使用kubeadm join 初始化
（1）初始化node节点；下边的命令是master初始化完成后，下边有提示的操作命令

[root@node1 ~]# kubeadm join 10.10.20.207:6443 --token nkxbv5.ojapt3936s6o76fk \
>     --discovery-token-ca-cert-hash sha256:38be88afcafafe027512e1f8e7dcb669a990ee32c069acce37badef2b019e393

node节点初始化报错

　
解决方法：
根据提示信息：（1）swap分区没有关闭。（2）iptables内建的桥接相关功能没有开启，那么我们这里就关闭swap分区，开启iptables内建的桥接相关功能即可。

[root@node1 ~]# swapoff -a
[root@node1 ~]# echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
[root@node1 ~]# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

重新初始化node节点

[root@node1 ~]# kubeadm join 10.10.20.207:6443 --token nkxbv5.ojapt3936s6o76fk     --discovery-token-ca-cert-hash sha256:38be88afcafafe027512e1f8e7dcb669a990ee32c069acce37badef2b019e393
W0522 14:17:31.522847    7484 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

2 、验证集群是否初始化成功
（1）查询2个节点的镜像，node节点也会拉取kube-proxy、flannel、pause镜像，因此可能会花费点时间，耐心等待。

[root@node2 ~]# docker image ls
REPOSITORY                                           TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy   v1.18.2             0d40868643c6        5 weeks ago         117MB
quay.io/coreos/flannel                               v0.12.0-amd64       4e9f801d2217        2 months ago        52.8MB
registry.aliyuncs.com/google_containers/pause        3.2                 80d28bedfe5d        3 months ago        683kB

（2）等2个从节点上下载好镜像，初始化完成，再在主上查询验证

[root@master ~]# kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   23h     v1.18.2
node1    Ready    <none>   8m38s   v1.18.2
node2    Ready    <none>   5m53s   v1.18.2

（3）在主节点查询kube-system名称空间下关于pod的信息

[root@master ~]# kubectl get pods -n kube-system -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP                NODE     NOMINATED NODE   READINESS GATES
coredns-7ff77c879f-5nrtt         1/1     Running   0          24h    10.244.0.3        master   <none>           <none>
coredns-7ff77c879f-wltqv         1/1     Running   0          24h    10.244.0.2        master   <none>           <none>
etcd-master                      1/1     Running   2          25h    192.168.147.132   master   <none>           <none>
kube-apiserver-master            1/1     Running   2          25h    192.168.147.132   master   <none>           <none>
kube-controller-manager-master   1/1     Running   5          25h    192.168.147.132   master   <none>           <none>
kube-flannel-ds-amd64-4pnvr      1/1     Running   0          125m   192.168.147.133   node1    <none>           <none>
kube-flannel-ds-amd64-h6pnc      1/1     Running   0          158m   192.168.147.132   master   <none>           <none>
kube-flannel-ds-amd64-mtrtp      1/1     Running   0          122m   192.168.147.134   node2    <none>           <none>
kube-proxy-8kk8t                 1/1     Running   1          24h    192.168.147.132   master   <none>           <none>
kube-proxy-f6znm                 1/1     Running   0          122m   192.168.147.134   node2    <none>           <none>
kube-proxy-mj5vv                 1/1     Running   0          125m   192.168.147.133   node1    <none>           <none>
kube-scheduler-master            1/1     Running   5          25h    192.168.147.132   master   <none>           <none>

至此，kubernetes集群已经搭建安装完成；kubeadm帮助我们在后台完成了所有操作。

这里有一个拉取镜像的小脚本。
拉取镜像
某些镜像是在google的网站上才能下载的，大多数时候我们访问不到，鉴于此，需要提前把镜像下载下来。
下面是一个通用的镜像下载脚本。

cat ./pull.sh
for i in `kubeadm config images list`; do   
	imageName=${i#k8s.gcr.io/}  
	docker pull registry.aliyuncs.com/google_containers/$imageName  	
	docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName  
	docker rmi registry.aliyuncs.com/google_containers/$imageName
done;