使用kubeadm安装配置Kubernetes集群

（个人笔记）
安装前准备

1. 准备服务器(所有主机操作)
   修改主机名：
   hostnamectl set-hostname master

hostnamectl set-hostname node01

hostnamectl set-hostname node02

修改所有节点的hosts

192.168.20.111 master
192.168.20.112 node01
192.168.20.113 node02

安装准备

1.1禁用防火墙：

systemctl stop firewalld
systemctl disable firewalld

1.2禁用SELINUX：

setenforce 0
vi /etc/selinux/config
SELINUX=disabled

1.2.1 关闭系统的Swap
关闭系统的Swap方法如下:
swapoff -a

想永久生效把 vim /etc/fstab 注释 swap 那一行

1.3 创建/etc/sysctl.d/k8s.conf文件，添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

执行如下命令使修改生效：

sysctl -p /etc/sysctl.d/k8s.conf

1.4 kube-proxy开启ipvs的前置条件

由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

在所有的Kubernetes节点node1和node2上执行以下脚本:

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

接下来还需要确保各个节点上已经安装了ipset软件包yum install ipset。 为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm yum install ipvsadm。

如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。


2. 安装docker（所有节点）

一般情况使用下面的方法安装即可
2.1 卸载旧版本(如果有的话)
安装旧版的docker
 
yum remove -y docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-selinux \
                  docker-engine-selinux \
                  docker-engine
				  dokcer-ce-cli 
 
2.2安装指定版本的docker
yum install -y --setopt=obsoletes=0 \
docker-ce-17.03.2.ce-1.el7.centos.x86_64 \
docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch 

2.3 添加加速 创建/etc/docker/daemon.json
cat /etc/docker/daemon.json 
{
  "registry-mirrors":["https://ujztwkm5.mirror.aliyuncs.com"], 
  "storage-driver": "overlay2",
  "storage-opts": ["overlay2.override_kernel_check=true"]

}

2.4启动docker	


systemctl enable docker
systemctl start docker

2.5安装kubeadm  kubectl kubelet  （所有节点）


阿里云的源进行安装： （所有节点）

添加仓库
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

源配置完成后，执行安装命令即可：

yum makecache fast && yum install -y kubelet-1.10.0-0 kubeadm-1.10.0-0 kubectl-1.10.0-0

systemctl enable kubelet 

2.6 配置 kubelet

安装完成后，我们还需要对kubelet进行配置，因为用yum源的方式安装的kubelet生成的配置文件将参数--cgroup-driver改成了systemd，而 docker 的cgroup-driver是cgroupfs，这二者必须一致才行，我们可以通过docker info命令查看：

$ docker info |grep Cgroup
Cgroup Driver: cgroupfs

修改文件 kubelet的配置文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，将其中的KUBELET_CGROUP_ARGS参数更改成cgroupfs：

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

另外还有一个问题是关于交换分区的，之前我们在手动搭建高可用的 kubernetes 集群一文中已经提到过，Kubernetes 从1.8开始要求关闭系统的 Swap ，如果不关闭，默认配置的 kubelet 将无法启动，我们可以通过 kubelet 的启动参数--fail-swap-on=false更改这个限制，所以我们需要在上面的配置文件中增加一项配置(在ExecStart之前)：

Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"


修改完成后，重新加载我们的配置文件即

systemctl daemon-reload
systemctl restart kubelet


3、镜像（能科学上网 的可以不用提前下载）

master节点，执行下面的命令：

docker pull cnych/kube-apiserver-amd64:v1.10.0
docker pull cnych/kube-scheduler-amd64:v1.10.0
docker pull cnych/kube-controller-manager-amd64:v1.10.0
docker pull cnych/kube-proxy-amd64:v1.10.0
docker pull cnych/k8s-dns-kube-dns-amd64:1.14.8
docker pull cnych/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker pull cnych/k8s-dns-sidecar-amd64:1.14.8
docker pull cnych/etcd-amd64:3.1.12
docker pull cnych/flannel:v0.10.0-amd64
docker pull cnych/pause-amd64:3.1

docker tag cnych/kube-apiserver-amd64:v1.10.0 k8s.gcr.io/kube-apiserver-amd64:v1.10.0
docker tag cnych/kube-scheduler-amd64:v1.10.0 k8s.gcr.io/kube-scheduler-amd64:v1.10.0
docker tag cnych/kube-controller-manager-amd64:v1.10.0 k8s.gcr.io/kube-controller-manager-amd64:v1.10.0
docker tag cnych/kube-proxy-amd64:v1.10.0 k8s.gcr.io/kube-proxy-amd64:v1.10.0
docker tag cnych/k8s-dns-kube-dns-amd64:1.14.8 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
docker tag cnych/k8s-dns-dnsmasq-nanny-amd64:1.14.8 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag cnych/k8s-dns-sidecar-amd64:1.14.8 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
docker tag cnych/etcd-amd64:3.1.12 k8s.gcr.io/etcd-amd64:3.1.12
docker tag cnych/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker tag cnych/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1



node01 node02，执行下面的命令：

docker pull cnych/kube-proxy-amd64:v1.10.0
docker pull cnych/flannel:v0.10.0-amd64
docker pull cnych/pause-amd64:3.1
docker pull cnych/kubernetes-dashboard-amd64:v1.8.3
docker pull cnych/heapster-influxdb-amd64:v1.3.3
docker pull cnych/heapster-grafana-amd64:v4.4.3
docker pull cnych/heapster-amd64:v1.4.2
docker pull cnych/k8s-dns-kube-dns-amd64:1.14.8
docker pull cnych/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker pull cnych/k8s-dns-sidecar-amd64:1.14.8

docker tag cnych/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
docker tag cnych/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag cnych/kube-proxy-amd64:v1.10.0 k8s.gcr.io/kube-proxy-amd64:v1.10.0

docker tag cnych/k8s-dns-kube-dns-amd64:1.14.8 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
docker tag cnych/k8s-dns-dnsmasq-nanny-amd64:1.14.8 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag cnych/k8s-dns-sidecar-amd64:1.14.8 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8

docker tag cnych/kubernetes-dashboard-amd64:v1.8.3 k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
docker tag cnych/heapster-influxdb-amd64:v1.3.3 k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
docker tag cnych/heapster-grafana-amd64:v4.4.3 k8s.gcr.io/heapster-grafana-amd64:v4.4.3
docker tag cnych/heapster-amd64:v1.4.2 k8s.gcr.io/heapster-amd64:v1.4.2

上面的这些镜像是在 Node 节点中需要用到的镜像，在 join 节点之前也需要先下载到节点上面。


4、集群安装初始化 

4.1 这里我们的准备工作就完成了，接下来我们就可以在master节点上用kubeadm命令来初始化我们的集群了：

$ kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.111

如果出现running with swap on is not supported. Please disable swap之类的错误，则我们还需要增加一个参数–ignore-preflight-errors=Swap来忽略 swap 的错误提示信息：

kubeadm init  \
--kubernetes-version=v1.10.0 \
--pod-network-cidr=10.244.0.0/16  \
--apiserver-advertise-address=192.168.20.111 
[root@master ~]# kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.111
[init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.20.111]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [master] and IPs [192.168.20.111]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 20.506124 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node master as master by adding a label and a taint
[markmaster] Master master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: h6a9u9.et5nhlqa2techud5
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.20.111:6443 --token 33n63c.cz5kdmipmvsxewe2 --discovery-token-ca-cert-hash sha256:80d158c6c9e5b033f7809e54e3ceb93df371589dbe78fa23dc3b83de5b402b1c



初始化过程说明：

    [preflight] kubeadm 执行初始化前的检查。
    [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
    [certificates] 生成相关的各种token和证书
    [kubeconfig] 生成 KubeConfig 文件，kubelet 需要这个文件与 Master 通信
    [control-plane] 安装 Master 组件，会从指定的 Registry 下载组件的 Docker 镜像。
    [bootstraptoken] 生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
    [addons] 安装附加组件 kube-proxy 和 kube-dns。
    Kubernetes Master 初始化成功，提示如何配置常规用户使用kubectl访问集群。
    提示如何安装 Pod 网络。
    提示如何注册其他节点到 Cluster。
	
###	要注意将上面的加入集群的命令保存下面，如果忘记保存上面的 token 和 sha256 值的话也不用担心，我们可以使用下面的命令来查找：	
	
kubeadm token list


4.2 集群初始化如果遇到问题，可以使用下面的命令进行清理：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/


5、安装Pod Network

[root@master k8s]#wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@master k8s]# kubectl apply  -f kube-flannel.yml 
podsecuritypolicy.extensions "psp.flannel.unprivileged" created
clusterrole.rbac.authorization.k8s.io "flannel" created
clusterrolebinding.rbac.authorization.k8s.io "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset.extensions "kube-flannel-ds-amd64" created
daemonset.extensions "kube-flannel-ds-arm64" created
daemonset.extensions "kube-flannel-ds-arm" created
daemonset.extensions "kube-flannel-ds-ppc64le" created
daemonset.extensions "kube-flannel-ds-s390x" created

另外需要注意的是如果你的节点有多个网卡的话，需要在 kube-flannel.yml 中使用--iface参数指定集群主机内网网卡的名称，否则可能会出现 dns 无法解析。flanneld 启动参数加上--iface=<iface-name>

args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0

安装完成后使用 kubectl get pods 命令可以查看到我们集群中的组件运行状态，如果都是Running 状态的话，那么恭喜你，你的 master 节点安装成功了。


6、将其它节点加入到集群中

[root@node01 ~]#   kubeadm join 192.168.20.111:6443 --token h6a9u9.et5nhlqa2techud5 --discovery-token-ca-cert-hash sha256:39a544c02c763a2304146beda7b8926945a208678c1d555df3de51bbfb5b1d74
[preflight] Running pre-flight checks.
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
	[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[discovery] Trying to connect to API Server "192.168.20.111:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.20.111:6443"
[discovery] Requesting info from "https://192.168.20.111:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.20.111:6443"
[discovery] Successfully established connection with API Server "192.168.20.111:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.


[root@node02 ~]# kubeadm join 192.168.20.111:6443 --token 12ywe4.1i7e2swkk626iy26 --discovery-token-ca-cert-hash sha256:96554547cf5ce2821735122e6b9c068736fefa3a65b267c07b33a494b83cf11c
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.20.111:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.20.111:6443"
[discovery] Requesting info from "https://192.168.20.111:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.20.111:6443"
[discovery] Successfully established connection with API Server "192.168.20.111:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

注意：如果遇到无法加入集群 执行kubectl rset  再重新加入
注意记录下初始化结果中的kubeadm join命令，部署worker节点时会用到


--------------------- 


我们可以看到该节点已经加入到集群中去了，然后我们把 master 节点的~/.kube/config文件拷贝到当前节点对应的位置即可使用 kubectl 命令行工具了。


配置kubectl

#创建普通用户并设置密码123456 
useradd centos && echo "centos:123456" | chpasswd centos 

#追加sudo权限,并配置sudo免密 
sed -i '/^root/a\centos  ALL=(ALL)       NOPASSWD:ALL' /etc/sudoers 

#保存集群安全配置文件到当前用户.kube目录 
su - centos mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config 

#启用 kubectl 命令自动补全功能（注销重新登录生效） 
echo "source <(kubectl completion bash)" >> ~/.bashrc
--------------------- 



日常操作：

查看集群状态

[root@master ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   



查看集群节点证书信息
[root@master ~]# kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
csr-kt5xm                                              25m       system:node:master        Approved,Issued
node-csr-6ad96lILrjuq--dJpuybk17c4N41Fk18QKcKiuFadyg   1m        system:bootstrap:bc181x   Approved,Issued
node-csr-T0QQetsqGZU5y8eSkXM2YyXvtXHocJflMZdL7gQrzn8   2m        system:bootstrap:bc181x   Approved,Issued

 
查看Pod状态

 kubectl get pods -n kube-system -o wide
[root@master k8s]# kubectl get pods --all-namespaces -o wide 
NAMESPACE     NAME                             READY     STATUS                  RESTARTS   AGE
kube-system   etcd-master                      1/1       Running                 0          29m
kube-system   kube-apiserver-master            1/1       Running                 0          29m
kube-system   kube-controller-manager-master   1/1       Running                 0          29m
kube-system   kube-dns-86f4d74b45-6g945        0/3       Pending                 0          52m
kube-system   kube-flannel-ds-amd64-8cnqj      0/1       Init:ImagePullBackOff   0          18m
kube-system   kube-flannel-ds-amd64-cfzbj      0/1       Init:ImagePullBackOff   0          18m
kube-system   kube-flannel-ds-amd64-g469z      0/1       Init:ImagePullBackOff   0          18m
kube-system   kube-proxy-f686b                 1/1       Running                 0          29m
kube-system   kube-proxy-hlpgx                 1/1       Running                 0          28m
kube-system   kube-proxy-v8776                 1/1       Running                 0          52m
kube-system   kube-scheduler-master            1/1       Running                 0          29m

kube-dns-86f4d74b45-6g945 处于  Pending  状态
kube-flannel-ds-amd64-g469z  处于 Init:ImagePullBackOff  状态
说明这个pod的镜像在对应节点上拉取失败，我们可以通过 kubectl describe pod 查看 Pod 具体情况，以确认拉取失败的镜像：
想查看更多信息，可以 describe 这个失败的 Pod：
kubectl describe pod -n kube-system kube-flannel-ds-amd64-b9smv   



注意：如果出现Error、Pending、ImagePullBackOff、CrashLoopBackOff都属于启动失败的Pod，原因需要仔细排除

a、查看 /var/log/messages系统日志

b、kubectl describe pod kube-dns-86f4d74b45-6g945 --namespace=kube-system 

c、kubectl logs -f kube-flannel-ds-amd64-b9smv -n kube-system kubedns

d、kubectl logs -f kube-flannel-ds-amd64-fd89z -n kube-system 




Pod调度到Master节点

出于安全考虑，默认配置下Kubernetes不会将Pod调度到Master节点。查看Taints字段默认配置：

[centos@k8s-master ~]$ kubectl describe node master 
......
Taints:             node-role.kubernetes.io/master:NoSchedule

如果希望将master也当作Node节点使用，可以执行如下命令,其中master是主机节点hostname：

kubectl taint node master node-role.kubernetes.io/master-

修改后Taints字段状态：

[centos@k8s-master ~]$ kubectl describe node master                             
......
Taints:             <none>


如果要恢复Master Only状态，执行如下命令：

kubectl taint node master node-role.kubernetes.io/master=:NoSchedule


移除节点和集群

kubernetes集群移除节点
以移除node2节点为例，在Master节点上运行：
kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2
上面两条命令执行完成后，在node2节点执行清理命令，重置kubeadm的安装状态：
kubeadm reset
在master上删除node并不会清理node2运行的容器，需要在删除节点上面手动运行清理命令。
如果你想重新配置集群，使用新的参数重新运行kubeadm init或者kubeadm join即可。

至此3个节点的集群搭建完成，后续可以继续添加node节点，或者部署dashboard、helm包管理工具、EFK日志系统、Prometheus Operator监控系统、rook+ceph存储系统等组件。
--------------------- 








遇到问题可以参考：官方教程

1、Error starting daemon: error initializing graphdriver: driver not supported

错误原因：error initializing graphdriver: driver not supported
解决办法：
在 /etc/docker 目录下创建daemon.json文件，并且加入以下配置

touch daemon.json
vi daemon.json
{
  "storage-driver": "overlay2",
  "storage-opts": [
   "overlay2.override_kernel_check=true"
  ]
}

2、running with swap on is not supported. Please disable swap
禁止 SWAP

swapoff -a

想永久生效把 vim /etc/fstab 注释 swap 那一行


3、this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
解决办法：把docker版本降到17.03


4、[ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: fork/exec /usr/bin/crictl -r /var/run/dockershim.sock info: no such file or directory
解决办法：

卸载 cri-tools

apt-get remove cri-tools
yum remove cri-tools


5、Error syncing pod cc476352-47d2-11e9-9093-005056b2366f ("kube-flannel-ds-amd64-fd89z_kube-system(cc476352-47d2-11e9-9093-005056b2366f)"), skipping: failed to "StartContainer" for "install-cni" with ImagePullBackOff: "Back-off pulling image \"quay.io/coreos/flannel:v0.11.0-amd64\""
解决办法:
原来打的TAG为V:0.10.0  把tag打成V:0.11.0即可

docker tag cnych/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64


问题1
#初始化的时候,报kubelet
[root@master yum.repos.d]# kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.111
[init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
	[WARNING Service-Kubelet]: kubelet service does not exist
[preflight] Some fatal errors occurred:
	[ERROR KubeletVersion]: couldn't get kubelet version: executable file not found in $PATH
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`


解决：卸载kubelet  重装kubelet
yum remove -y kubelet 
yum install -y kubeadm-1.10.0-0.x86_64 kubelet-1.10.0-0.x86_64  kubernetes-cni-0.6.0-0.x86_64


Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

执行下面的命令：

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

	  
	  
问题2	  
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

解决办法：
 systemctl restart kubelet
 
问题3
	[ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: fork/exec /usr/bin/crictl -r /var/run/dockershim.sock info: no such file or directory

解决办法：卸载cri-tools

yum remove -y cri-tools