kubeadm部署kubernetes-1.21.1

安装说明虽然K8s 1.20版本宣布将在1.23版本之后将不再维护dockershim，意味着K8s将不直接支持Docker，不过大家不必过于担心。一是在1.23版本之前我们仍然可以使用Docker，二是dockershim肯定会有人接盘，我们同样可以使用Docker，三是Docker制作的镜像仍然可以在其他Runtime环境中使用，所以大家不必过于恐慌。本次安装采用的是Kubeadm安装工具，安

呵~女人

1969人浏览 · 2021-06-07 15:44:37

呵~女人 · 2021-06-07 15:44:37 发布

安装说明

虽然K8s 1.20版本宣布将在1.23版本之后将不再维护dockershim，意味着K8s将不直接支持Docker，不过大家不必过于担心。一是在1.23版本之前我们仍然可以使用Docker，二是dockershim肯定会有人接盘，我们同样可以使用Docker，三是Docker制作的镜像仍然可以在其他Runtime环境中使用，所以大家不必过于恐慌。

本次安装采用的是Kubeadm安装工具，安装版本是K8s 1.21.1，采用的系统为CentOS 7.9，其中Master节点1台，Node节点2台,高可用的配置我会写在文章后面。

节点规划

192.168.183.21 master1
192.168.183.24 node1
192.168.183.25 node2

版本说明

docker version 19.03…x
kubernetes version 1.21.1
centos version 7.9

基本配置

所有节点配置hosts

[root@master1 ~]# cat /etc/hosts
192.168.183.21 master1
192.168.183.24 node1
192.168.183.25 node2

yum源配置

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
yum clean all
yum makecache

必备工具安装

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

所有节点关闭防火墙、selinux、dnsmasq、swap。服务器配置如下

systemctl stop firewalld
systemctl disable firewalld 
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

关闭swap分区

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

同步时间

安装ntpdate

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

所有节点同步时间。时间同步配置如下

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com

写一个计划任务，每五分钟同步一次时间

crontab -e

*/5 * * * * ntpdate time2.aliyun.com

所有节点配置limit：

ulimit -SHn 65535

vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

master1节点免密钥登录其他节点

ssh-keygen -t rsa
for i in master1 node1 node2;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

所有节点升级系统并重启

yum update -y  && reboot

内核配置

所有节点安装ipvsadm

yum install ipvsadm ipset sysstat conntrack libseccomp -y

所有节点配置ipvs模块

vim /etc/modules-load.d/ipvs.conf

ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

加载内核配置

systemctl enable --now systemd-modules-load.service

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

如果加载失败会没有ip_vs_fo模块，将其注释掉重新加载

基本组件安装

所有节点安装Docker-ce 19.03

yum install docker-ce-19.03.* -y

所有节点设置开机自启动Docker

systemctl daemon-reload && systemctl start docker && systemctl enable docker

配置docker加速

vim /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://jzngeu7d.mirror.aliyuncs.com"]
}

安装k8s组件

yum list kubeadm.x86_64 --showduplicates | sort -r

所有节点安装最新版本kubeadm

yum install kubeadm -y

设置Kubelet开机自启动

systemctl daemon-reload
systemctl enable kubelet

集群初始化

生产kubeadmin-config.yaml文件

 kubeadm config print init-defaults > kubeadm-config.yaml

cat kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.183.21
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: master1
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.21.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

下载镜像

kubeadm config images pull --config /root/yaml/kubeadmin-config.yaml
如果下载失败，请更改镜像地址
registry.cn-hangzhou.aliyuncs.com/google_containers

所有节点设置开机自启动kubelet

systemctl enable kubelet（如果启动失败无需管理，初始化成功以后即可启动）

master1节点初始化

kubeadm init --config /root/yaml/kubeadmin-config.yaml --upload-certs

初始化成功以后，会产生Token值，用于其他节点加入时使用，因此要记录下初始化成功生成的token值（令牌值）

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.183.21:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:cafecdaa3053860f1d94d4cae6abdcc99ba6702e8c7ff1ae348a283fce18790e

master1节点配置环境变量，用于访问Kubernetes集群

cat <<EOF >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc

master1上执行

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

node节点执行

kubeadm join 192.168.183.21:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:cafecdaa3053860f1d94d4cae6abdcc99ba6702e8c7ff1ae348a283fce18790e

查看集群状态

kubectl get nodes
目前都为NotReady状态，因为目前还未安装flannel网络

查看cs状态

[root@master1 yaml]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}

如果为下图状态

在这里插入图片描述

解决方法

修改以下配置文件
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
将两个文件中的
- --port=0
这一行注释掉

安装flannel

curl -o kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@master1 yaml]#  kubectl apply -f kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged configured
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

再次查看node状态

[root@master1 yaml]# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
master1   Ready    control-plane,master   45h   v1.21.1
node1     Ready    <none>                 45h   v1.21.1
node2     Ready    <none>                 45h   v1.21.1

部署flannel网络插件时发现flannel一直处于CrashLoopBackOff状态，查看日志提示没有分配cidr

解决

vim /etc/kubernetes/manifests/kube-controller-manager.yaml
增加参数：
--allocate-node-cidrs=true
--cluster-cidr=10.244.0.0/16
重启kubelet
systemctl restart kubelet

添加自动补全脚本到系统

echo "source <(kubectl completion bash)" >> ~/.bashrc

如果不适用flannel网络，使用Calico的话，安装步骤如下

以下步骤只在master1执行

cd /root/ ; git clone https://github.com/dotbalo/k8s-ha-install.git
cd /root/k8s-ha-install && git checkout manual-installation-v1.20.x && cd calico/

修改calico-etcd.yaml的以下位置

sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.0.201:2379,https://192.168.0.202:2379,https://192.168.183.21:2379"#g' calico-etcd.yaml


ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml


sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml

POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`

sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

创建calico

kubectl apply -f calico-etcd.yaml

注意：git clone https://github.com/dotbalo/k8s-ha-install.git这个基本上下不下来，需要翻墙去下载

Metrics Server部署

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率

将master1节点的front-proxy-ca.crt复制到所有Node节点

scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node01:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node(其他节点自行拷贝):/etc/kubernetes/pki/front-proxy-ca.crt

安装metrics server

此文件还是在刚刚clone下的路径下

cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/
[root@master1 yaml]# kubectl  create -f comp.yaml 
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

等待kube-system命令空间下的Pod全部启动后，查看状态

[root@master1 yaml]# kubectl  top node
W0607 14:40:59.446566   67589 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master1   156m         7%     1193Mi          69%       
node1     77m          3%     743Mi           43%       
node2     210m         10%    994Mi           57%

部署dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml
如果yaml文件下载不下来，请到浏览器粘贴复制，或者使用刚刚clone下的文件，里面有dashboard的yaml文件

查看pod运行状态

[root@master1 yaml]# kubectl get pod -n kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-856586f554-cvfwr   1/1     Running   0          4h3m
kubernetes-dashboard-7795fd6d89-bdstf        1/1     Running   0          4h3m

修改Dashboard,通过NodePort方式暴露端口,这里指定30001端口

sed -i '/targetPort:/a\ \ \ \ \ \ nodePort: 30001\n\ \ type: NodePort' kubernetes-dashboard.yaml

选择token方式

创建dashboard-adminuser.yaml：

cat > dashboard-adminuser.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard  
EOF

创建登录用户

kubectl apply -f dashboard-adminuser.yaml

说明

上面创建了一个叫admin-user的服务账号，并放在kubernetes-dashboard 命名空间下，并将cluster-admin角色绑定到admin-user账户，这样admin-user账户就有了管理员的权限。默认情况下，kubeadm创建集群时已经创建了cluster-admin角色，我们直接绑定即可

查看admin-user账户的token

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

登录：https://192.168.183.21:30001

在这里插入图片描述

如果你部署的是高可用集群的话，配置如下

组件安装

所有Master节点通过yum安装HAProxy和KeepAlived

yum install keepalived haproxy -y

所有Master节点配置HAProxy（所有Master节点的HAProxy配置相同）

vim /etc/haproxy/haproxy.cfg

global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.183.21:6443  check
  server k8s-master02	192.168.183.22:6443  check
  server k8s-master03	192.168.183.23:6443  check

所有Master节点配置KeepAlived，配置不一样，注意区分

注意每个节点的IP和网卡（interface参数)

Master1节点的配置

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.183.21
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        虚拟ip的地址
    }
#    track_script {
#       chk_apiserver
#    }
}

Master2节点的配置

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.183.22
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        虚拟ip的地址
    }
#    track_script {
#       chk_apiserver
#    }
}

Master3节点的配置

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.183.23
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        虚拟ip的地址
    }
#    track_script {
#       chk_apiserver
#    }
}

注意上述的健康检查是关闭的，集群建立完成后再开启

配置KeepAlived健康检查文件

vim /etc/keepalived/check_apiserver.sh

#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

添加执行权限：chmod +x /etc/keepalived/check_apiserver.sh
启动haproxy和keepalived
[root@k8s-master01 keepalived]# systemctl daemon-reload
[root@k8s-master01 keepalived]# systemctl enable --now haproxy
[root@k8s-master01 keepalived]# systemctl enable --now keepalived

测试VIP

root@k8s-master01 ~]# ping vip -c 4
PING 192.168.0.236 (192.168.0.236) 56(84) bytes of data.
64 bytes from vip: icmp_seq=1 ttl=64 time=0.464 ms
64 bytes from vip: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from vip: icmp_seq=3 ttl=64 time=0.062 ms
64 bytes from vip: icmp_seq=4 ttl=64 time=0.063 ms

集群初始化配置文件

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.201
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - vip
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: vip:16443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

注意：如果不是高可用集群，vip:16443改为master1的地址，16443改为apiserver的端口，默认是6443，注意更改v1.21.1为自己服务器kubeadm的版本：kubeadm version

将kubeadm-config.yaml文件复制到其他master节点，之后所有Master节点提前下载镜像，可以节省初始化时间

所有节点设置开机自启动kubelet

systemctl enable --now kubelet（如果启动失败无需管理，初始化成功以后即可启动）

初始化

kubeadm init --config kubeadm-config.yaml  --upload-certs

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join vip:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:9c87ecb666be3b372851a9af2c7ca1f7f60c12c68f6ffe1eb513791a1b8a776 \
    --control-plane --certificate-key ac2854defdflpgbdsepw232fdsw1fd2kmk34km54lk5ml456mlk56l56l56l5

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join vip:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:dfd4354kk4k56jknhj7657jds0f0dgfjr34o5j09gf904f0derf948d9g9234ji5

master节点加入

kubeadm join vip:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:9c87ecb666be3b372851a9af2c7ca1f7f60c12c68f6ffe1eb513791a1b8a776 \
    --control-plane --certificate-key ac2854defdflpgbdsepw232fdsw1fd2kmk34km54lk5ml456mlk56l56l56l5ds

node节点加入

kubeadm join vip:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:dfd4354kk4k56jknhj7657jds0f0dgfjr34o5j09gf904f0derf948d9g9234ji5

分别在三个master节点上初始化，也可以将master1上的证书拷贝到另外两个master上

scp /etc/kubernetes/pki/ca.crt master2:/etc/kubernetes/pki/ca.crt
scp /etc/kubernetes/pki/ca.key master2:/etc/kubernetes/pki/ca.key
scp /etc/kubernetes/pki/sa.key master2:/etc/kubernetes/pki/sa.key
scp /etc/kubernetes/pki/sa.pub master2:/etc/kubernetes/pki/sa.pub
scp /etc/kubernetes/pki/front-proxy-ca.crt master2:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.key master2:/etc/kubernetes/pki/front-proxy-ca.key
scp /etc/kubernetes/pki/etcd/ca.crt master2:/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/etcd/ca.key master2:/etc/kubernetes/pki/etcd/ca.key
scp /etc/kubernetes/admin.conf master2:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf master2:~/.kube/config
#################拷贝到master2###########################
scp /etc/kubernetes/pki/ca.crt master3:/etc/kubernetes/pki/ca.crt
scp /etc/kubernetes/pki/ca.key master3:/etc/kubernetes/pki/ca.key
scp /etc/kubernetes/pki/sa.key master3:/etc/kubernetes/pki/sa.key
scp /etc/kubernetes/pki/sa.pub master3:/etc/kubernetes/pki/sa.pub
scp /etc/kubernetes/pki/front-proxy-ca.crt master3:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.key master3:/etc/kubernetes/pki/front-proxy-ca.key
scp /etc/kubernetes/pki/etcd/ca.crt master3:/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/etcd/ca.key master3:/etc/kubernetes/pki/etcd/ca.key
scp /etc/kubernetes/admin.conf master3:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf master3:~/.kube/config

测试集群是否可用

在kubernetes集群中创建一个pod，然后暴露端口，验证是否正常访问

[root@master1 yaml]#  kubectl create deployment nginx-deploy --image=nginx
[root@master1 yaml]#  kubectl expose deployment nginx-deploy --port=80 --type=NodePort
[root@master1 yaml]#  kubectl get pod,svc
NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-deploy-8588f9dfb-v8x4j   1/1     Running   0          14m

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP        47h
service/nginx-deploy   NodePort    10.99.172.209   <none>        80:30703/TCP   14m

在这里插入图片描述

Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。一、云计算历经十余年

Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松今天的内容会从以下几个方面展开：负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点一、产品简介 1. 以NGINX为例提到负载均衡，我们以

Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这