Kubernetes----kubeadm初始化集群

一、系统环境virtualboxcentos7（CentOS-7-x86_64-Minimal-1810）配置如下：master：2CPU、2G内存、20G磁盘、桥接网络node1：2CPU、4G内存、20G磁盘、桥接网络摘自virtualbox安装centos7教程示例...

cyfblog

2659人浏览 · 2019-07-29 20:10:08

cyfblog · 2019-07-29 20:10:08 发布

kubeadm初始化k8s集群

一、虚拟机配置
二、系统配置
三、下载必要组件
- 1、master节点
- 2、node节点
四、配置docker和kubelet
- 1、修改docker Cgroup Driver
- 2、配置kubelet忽略Swap
五、kubeadm初始化集群
- 1、master节点初始化：kubeadm init
- 2、node节点加入集群
六、错误总结

一、虚拟机配置

Virtualbox
Centos7

配置如下：

master节点：2CPU、3G内存、40G磁盘、桥接网络
node节点：2CPU、3G内存、40G磁盘、桥接网络

二、系统配置

1、关闭防火墙

# 关闭防火墙，并禁止开机启动
[root@k8s-master ~]# systemctl stop firewalld.service && systemctl disable firewalld.service

2、关闭swap

echo "vm.swappiness = 0">> /etc/sysctl.conf 
sysctl -p

3、加载br_netfilter

[root@k8s-master ~]# modprobe br_netfilter

方法一：配置sysctl.conf文件

[root@k8s-master ~]# cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-master ~]# sysctl -p
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

方法二：直接修改bridge-nf-call-ip6tables和bridge-nf-call-iptables文件

[root@k8s-master ~]# ls /proc/sys/net/bridge/bridge-nf-call-ip6tables
1
[root@k8s-master ~]# ls /proc/sys/net/bridge/bridge-nf-call-iptables
1

4、配置ipv4转发

echo 1 > /proc/sys/net/ipv4/ip_forward
sysctl -p

5、关闭SELinux

[root@k8s-master ~]# sed -i s/enforcing/disabled/g /etc/selinux/config
[root@k8s-master ~]# reboot

6、hostname解析

[root@k8s-master ~]# cat >> /etc/hosts <<EOF
192.168.10.250 master
192.168.10.251 node1
192.168.10.252 node2
EOF

三、下载必要组件

1、master节点

kubelet、kubeadm、kubectl、docker-ce、docker-ce-cli

[root@k8s-master ~]# yum install -y wget
[root@k8s-master ~]# wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@k8s-master ~]# cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg \
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@k8s-master ~]# yum install -y kubelet kubeadm kubectl docker-ce docker-ce-cli

# 下载制定版本
yum install -y kubelet-1.14.2 kubeadm-1.14.2 kubectl-1.14.2 docker-ce-18.09.1 docker-ce-cli-18.09.1

2、node节点

kubelet、kubeadm、docker-ce

[root@k8s-node1 ~]# yum install -y wget
[root@k8s-node1 ~]# wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@k8s-node1 ~]# cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg \
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@k8s-node1 ~]# yum install -y kubelet kubeadm docker-ce docker-ce-cli

四、配置docker和kubelet

1、修改docker Cgroup Driver

启动docker，并设置开机自启

[root@k8s-master ~]# systemctl start docker && systemctl enable docker

查看docker驱动，docker服务的Cgroup Driver默认值为cgroupfs

[root@k8s-master ~]# docker info
...
Cgroup Driver: cgroupfs 
...

当我们用kubeadm初始化的时候会出现如下警告：

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:

需要修改Cgroup Driver的值为systemd

[root@k8s-master ~]# cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
[root@k8s-master ~]# systemctl daemon-reload && systemctl restart docker

2、配置kubelet忽略Swap

由于我系统资源不够忽略了swap，安装完kubelet后配置kubelet:

[root@k8s-master ~]# rpm -ql kubelet 
/etc/kubernetes/manifests
/etc/sysconfig/kubelet
/usr/bin/kubelet
/usr/lib/systemd/system/kubelet.service
[root@k8s-master ~]# cat > /etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
EOF
[root@k8s-master ~]# systemctl enable kubelet

五、kubeadm初始化集群

1、master节点初始化：kubeadm init

可以指定k8s部署版本--kubernetes-version

[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.14.2 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
	[WARNING Swap]: running with swap on is not supported. Please disable swap
	[WARNING Hostname]: hostname "k8s-master" could not be reached
	[WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 114.114.114.114:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.10.250 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.10.250 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.250]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.004277 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: gwt0bu.3k3eic6183wxtacs
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.250:6443 --token gwt0bu.3k3eic6183wxtacs \
    --discovery-token-ca-cert-hash sha256:4339a6368879ad46c6d91c363d00d1c4dc8222f27d76f9d7741b3a4be4ab4ac6
    
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master ~]# ss -tln | grep 6443 #查看6443端口是否开启
LISTEN     0      128         :::6443                    :::*

提示：如果忘记加入集群命令，使用如下命令：

[root@k8s-master ~]# kubeadm token create --print-join-command
kubeadm join 192.168.10.250:6443 --token 0pm7y8.2k4w73xxtn35gp11     --discovery-token-ca-cert-hash sha256:4339a6368879ad46c6d91c363d00d1c4dc8222f27d76f9d7741b3a4be4ab4ac6

此时查看node状态NotReady，需要配置flannel组件

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   master   51s   v1.14.2
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

此时查看pod状态，都已经启动完毕，master节点初始化完毕

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-8686dcc4fd-2csrf             1/1     Running   0          10m
kube-system   coredns-8686dcc4fd-thssb             1/1     Running   0          10m
kube-system   etcd-k8s-master                      1/1     Running   0          9m27s
kube-system   kube-apiserver-k8s-master            1/1     Running   0          9m31s
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          9m26s
kube-system   kube-flannel-ds-amd64-crghf          1/1     Running   0          7m28s
kube-system   kube-flannel-ds-amd64-sdqts          1/1     Running   0          9m29s
kube-system   kube-proxy-d4v6t                     1/1     Running   0          7m28s
kube-system   kube-proxy-sjx66                     1/1     Running   0          10m
kube-system   kube-scheduler-k8s-master            1/1     Running   0          9m14s

2、node节点加入集群

由于我没有关闭swap，需要使用--ignore-preflight-errors忽略swap警告（注意Swap首字母大写）

[root@k8s-node1 ~]# kubeadm join 192.168.10.250:6443 --token gwt0bu.3k3eic6183wxtacs     --discovery-token-ca-cert-hash sha256:4339a6368879ad46c6d91c363d00d1c4dc8222f27d76f9d7741b3a4be4ab4ac6 --ignore-preflight-errors=Swap
[preflight] Running pre-flight checks
	[WARNING Swap]: running with swap on is not supported. Please disable swap
	[WARNING Hostname]: hostname "k8s-node1" could not be reached
	[WARNING Hostname]: hostname "k8s-node1": lookup k8s-node1 on 114.114.114.114:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

此时从master节点查看节点状态Ready，此时加入节点即为成功

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   Ready      master   3m43s   v1.14.2
k8s-node1    NotReady   <none>   33s     v1.14.2

此时ROLES为<none>状态，修改node节点角色

[root@k8s-master ~]# kubectl label node k8s-node1 node-role.kubernetes.io/worker=worker
node/k8s-node1 labeled
[root@k8s-master ~]# kubectl get nodes 
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   9m56s   v1.14.2
k8s-node1    Ready    worker   6m46s   v1.14.2

查看所有组件启动状态，均为running状态即可

[root@k8s-master ~]# kubectl get pods -A -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
kube-system   coredns-8686dcc4fd-2csrf             1/1     Running   0          9m53s   10.244.0.2       k8s-master   <none>           <none>
kube-system   coredns-8686dcc4fd-thssb             1/1     Running   0          9m53s   10.244.0.3       k8s-master   <none>           <none>
kube-system   etcd-k8s-master                      1/1     Running   0          9m2s    192.168.10.250   k8s-master   <none>           <none>
kube-system   kube-apiserver-k8s-master            1/1     Running   0          9m6s    192.168.10.250   k8s-master   <none>           <none>
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          9m1s    192.168.10.250   k8s-master   <none>           <none>
kube-system   kube-flannel-ds-amd64-crghf          1/1     Running   0          7m3s    192.168.10.251   k8s-node1    <none>           <none>
kube-system   kube-flannel-ds-amd64-sdqts          1/1     Running   0          9m4s    192.168.10.250   k8s-master   <none>           <none>
kube-system   kube-proxy-d4v6t                     1/1     Running   0          7m3s    192.168.10.251   k8s-node1    <none>           <none>
kube-system   kube-proxy-sjx66                     1/1     Running   0          9m54s   192.168.10.250   k8s-master   <none>           <none>
kube-system   kube-scheduler-k8s-master            1/1     Running   0          8m49s   192.168.10.250   k8s-master   <none>           <none>

至此k8s集群搭建完毕

六、错误总结

1、错误：[WARNING Swap]: running with swap on is not supported. Please disable swap

解决办法：
方式一：
	注释：/dev/mapper/centos-swap swap                    swap    defaults        0 0
	echo "vm.swappiness = 0">> /etc/sysctl.conf
方式二：
	swapoff -a # 临时生效

2、使用国内源安装`kubeadm kubelet kubectl`，报如下错误：

错误：软件包：kubelet-1.14.2-0.x86_64 (kubernetes)
          需要：kubernetes-cni = 0.7.5
          可用: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
              kubernetes-cni = 0.3.0.1-0.07a8a2
          可用: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
              kubernetes-cni = 0.5.1-0
          可用: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
              kubernetes-cni = 0.5.1-1
          可用: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
              kubernetes-cni = 0.6.0-0
          可用: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
              kubernetes-cni = 0.7.5-0
错误：软件包：kubeadm-1.14.2-0.x86_64 (kubernetes)
          需要：kubernetes-cni >= 0.7.5
          可用: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
              kubernetes-cni = 0.3.0.1-0.07a8a2
          可用: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
              kubernetes-cni = 0.5.1-0
          可用: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
              kubernetes-cni = 0.5.1-1
          可用: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
              kubernetes-cni = 0.6.0-0
          可用: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
              kubernetes-cni = 0.7.5-0
 您可以尝试添加 --skip-broken 选项来解决该问题
 您可以尝试执行：rpm -Va --nofiles --nodigest

解决办法：在`yum install`添加`--setopt=obsoletes=0`参数

3、报错：[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

解决办法：echo 1 > /proc/sys/net/ipv4/ip_forward

Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。一、云计算历经十余年

Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松今天的内容会从以下几个方面展开：负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点一、产品简介 1. 以NGINX为例提到负载均衡，我们以

Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这

Cloudpods

所有评论(0)

查看更多评论

cyfblog

@cyfblog

已为社区贡献8条内容