CentOS7 安装部署k8s
1、官网说明：Kubernetes 安装 kubeadm使用kubeadm创建Kubernetes集群2、准备工作直接使用 root 身份关闭防火墙#systemctl stop firewalld#systemctl disable firewalld关闭 swap#swapoff -a#vim /etc/fstab## /etc/fstab# Created by anaconda on Mo
1、官网说明：
2、准备工作
学习和练手，一切从简！
-
直接使用 root 身份
-
关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
setenforce 0sed -i ‘s/^SELINUX=enforcing$/SELINUX=disabled/’ /etc/selinux/config && setenforce 0
或者 vim /etc/selinux/config 修改
-
关闭 swap
swapoff -avim /etc/fstab
# # /etc/fstab # Created by anaconda on Mon Jun 28 23:11:04 2021 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/cl-root / xfs defaults 0 0 UUID=0b4346b6-cee1-4abb-932e-0c1cb4cda404 /boot xfs defaults 0 0 /dev/mapper/cl-home /home xfs defaults 0 0 # wzh 20211026 for k8s # /dev/mapper/cl-swap swap swap defaults 0 0 -
修改并加上所有节点主机名
vim /etc/hosts127.0.0.1 centos7-141 192.168.0.141 centos7-141 192.168.0.142 centos7-142 192.168.0.143 centos7-143 192.168.0.144 centos7-144 -
验证
free -mtotal used free shared buff/cache available Mem: 3789 193 2961 8 634 3350 Swap: 0 0 0
3、安装 Docker
官方文档
Install Docker Engine on CentOS
简单摘录一下步骤：
-
yum install -y yum-utils
-
yum-config-manager
–add-repo
https://download.docker.com/linux/centos/docker-ce.repo -
yum install docker-ce docker-ce-cli containerd.io
-
设置 Docker 镜像，并设置cgroupDriver
vim /etc/docker/daemon.json{ "exec-opts":["native.cgroupdriver=systemd"], "registry-mirrors": ["https://2vgbfb0x.mirror.aliyuncs.com"] } -
启动服务，并设置开机启动
systemctl enable docker && systemctl start docker -
验证 Docker
docker run hello-world
4、安装kubectl、kubelet和kubeadm
配置yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
5、master 节点执行初始化
-
配置初始化文件
mkdir working && cd workingkubeadm config print init-defaults > kubeadm-config.yaml
vim kubeadm-config.yaml
修改
1). advertiseAddress: 192.168.0.141
2). imageRepository: registry.aliyuncs.com/google_containers
3). name: 改成 /etc/hosts 中设置好的主机名称apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.0.141 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock imagePullPolicy: IfNotPresent name: centos7-141 taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: 1.22.0 networking: podSubnet: 10.244.0.0/16 dnsDomain: cluster.local -
预先拉取所需镜像
kubeadm config images pull --config=kubeadm-config.yaml[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.22.0 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.5 [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.0-0 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.4这一步非必需，预先拉取可以提前发现失败的 images，提前修改为镜像方式获取
只需要执行一次，可以 docker images 确认一下 -
初始化
加上 tee kubeadm-init.log，方便后续查看 token 和初始化信息
kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log[init] Using Kubernetes version: v1.22.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' ... Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.0.141:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:57df376d612009f381bd3f3835464578666536080c6f779cffcf8bc90af10930按照提示，root 身份简单设置
echo “export KUBECONFIG=/etc/kubernetes/admin.conf” >> /etc/profile
启动生效
#source /etc/profile -
大约1分钟后，确认所有服务健康状态：Healthy
kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Healthy ok etcd-0 Healthy {"health":"true","reason":""}我这里scheduler总是Unhealthy，手工修改以下 2 个文件
vim /etc/kubernetes/manifests/kube-scheduler.yaml
vim /etc/kubernetes/manifests/kube-controller-manager.yaml删除或者注释掉 - --port=0
重启kubelet服务生效
systemctl restart kubelet再等1分钟
kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} controller-manager Healthy ok如果发生错误，随时 kubeadm reset 再重来
-
确认 configmap 配置状态
kubectl get -n kube-system configmapNAME DATA AGE coredns 1 9m54s extension-apiserver-authentication 6 10m kube-flannel-cfg 2 43s kube-proxy 2 9m54s kube-root-ca.crt 1 9m43s kubeadm-config 1 9m56s kubelet-config-1.22 1 9m56s
6、master节点安装pod网络
-
获取 kube-flannel.yml
curl -o kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
把yml文件中的所有的quay.io改为quay.mirrors.ustc.edu.cn
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' kube-flannel.yml
或者
sed -i 's/quay.io/quay-mirror.qiniu.com/g' kube-flannel.yml
-
生成 flannel 插件pod
kubectl apply -f kube-flannel.ymlWarning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created-
确认配置正确
kubectl get -n kube-system configmapNAME DATA AGE coredns 1 9m54s extension-apiserver-authentication 6 10m kube-flannel-cfg 2 43s kube-proxy 2 9m54s kube-root-ca.crt 1 9m43s kubeadm-config 1 9m56s kubelet-config-1.22 1 9m56s -
确认所有的Pod都处于Running状态
kubectl get pod -n kube-systemNAME READY STATUS RESTARTS AGE coredns-7f6cbbb7b8-wb7xf 1/1 Running 0 12m coredns-7f6cbbb7b8-ww5z4 1/1 Running 0 12m etcd-centos7-141 1/1 Running 7 12m kube-apiserver-centos7-141 1/1 Running 1 12m kube-controller-manager-centos7-141 1/1 Running 1 (12m ago) 12m kube-flannel-ds-bvvq6 1/1 Running 0 3m31s kube-proxy-8f8bq 1/1 Running 0 12m kube-scheduler-centos7-141 1/1 Running 3 (12m ago) 12m
-
6、worker节点join
-
每一个节点服务器也和 master 主节点一样安装 Docker、kubectl、kubelet和kubeadm
如果master 重新init，则work节点join之前先执行 kubeadm reset
-
按照 master 初始化的输出提示加入集群
kubeadm join 192.168.0.141:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:57df376d612009f381bd3f3835464578666536080c6f779cffcf8bc90af10930返回结果大致如下
[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
如果没有记住刚才的 token ， master 主机 # cat kubeadm-init.log 可以找到
或者 kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
abcdef.0123456789abcdef 23h 2021-11-10T08:01:53Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
如果超过 24 小时没有 join ，token 过期，需要在 master 重新获取 token
kubeadm token create
8mfiss.yvbnl8m319ysiflh
-
验证node和 Pod状态，全部为Running
kubectl get nodesNAME STATUS ROLES AGE VERSION centos7-141 Ready control-plane,master 30m v1.22.2 centos7-143 Ready <none> 7m48s v1.22.2 centos7-144 Ready <none> 2m22s v1.22.2kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-7f6cbbb7b8-wb7xf 1/1 Running 0 28m kube-system coredns-7f6cbbb7b8-ww5z4 1/1 Running 0 28m kube-system etcd-centos7-141 1/1 Running 7 29m kube-system kube-apiserver-centos7-141 1/1 Running 1 29m kube-system kube-controller-manager-centos7-141 1/1 Running 1 (28m ago) 28m kube-system kube-flannel-ds-b5sg8 1/1 Running 0 47s kube-system kube-flannel-ds-bl9vr 1/1 Running 0 6m13s kube-system kube-flannel-ds-bvvq6 1/1 Running 0 19m kube-system kube-proxy-8f8bq 1/1 Running 0 28m kube-system kube-proxy-j679n 1/1 Running 0 47s kube-system kube-proxy-qczzf 1/1 Running 0 6m13s kube-system kube-scheduler-centos7-141 1/1 Running 3 (28m ago) 28m
7、部署dashboard
另外写一个博文
k8s 配置dashboard
8、错误处理
-
[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error: Get “http://localhost:10248/healthz”: dial tcp [::1]:10248: connect: connection refused.
给 Docker 设置 cgroupDriver
vim /etc/docker/daemon.json{ "exec-opts":["native.cgroupdriver=systemd"], "registry-mirrors": ["https://2vgbfb0x.mirror.aliyuncs.com"] }systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet -
This error is likely caused by:
- The kubelet is not running重启kubelet 后查看kubelet status
遇到奇怪的错误: failed to run Kubelet: unable to load bootstrap kubecon…r directory... 11月 10 16:39:04 centos7-189 kubelet[14277]: E1110 16:39:04.095861 14277 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubecon...r directory" 11月 10 16:39:04 centos7-189 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE 11月 10 16:39:04 centos7-189 systemd[1]: Unit kubelet.service entered failed state. 11月 10 16:39:04 centos7-189 systemd[1]: kubelet.service failed. Hint: Some lines were ellipsized, use -l to show in full.因为这个电脑来回折腾，之前作为 worker node ，后来又实用普通 user 安装配置过！也许有什么遗留没有清理干净？
cat: /var/lib/kubelet/kubeadm-flags.env: 没有那个文件或目录
这个文件是 kubeadm init 生成的
所以，重新 kubeadm reset 后 执行 # kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log重新 安装kubectl、kubelet和kubeadm
他自己就恢复了？怎么回事？因为我重装了？/var/lib/kubelet/kubeadm-flags.env 现在创建了
cat /var/lib/kubelet/kubeadm-flags.envKUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.5"很多地方说 Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf 这个文件里面加上 --cgroup-driver=systemd，如下：Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cgroup-driver=systemd"但是，这次没有加 --cgroup-driver=systemd ，也是 ok 的！
-
服务器关机后再开机，发现 coredns 状态 ContainerCreating ，kube-flannel-ds-k8cgb 状态 CrashLoopBackOff ，只好kubeadm reset 后重来，发生以下错误
…
[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with--ignore-preflight-errors=...需要配置 ipv4 转发
vim /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1设置生效
sysctl -p /etc/sysctl.d/k8s.conf -
worker 节点 join 发生 bridge-nf-call-iptables contents are not set to 1 错误
…
I1115 10:16:17.248205 14547 checks.go:432] validating if the connectivity type is via proxy or direct
[preflight] Some fatal errors occurred:
[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with--ignore-preflight-errors=...
error execution phase preflight
[root@centos7-185 ~]# kubeadm reset
之后再来设置
[root@centos7-185 ~]# echo “1” >/proc/sys/net/bridge/bridge-nf-call-iptables
确认
[root@centos7-185 ~]# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
重新 join ， ok！
9 、images 被墙时使用国内镜像地址替换
在应用yaml文件创建资源时，将文件中镜像地址进行内容替换即可：
1.k8s.gcr.io 地址替换
registry.cn-hangzhou.aliyuncs.com/google_containers
或者
registry.aliyuncs.com/google_containers
或者
mirrorgooglecontainers
-
quay.io 地址替换
quay-mirror.qiniu.com -
gcr.io 地址替换
registry.aliyuncs.com
开源、云原生的融合云平台
更多推荐
2
0- 0
已为社区贡献3条内容
所有评论(0)