目录​​​​​​

一、环境准备（所有节点）

安装docker

 启动docker

设置docker阿里云镜像源

关闭防火墙

禁用selinux

设置内核参数

关闭swap

二、安装kubeadm、kubelet、kubectl

修改yum安装源（所有节点）

初始化集群（master节点）

三、安装flannel网络

 四、修改nodeport类型的端口范围

五、子节点加入集群

---

一、环境准备（所有节点）

系统	ip	主机名	配置
centos 7.8	192.168.1.10	master01	2核4G
centos 7.8	192.168.1.11	node01	2核4G
centos 7.8	192.168.1.12	node02	2核4G

安装docker

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

 启动docker

 systemctl start docker 
 systemctl enable docker

设置docker阿里云镜像源

 sudo tee /etc/docker/daemon.json <<-'EOF'
     {
       "registry-mirrors": ["https://f9dk003m.mirror.aliyuncs.com"]
     }
 

systemctl daemon-reload
systemctl restart docker

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

禁用selinux

# 临时禁用
setenforce 0
# 永久禁用 
vim /etc/selinux/config    # 或者修改/etc/sysconfig/selinux
SELINUX=disabled

设置内核参数

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

关闭swap

# 临时关闭
swapoff -a

sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

二、安装kubeadm、kubelet、kubectl

修改yum安装源（所有节点）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

查看可安装版本

yum list kubeadm --showduplicates | sort -r

安装

yum install -y kubelet-1.22.1-0 kubeadm-1.22.1-0 kubectl-1.22.1-0
systemctl enable kubelet && systemctl start kubelet

初始化集群（master节点）

kubeadm init \
--apiserver-advertise-address=192.168.1.10 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.22.1 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.50.0.0/16

参数说明：

–apiserver-advertise-address：用于指定kube-apiserver监听的ip地址,就是 master本机IP地址
–image-repository: 指定阿里云镜像仓库地址
–kubernetes-version: 用于指定k8s版本
–service-cidr：用于指定SVC的网络范围
–pod-network-cidr：用于指定Pod的网络范围； 10.244.0.0/16

可能出现的问题

出现ERROR FileContent--proc-sys-net-ipv4-ip_forward 这个错误是因为环境准备没有做好

解决办法：

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720         
EOF

ysctl --system

出现此错误信息，是因为阿里云没有此镜像，需要手动下载镜像并打tag。

[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/coredns:v1.8.4: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/coredns:v1.8.4 not found: manifest unknown: manifest unknown

解决办法：

# 查看初始化集群时需要拉的镜像名
kubeadm config images list

docker pull coredns/coredns

docker tag coredns/coredns:latest  registry.aliyuncs.com/google_containers/coredns:v1.8.4

出现下面错误

The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
error execution phase kubelet-start: error uploading crisocket: timed out waiting for the condition

 使用systemctl status kubelet 查看

"Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""

kubelet.service: main process exited, code=exited, status=1/FAILURE

解决办法：

修改默认docker和kubelet的控制平台为systemd

在/etc/docker/daemon.json中，添加"exec-opts": ["native.cgroupdriver=systemd"]

cat > /var/lib/kubelet/config.yaml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

#重启

systemctl daemon-reload

systemctl restart docker

systemctl restart kubelet

#查看是否修改成功

docker info|grep "Cgroup Driver"

 使用kubeadm reset后重新执行 kubeadm init 提示initialized successfully！

 根据提示创建建kubectl用户

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

三、安装flannel网络

flannel地址：https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 

需要获取到flannel地址返回的yaml写入到/opt/yaml/kube-flannel.yaml文件里

sed -i 's#quay.io/coreos/flannel#quay.mirrors.ustc.edu.cn/coreos/flannel#' /opt/yaml/kube-flannel.yaml

sed -i 's#quay.mirrors.ustc.edu.cn/coreos/flannel#quay.io/coreos/flannel#' /opt/yaml/kube-flannel.yaml

这里因为每次获取的都不一样，建议直接使用本文最下方的配置

kubectl apply -f /opt/yaml/kube-flannel.yaml

使用kubectl get nodes 查看当前节点

 四、修改nodeport类型的端口范围

编辑kube-apiserver.yaml

vim /etc/kubernetes/manifests/kube-apiserver.yaml

 找到 --service-cluster-ip-range 这一行，在这一行的下一行增加 如下内容

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --service-node-port-range=1-65535

因为我的hostname是master01所以执行：

kubectl delete pod kube-apiserver-master01 -n kube-system

五、子节点加入集群

kubeadm token create --print-join-command
kubeadm join 192.168.1.10:6443 --token 5wmpns.84ltuxc6fgydsum9 \
--discovery-token-ca-cert-hash sha256:b0bf6365b53672f4f1cb40c4558105e43023348f1fc98a3a61ef2a683d294b2c

# token过期后，新节点的加入方法
kubeadm token list
# 创建token
kubeadm token create
# 创建加密
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

kubeadm join --token aa78f6.8b4cafc8ed26c34f --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538 192.168.1.10:6443
# join时可以加 --node-name k8s-new-node，节点名

子节点使用kubectl命令

将主节点中的 /etc/kubernetes/admin.conf 文件拷贝到 node节点相同目录下，然后对node节点配置环境变量

master节点上进行远程复制到node节点

#scp /etc/kubernetes/admin.conf root@192.168.1.15:/etc/kubernetes/

node节点上配置环境变量：

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile

---

kube-flannel.yml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

参考文章：

【kubeadm初始化报错】failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd" - 东北小狐狸 - 博客园

kubernetes修改 nodeport 类型的端口范围_不屑哥的专栏-CSDN博客

kubeadm安装k8s 1.22.1版本 - wsongl - 博客园