《kubernetes-1.8.0》04-master搭建

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-21

一、api-HA介绍

部署三台master：
目前所谓的 Kubernetes HA 其实主要的就是 API Server 的 HA，master 上其他组件比如 controller-manager 等都是可以通过 Etcd 做选举；而 API Server 只是提供一个请求接收服务，所以对于 API Server 一般有两种方式做 HA；一种是对多个 API Server 做 vip（HAproxy或者keepalive），另一种使用 nginx 反向代理，本文采用 nginx 方式。

master 之间除 api server 以外其他组件通过 etcd 选举，api server 默认不作处理；在每个 node 上启动一个 nginx，每个 nginx 反向代理所有 api server，node 上 kubelet、kube-proxy 连接本地的 nginx 代理端口，当 nginx 发现无法连接后端时会自动踢掉出问题的 api server，从而实现 api server 的 HA。

源自：mritd.me/部署-ha-master

一、config 通用配置（/etc/kubernetes/config）

三台master上证书与 rpm 都安装完成后，只需要修改配置(配置位于 /etc/kubernetes 目录)后启动相关组件即可

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=2"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"

• KUBE_MASTER:用于controller-manager, scheduler, and proxy find the apiserver；

二、apiserver配置（/etc/kubernetes/apiserver）

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=172.18.169.131 --insecure-bind-address=127.0.0.1 --bind-address=172.18.169.131"

# The port on the local server to listen on.
KUBE_API_PORT="--insecure-port=8080 --secure-port=6443"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction"

# Add your own!
KUBE_API_ARGS="--authorization-mode=RBAC,Node \
               --runtime-config=batch/v2alpha1=true \
               --anonymous-auth=false \
               --kubelet-https=true \
               --enable-bootstrap-token-auth \
               --token-auth-file=/etc/kubernetes/token.csv \
               --service-node-port-range=30000-50000 \
               --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
               --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
               --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
               --service-account-key-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
               --etcd-quorum-read=true \
               --storage-backend=etcd3 \
               --etcd-cafile=/etc/etcd/ssl/etcd-root-ca.pem \
               --etcd-certfile=/etc/etcd/ssl/etcd.pem \
               --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
               --enable-swagger-ui=true \
               --apiserver-count=3 \
               --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
               --audit-log-maxage=30 \
               --audit-log-maxbackup=3 \
               --audit-log-maxsize=100 \
               --audit-log-path=/var/log/kube-audit/audit.log \
               --event-ttl=1h"

• KUBE_API_ADDRESS:制定apiserver监听的IP，http监听127.0.0.1(不对外)，https监听本机网卡地址。
• --authorization-mode=RBAC,Node:授权模型增加了 Node 参数，因为 1.8 后默认 system:node role 不会自动授予 system:nodes 组，具体请参看 CHANGELOG(before-upgrading 段最后一条说明)
• 由于以上原因，–admission-control 同时增加了 NodeRestriction 参数，关于关于节点授权器请参考 Using Node Authorization
• --enable-bootstrap-token-auth:用于开启apiserver token认证，支持kubelet通过token的方式进行注册。
• --token-auth-file=/etc/kubernetes/token.csv：对应记录token的文件位置，后续需创建。
• 增加 --audit-policy-file参数用于指定高级审计配置，具体可参考 CHANGELOG(before-upgrading 第四条)、Advanced audit,后续创建对应的audit yaml文件。
• • 增加--runtime-config=batch/v2alpha1=true 参数用于cron job定时任务的支持。

创建对应的token文件、kubelet TLS相关配置文件、kube-proxy TLS相关配置文件以及audit-prolicy.yaml文件

##设置环境变量，生成token随机数
export KUBE_APISERVER="https://127.0.0.1:6443"
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
echo "Tokne: ${BOOTSTRAP_TOKEN}"

##创建对应的token文件
$ cat > /etc/kubernetes/token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

##创建kubelet以及kube-proxy的配置文件
##kubelet配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=k8s-root-ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

##kube-proxy配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=k8s-root-ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

##生成高级审计配置
cat >> audit-policy.yaml <<EOF
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
EOF

分发token文件、kubelet TLS相关配置文件、kube-proxy TLS相关配置文件以及audit-prolicy.yaml文件至三台master对应目录

for IP in `seq 131 133`;do
    scp *.kubeconfig /etc/kubernetes/token.csv audit-policy.yaml root@172.18.169.$IP:/etc/kubernetes
    ssh root@172.18.169.$IP chown -R kube:kube /etc/kubernetes/ssl
done

设置 log 目录权限

for IP in `seq 131 133`;do
    ssh root@172.18.169.$IP mkdir -p /var/log/kube-audit /usr/libexec/kubernetes
    ssh root@172.18.169.$IP chown -R kube:kube /var/log/kube-audit /usr/libexec/kubernetes
    ssh root@172.18.169.$IP chmod -R 755 /var/log/kube-audit /usr/libexec/kubernetes
done

三、controller-manager 配置（/etc/kubernetes/controller-manager）

###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=0.0.0.0 \
                              --service-cluster-ip-range=10.254.0.0/16 \
                              --cluster-name=kubernetes \
                              --cluster-signing-cert-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
                              --cluster-signing-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem \
                              --service-account-private-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem \
                              --root-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
                              --leader-elect=true \
                              --node-monitor-grace-period=40s \
                              --node-monitor-period=5s \
                              --pod-eviction-timeout=5m0s"

四、scheduler 配置（/etc/kubernetes/scheduler）

###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS="--leader-elect=true --address=0.0.0.0"

五、启动服务并查看群集组件状态

$ sudo systemctl daemon-reload
$ sudo systemctl start kube-apiserver
$ sudo systemctl start kube-controller-manager
$ sudo systemctl start kube-scheduler
$ sudo systemctl enable kube-apiserver
$ sudo systemctl enable kube-controller-manager
$ sudo systemctl enable kube-scheduler

$ sudo kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"} 

至此master节点基本部署完成

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考链接：

https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

https://github.com/opsnull/follow-me-install-kubernetes-cluster

https://kubernetes.io/docs/reference/generated/kube-apiserver/