配置环境说明：

  • 服务器配置说明
hostnameuseosip
test10DNS+docker_hubcentos710.10.10.10
test11master+nodecentos710.10.10.11
test12nodecentos710.10.10.12
  • 业务-分布说明
服务器名OSIP运行程序组件名组件说明
test10centos710.10.10.10dnsmasqDNS域名解析
test10centos710.10.10.10harbordocker镜像管理工具
test10centos710.10.10.10flannel管理docker及宿主机-网络工具
test11centos710.10.10.11etcd保存系统的配置信息和各种资源的状态信息
test11centos710.10.10.11flannel管理docker及宿主机-网络工具
test11centos710.10.10.11kube-apiserver提供Restful api。各种客户端工具或者其他组件可以调用其完成资源调用
test11centos710.10.10.11kube-scheduler调度服务，决定将容器创建在哪个Node上
test11centos710.10.10.11kube-controller-manager管理系统中各种资源，保证资源处于预期的状态
test11centos710.10.10.11kubectl集群中一个重要且便捷的管理工具
test11centos710.10.10.11kubelet接收Master节点发来的创建请求信息，并向Master报告运行状态。
test11centos710.10.10.11kube-proxy访问控制
test12centos710.10.10.12flannel管理docker及宿主机-网络工具
test12centos710.10.10.12kubectl集群中一个重要且便捷的管理工具
test12centos710.10.10.12kubelet接收Master节点发来的创建请求信息，并向Master报告运行状态。
test12centos710.10.10.12kube-proxy访问控制
  • 安装顺序
安装序号程序名称对应服务器
1dnsmasqtest10
2docker_hub-harbortest10
3etcdtest11
4flanneltest10,test11,test12
5kube-apiservertest11
6kube-schedulertest11
7kube-controller-managertest11
8kubectltest11,test12
9kubelettest11,test12
10kube-proxytest11,test12

kuberneter版本： 1.14.0
安装方式： 二进制文件
认证方式：key
单管理节点服务

预配置

配置服务器（所有服务器）
  • 关闭所有服务器selinux
setenforce 0
sed -i "s/^SELINUX\=.*/SELINUX=disabled/g" /etc/selinux/config
  • 关闭所有服务器防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
  • 关闭所有服务器swap
swapoff -a
##注释 /etc/fstab 中swap启动项
  • 下载日常插件并设置yum源文件为阿里云
yum -y install epel-release wget yum-utils device-mapper-persistent-data lvm2
mkdir -p /etc/yum.repos.d/bak
mv /etc/yum.repos.d/* /etc/yum.repos.d/bak
wget http://mirrors.aliyun.com/repo/Centos-7.repo -P /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/epel-7.repo -P /etc/yum.repos.d/
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install wget vim ntp unzip zip net-snmp* telnet lrzsz bash-completion net-tools ntpdate supervisor
  • 安装docker-ce
##安装
yum -y install install docker-ce docker-ce-cli containerd.io

##配置阿里云docker源
mkdir -p /etc/docker
cat >> /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://hh3tvdpc.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
  • 同步时间
ntpdate ntp1.aliyun.com
echo '*/5 * * * * /usr/sbin/ntpdate ntp1.aliyun.com > /dev/null 2>&1' >> /var/spool/cron/root
  • 修改主机名及hosts
hostname test10
hostnamectl set-hostname test10
  • 配置内核参数
cat >> /etc/sysctl.conf <<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/rc.local
sysctl -p /etc/sysctl.conf
  • 服务器SSH免密
##master机器
ssh-keygen

##授信其他服务器
ssh-copy-id 10.10.10.10
ssh-copy-id 10.10.10.11
ssh-copy-id 10.10.10.12
配置证书
  • 下载并安装证书生成工具
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
mkdir -p /data/k8s_key
cd /data/k8s_key

需创建证书如下：
ca证书
etcd证书
apiserver证书
proxy证书
kubectl证书

  • 生成ca证书
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JS",
      "L": "NJ",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

根据需求修改 'C' , 'ST' , 'L' , 'O' , 'OU' 值
"CN"：Common Name，kube-apiserver从证书中提取该字段作为请求的用户名(User name)；浏览器检验该字段验证网站是否合法；
“O”：Organization，kube-apiserver从证书提取该字段作为请求用户所属的组（Group）；

  • 生成etcd证书
cat > etcd-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.10.10.11",
    "etcd.k8s.test"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JS",
      "L": "NJ",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

'hosts'为必填项，根据ETCD程序实际部署IP地址及域名进行填写，否则证书校验会出错

  • 生成apiserver证书
cat > apiserver-csr.json << EOF
{
  "CN": "apiserver",
  "hosts": [
    "127.0.0.1",
    "10.10.10.11",
    "apiserver.k8s.test"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JS",
      "L": "NJ",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver

'hosts'为必填项，根据apiserver程序实际部署IP地址及域名进行填写，否则证书校验会出错

  • 生成proxy证书
cat > proxy-csr.json << EOF
{
  "CN": "proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JS",
      "L": "NJ",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes proxy-csr.json | cfssljson -bare proxy

'hosts'可以为空，集群中增加新节点也不需要重新生成证书

  • 生成kubectl证书
cat > kubectl-csr.json << EOF
{
  "CN": "kubectl",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JS",
      "L": "NJ",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubectl-csr.json | cfssljson -bare kubectl
  • 将证书同步到其他服务器
ssh 10.10.10.10 "mkdir -p /data/k8s_key"
scp -r /data/k8s_key 10.10.10.10:/data/

ssh 10.10.10.12 "mkdir -p /data/k8s_key"
scp -r /data/k8s_key 10.10.10.12:/data/

部署dnsmasq

test10

  • 安装
yum -y install dnsmasq
  • 修改配置文件
cp /etc/dnsmasq.conf /etc/dnsmasq.conf.bak


cat > /etc/dnsmasq.conf <<EOF
port=53
user=nobody
group=nobody
listen-address=10.10.10.10,127.0.0.1
no-hosts
addn-hosts=/data/dnsmasq/dnsmasq.hosts
hostsdir=/data/dnsmasq/dnsmasq.d
log-queries
log-facility=/data/dnsmasq/log/dnsmasq.log
log-async=50
edns-packet-max=4096
interface=ens33
no-dhcp-interface=ens33
resolv-file=/data/dnsmasq/resolv.dnsmasq
strict-order
clear-on-reload
domain-needed
local-ttl=0
cache-size=1000
no-negcache
dns-forward-max=1000
EOF
  • 创建相关配置文件及文件夹
mkdir -p /data/dnsmasq/{dnsmasq.d,log}
touch /data/dnsmasq/{dnsmasq.hosts,resolv.dnsmasq}
  • 填写DNS转发服务器（提供非自定义域名查询）
cat > /data/dnsmasq/resolv.dnsmasq << EOF
nameserver 223.5.5.5
nameserver 1.2.4.8
EOF
  • 填写hosts主机记录（提供域名hosts记录集中查询）
cat > /data/dnsmasq/dnsmasq.hosts << EOF
10.10.10.10 test10
10.10.10.11 test11
10.10.10.12 test12
EOF

修改addn-hosts指定hosts记录文件，需重启dnsmasq，可以通过hostsdir指定域名配置文件添加解析。

  • 填写自定义域名（提供内网自定义域名查询）
cat > /data/dnsmasq/dnsmasq.d/k8s.tes <<EOF
10.10.10.10 harbor.k8s.test
10.10.10.11 etcd.k8s.test
10.10.10.11 apiserver.k8s.test
EOF
  • 启动服务并设置开机启动
systemctl start dnsmasq.service
systemctl enable dnsmasq.service
  • 所有服务器设置DNS指向10.10.10.10
#修改配置项 /etc/sysconfig/network-scripts/ifcfg-eth0
PEERDNS=no  #拒绝接受DHCP分发的DNS配置
DNS1=10.10.10.10  #自定义配置DNS服务器地址
 
#重启网络配置
systemctl restart network.service

部署etcd

test11

  • 下载、安装
mkdir -p /setup/ /opt/etcd/{bin,conf} /data/etcd/
wget https://github.com/coreos/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz -P /setup/
cd /setup/
tar zxvf etcd-v3.3.13-linux-amd64.tar.gz
mv etcd-v3.3.13-linux-amd64/etcd* /opt/etcd/bin/
chmod +x /opt/etcd/bin/etcd*
ln -s /opt/etcd/bin/etcd /usr/bin/
ln -s /opt/etcd/bin/etcdctl /usr/bin/
  • 创建配置文件 /opt/etcd/conf/etcd.conf
ETCD_CONF='--name test11 \
--data-dir /data/etcd \
--listen-peer-urls https://0.0.0.0:2380 \
--listen-client-urls https://0.0.0.0:2379 \
--advertise-client-urls https://10.10.10.11:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster-state new \
--initial-advertise-peer-urls https://10.10.10.11:2380 \
--initial-cluster test11=https://10.10.10.11:2380 \
--client-cert-auth \
--trusted-ca-file /data/k8s_key/ca.pem \
--cert-file /data/k8s_key/etcd.pem \
--key-file /data/k8s_key/etcd-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file /data/k8s_key/ca.pem \
--peer-cert-file /data/k8s_key/etcd.pem \
--peer-key-file /data/k8s_key/etcd-key.pem'
  • 创建启动文件 /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/conf/etcd.conf
ExecStart=/opt/etcd/bin/etcd $ETCD_CONF
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 启动etcd并设置为开机启动
systemctl daemon-reload
systemctl start etcd.service
systemctl enable etcd.service
  • 查看部署情况
#查看成员状态
etcdctl --ca-file=/data/k8s_key/ca.pem --cert-file=/data/k8s_key/etcd.pem --key-file=/data/k8s_key/etcd-key.pem --endpoints=https://127.0.0.1:2379 member list
#查看集群状态
etcdctl --ca-file=/data/k8s_key/ca.pem --cert-file=/data/k8s_key/etcd.pem --key-file=/data/k8s_key/etcd-key.pem --endpoints=https://127.0.0.1:2379 cluster-health

部署harbor （docker镜像仓库，提供项目docker镜像托管及版本管理）

test10

  • 下载
yum -y install docker-compose.noarch

mkdir -p /setup /opt/harbor/
wget https://storage.googleapis.com/harbor-releases/release-1.8.0/harbor-offline-installer-v1.8.1.tgz -P /setup
tar xvf harbor-offline-installer-v1.8.1.tgz -C /opt/
  • 修改配置文件
##修改配置文件 /opt/harbor/harbor.yml
hostname: harbor.k8s.test  #侦听域名
harbor_admin_password: admin123  #管理平台admin管理员密码
  • 安装Harbor
cd /opt/harbor/
./install.sh

如有保存，按照错误提醒解决问题。（一般为缺少依赖，或者依赖版本不匹配）

  • 访问harbor管理界面
#访问url  http://harbor.k8s.test
#用户名/密码  admin / admin123
  • 修改docker配置，访问harbor http模式（配置在上传docker镜像机器中）
##修改配置文件 /etc/docker/daemon.json
{
  "registry-mirrors": ["https://hh3tvdpc.mirror.aliyuncs.com"],
  "insecure-registries":["harbor.k8s.test"]
}

##重启docker
systemctl restart docker.service

##docker登录仓库账户
docker login harbor.k8s.test
admin
admin123
  • 上传镜像步骤
##管理界面中创建项目 dockertest
##将已准备的镜像包使用tag转成私网镜像包
docker tag SOURCE_IMAGE[:TAG] harbor.k8s.test/dockertest/IMAGE[:TAG]
docker tag SOURCE_IMAGE[:TAG] harbor.k8s.test/dockertest/IMAGE:latest
##上传镜像包(所有)
docker push harbor.k8s.test/dockertest/IMAGE
##管理界面中查看dockertest 项目中是否有已上传镜像

部署flannel（所有服务器都部署）,并修改docker-ce配置项（使用flannel网络）

test10、test11、test12
test10的docker不用修改启动项。

  • 下载、安装
mkdir -p /setup/ /opt/flannel/{bin,conf} /data/flannel
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz -P /setup/
cd /setup/
tar zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel/bin/
chmod +x /opt/flannel/bin/*
ln -s /opt/flannel/bin/flanneld /usr/bin/

  • 向etcd中注册网段信息

    master机器执行一遍即可

/opt/etcd/bin/etcdctl \
--ca-file=/data/k8s_key/ca.pem \
--cert-file=/data/k8s_key/etcd.pem \
--key-file=/data/k8s_key/etcd-key.pem \
--endpoints=https://127.0.0.1:2379 \
set /test_1/network/config '{"Network": "172.30.0.0/16","SubnetLen": 24, "SubnetMin": "172.30.1.0","SubnetMax": "172.30.20.0", "Backend": {"Type": "vxlan"}}'

注册使用网段172.30.0.0/16，每个子网分配子网掩码24位，网络分配从172.30.1.0/24分配到172.30.20.0/24，协议使用vxlan

  • 创建配置文件 /opt/flannel/conf/flannel.conf
FLANNEL_CONF="-etcd-cafile=/data/k8s_key/ca.pem \
-etcd-certfile=/data/k8s_key/etcd.pem \
-etcd-keyfile=/data/k8s_key/etcd-key.pem \
-etcd-endpoints=https://etcd.k8s.test:2379 \
-etcd-prefix=/test_1/network"
  • 创建启动文件 /usr/lib/systemd/system/flannel.service
[Unit]
Description=Flanneld overlay address etcd agent
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=-/opt/flannel/conf/flannel.conf
ExecStart=/opt/flannel/bin/flanneld $FLANNEL_CONF
ExecStartPost=/opt/flannel/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
  • 启动flannel并设置为开机启动
systemctl daemon-reload
systemctl start flannel.service
systemctl enable flannel.service
  • 检测启动状态
#查看etcd中注册信息
etcdctl -ca-file=/data/k8s_key/ca.pem --cert-file=/data/k8s_key/etcd.pem --key-file=/data/k8s_key/etcd-key.pem --endpoints=https://etcd.k8s.test:2379 ls -r /test_1/network/subnets/
etcdctl -ca-file=/data/k8s_key/ca.pem --cert-file=/data/k8s_key/etcd.pem --key-file=/data/k8s_key/etcd-key.pem --endpoints=https://etcd.k8s.test:2379 get /test_1/network/subnets/172.30.5.0-24

#查看本地网络情况
ip add

#查看docker网络配置文件
cat /run/flannel/docker

  • 修改docker启动文件（使用flannel网络）

    /usr/lib/systemd/system/docker.service

[Service]
#修改
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
#新增
EnvironmentFile=/run/flannel/docker
  • 启动docker并设置为开机启动
systemctl daemon-reload
systemctl start docker.service
systemctl enable docker.service
  • 检查docker网络
##检查本地网络情况
ip a show docker0

##查看docker的配置
docker inspect bridge

部署kube-apiserver

test11

  • 下载、安装
mkdir /setup && cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-server-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/server/bin/kube-apiserver /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/kube-apiserver
  • 创建token.csv文件
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > /opt/kubernetes/conf/token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
  • 创建高级审计配置文件 /opt/kubernetes/conf/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"
  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]
  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]
  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.
  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
  • 创建配置文件 /opt/kubernetes/conf/kube-apiserver.conf
KUBE_APISERVER="--apiserver-count=1 \
--logtostderr=false \
--audit-log-path=/data/kubernetes/logs/kube-apiserver.log \
--audit-policy-file=/opt/kubernetes/conf/audit-policy.yaml \
--v=4 \
--bind-address=10.10.10.11 \
--secure-port=6443 \
--advertise-address=10.10.10.11 \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/conf/token.csv \
--client-ca-file=/data/k8s_key/ca.pem \
--requestheader-client-ca-file=/data/k8s_key/ca.pem \
--etcd-cafile=/data/k8s_key/ca.pem \
--etcd-certfile=/data/k8s_key/apiserver.pem \
--etcd-keyfile=/data/k8s_key/apiserver-key.pem \
--etcd-servers=https://etcd.k8s.test:2379 \
--service-account-key-file=/data/k8s_key/ca-key.pem \
--service-cluster-ip-range=10.254.0.0/16 \
--service-node-port-range=30000-50000 \
--kubelet-client-certificate=/data/k8s_key/apiserver.pem \
--kubelet-client-key=/data/k8s_key/apiserver-key.pem \
--tls-cert-file=/data/k8s_key/apiserver.pem \
--tls-private-key-file=/data/k8s_key/apiserver-key.pem"
  • 创建启动文件 /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
Type=notify
EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 启动kube-apiserver并设置为开机启动
systemctl daemon-reload
systemctl start kube-apiserver.service
systemctl enable kube-apiserver.service

部署kube-scheduler

test11

  • 下载、安装
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-server-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/server/bin/kube-scheduler /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/kube-scheduler
  • 创建配置文件 /opt/kubernetes/conf/kube-scheduler.conf
KUBE_SCHEDULER="--leader-elect \
--logtostderr=false \
--log-dir=/data/kubernetes/logs/ \
--log-file=/data/kubernetes/logs/kube-scheduler.log \
--v=4 \
--master=http://127.0.0.1:8080"
  • 创建启动文件 /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=kube-apiserver.service

[Service]
Type=simple
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 启动kube-scheduler并设置为开机启动
systemctl daemon-reload
systemctl start kube-scheduler.service
systemctl enable kube-scheduler.service

部署kube-controller-manager

test11

  • 下载、安装
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-server-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/server/bin/kube-controller-manager /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/kube-controller-manager
  • 创建配置文件 /opt/kubernetes/conf/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER="--logtostderr=false \
--log-dir=/data/kubernetes/logs/ \
--log-file=/data/kubernetes/logs/kube-controller-manager.log \
--v=4 \
--master=http://127.0.0.1:8080 \
--leader-elect=true \
--address=0.0.0.0 \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/data/k8s_key/ca.pem \
--cluster-signing-key-file=/data/k8s_key/ca-key.pem  \
--root-ca-file=/data/k8s_key/ca.pem \
--service-account-private-key-file=/data/k8s_key/ca-key.pem"
  • 创建启动文件 /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=kube-apiserver.service

[Service]
Type=simple
EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 启动kube-controller-manager并设置为开机启动
systemctl daemon-reload
systemctl start kube-controller-manager.service
systemctl enable kube-controller-manager.service

master配置kubectl

test11

  • 下载、安装
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-server-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/server/bin/kubectl /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/kubectl
ln -s /opt/kubernetes/bin/kubectl /usr/sbin/
  • 查看master集群状态
kubectl get cs,nodes

NAME                                 STATUS    MESSAGE             ERROR
componentstatus/scheduler            Healthy   ok                  
componentstatus/controller-manager   Healthy   ok                  
componentstatus/etcd-0               Healthy   {"health":"true"}

node服务器使用kubectl

test12
kubectl默认通过localhost:8080连接apiserver 。所以需要配置kubectl使用证书访问api接口。

  • 下载、安装
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-node-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/node/bin/kubectl /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/kubectl
ln -s /opt/kubernetes/bin/kubectl /usr/sbin/
  • 在master节点上添加kubectl用户
##创建资源配置文件（添加授权kubectl用户） kubectl.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectl
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kubectl
  
##应用资源配置文件
kubectl create -f kubectl.yaml
  • node节点增加配置文件
> 创建/root/.kube/config文件

# 设置集群参数,--server指定Master节点ip
kubectl config set-cluster kubernetes \
--certificate-authority=/data/k8s_key/ca.pem \
--server=https://apiserver.k8s.test:6443


# 设置客户端认证参数
kubectl config set-credentials kubectl \
--certificate-authority=/data/k8s_key/ca.pem \
--client-certificate=/data/k8s_key/kubectl.pem \
--client-key=/data/k8s_key/kubectl-key.pem


# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubectl

# 设置默认上下文
kubectl config use-context default
  • node节点上验证
##查看配置文件
cat /root/.kube/config

##使用命令查看
kubectl get node

部署kubelet

test11、test12

  • 下载、部署
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-node-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/node/bin/{kubelet,kubectl} /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/*
ln -s /opt/kubernetes/bin/kubectl /usr/sbin/
  • 创建bootstrap.kubeconfig.（master机器上执行）
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/data/k8s_key/ca.pem \
--embed-certs=true \
--server="https://apiserver.k8s.test:6443" \
--kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数 (token值必须对应apiserver中token.csv中token值)
kubectl config set-credentials kubelet-bootstrap \
--token=2aa4a30846d730e004da12aaa5b43142 \
--kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

# 设置默认上下文并生成文件
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
  • 将kubelet-bootstrap用户绑定到系统集群角色 （master机器上执行）
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
  • 将bootstrap.kubeconfig复制到配置文件目录并scp到所有node服务器
cp bootstrap.kubeconfig /opt/kubernetes/conf
scp /opt/kubernetes/conf/bootstrap.kubeconfig test12:/opt/kubernetes/conf/
  • 创建kubelet参数配置模板文件 /opt/kubernetes/conf/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 10.10.10.12
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.10.10.10"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true

address 本机IP

  • 创建kubelet配置文件 /opt/kubernetes/conf/kubelet.conf
KUBELET="--logtostderr=false \
--log-dir=/data/kubernetes/logs/ \
--log-file=/data/kubernetes/logs/kubelet.log \
--v=4 \
--hostname-override=10.10.10.12 \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/conf/bootstrap.kubeconfig \
--config=/opt/kubernetes/conf/kubelet.config \
--cert-dir=/data/k8s_key/ \
--pod-infra-container-image=harbor.k8s.test/test/nginx:laster"

hostname-override 取本机IP

  • 创建启动文件 /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
After=docker.service

[Service]
Type=simple
EnvironmentFile=-/opt/kubernetes/conf/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 启动kubelet并设置为开机启动
systemctl daemon-reload
systemctl start kubelet.service
systemctl enable kubelet.service
  • master服务器上确认csr请求
#查看csr请求信息
kubectl get csr

#确认csr请求
kubectl certificate approve node-csr-OJXR5HcB9oYb8cZCdWeg6iQZgcLiSaORWwsImOBZVS8

#查看集群状态(确认csr后注册的node节点)
kubectl get nodes
  • 删除过期的csr
kubectl delete csr node-csr-XiPfkW9BD3t3XmpcxbQ3m7CA5NPjNp_2OQSY2Tl94gA
  • 删除过期的node
kubectl delete node 10.10.10.12
  • 集群打标签
kubectl label node 10.10.10.11  node-role.kubernetes.io/master='master'
kubectl label node 10.10.10.12  node-role.kubernetes.io/node='node'
  • 查看cs,service,node,csr信息
kubectl get cs,service,node,csr
  • 查看详细信息（服务端）
kubectl describe service

部署kube-proxy

test11、test12

  • 下载、安装
cd /setup
wget https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz -P /setup
tar zxvf kubernetes-node-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,conf} /data/kubernetes/logs/
cp kubernetes/node/bin/{kube-proxy,kubectl} /opt/kubernetes/bin/
chmod +x /opt/kubernetes/bin/*
ln -s /opt/kubernetes/bin/kubectl /usr/sbin/
  • 创建kube-proxy.kubeconfig （master机器上执行）
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/data/k8s_key/ca.pem \
--embed-certs=true \
--server="https://apiserver.k8s.test:6443" \
--kubeconfig=kube-proxy.kubeconfig

# 设置客户端认证参数 
kubectl config set-credentials kube-proxy \
--client-certificate=/data/k8s_key/proxy.pem \
--client-key=/data/k8s_key/proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
  • 将bootstrap.kubeconfig复制到配置文件目录并scp到所有node服务器
cp kube-proxy.kubeconfig /opt/kubernetes/conf
scp /opt/kubernetes/conf/kube-proxy.kubeconfig test12://opt/kubernetes/conf
  • 创建配置文件 /opt/kubernetes/conf/kubelet.conf
KUBELET="--logtostderr=false \
--log-dir=/data/kubernetes/logs/ \
--log-file=/data/kubernetes/logs/kubelet.log \
--v=4 \
--hostname-override=10.10.10.12 \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/conf/bootstrap.kubeconfig \
--config=/opt/kubernetes/conf/kubelet.config \
--cert-dir=/data/k8s_key/ \
--pod-infra-container-image=harbor.k8s.test/test/nginx:laster"

--hostname-override: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；

  • 创建启动文件 /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
 
[Service]
Type=simple
EnvironmentFile=-/opt/kubernetes/conf/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY
Restart=on-failure
RestartSec=2
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target
  • 启动kube-proxy程序
systemctl daemon-reload
systemctl start kube-proxy.service
systemctl enable kube-proxy.service

转载于:https://www.cnblogs.com/taoyuxuan/p/11205430.html

# 运维 # json # 开发工具
Logo
Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。 一、云计算历经十余年

avatar Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松 今天的内容会从以下几个方面展开： 负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点 一、产品简介​ 1. 以NGINX为例​ 提到负载均衡，我们以

avatar Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。 单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这

avatar Cloudpods