一、存储卷概述及类型

1、概述
由于容器本身是非持久化的，因此需要解决在容器中运行应用程序遇到的一些问题。首先，当容器崩溃时，kubelet将重新启动容器，但是写入容器的文件将会丢失，容器将会以镜像的初始状态重新开始；第二，在通过一个Pod中一起运行的容器，通常需要共享容器之间一些文件。Kubernetes通过存储卷解决上述的两个问题。

在Docker有存储卷的概念卷，但Docker中存储卷只是磁盘的或另一个容器中的目录，并没有对其生命周期进行管理。Kubernetes的存储卷有自己的生命周期，它的生命周期与使用的它Pod生命周期一致。因此，相比于在Pod中运行的容器来说，存储卷的存在时间会比的其中的任何容器都长，并且在容器重新启动时会保留数据。当然，当Pod停止存在时，存储卷也将不再存在。在Kubernetes支持多种类型的卷，而Pod可以同时使用各种类型和任意数量的存储卷。在Pod中通过指定下面的字段来使用存储卷：

  spec.volumes：通过此字段提供指定的存储卷
  spec.containers.volumeMounts：通过此字段将存储卷挂接到容器中

2、存储卷类型和示例
当前Kubernetes支持如下所列这些存储卷类型。

[root@master ~]# kubectl explain pod.spec.volumes
KIND:     Pod
VERSION:  v1

RESOURCE: volumes <[]Object>

DESCRIPTION:
     List of volumes that can be mounted by containers belonging to the pod.
     More info: https://kubernetes.io/docs/concepts/storage/volumes

     Volume represents a named volume in a pod that may be accessed by any
     container in the pod.

FIELDS:
   awsElasticBlockStore	<Object>
     AWSElasticBlockStore represents an AWS Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore

   azureDisk	<Object>
     AzureDisk represents an Azure Data Disk mount on the host and bind mount to
     the pod.

   azureFile	<Object>
     AzureFile represents an Azure File Service mount on the host and bind mount
     to the pod.

   cephfs	<Object>
     CephFS represents a Ceph FS mount on the host that shares a pod's lifetime

   cinder	<Object>
     Cinder represents a cinder volume attached and mounted on kubelets host
     machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md

   configMap	<Object>
     ConfigMap represents a configMap that should populate this volume

   csi	<Object>
     CSI (Container Storage Interface) represents storage that is handled by an
     external CSI driver (Alpha feature).

   downwardAPI	<Object>
     DownwardAPI represents downward API about the pod that should populate this
     volume

   emptyDir	<Object>
     EmptyDir represents a temporary directory that shares a pod's lifetime.
     More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir

   fc	<Object>
     FC represents a Fibre Channel resource that is attached to a kubelet's host
     machine and then exposed to the pod.

   flexVolume	<Object>
     FlexVolume represents a generic volume resource that is
     provisioned/attached using an exec based plugin.

   flocker	<Object>
     Flocker represents a Flocker volume attached to a kubelet's host machine.
     This depends on the Flocker control service being running

   gcePersistentDisk	<Object>
     GCEPersistentDisk represents a GCE Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk

   gitRepo	<Object>
     GitRepo represents a git repository at a particular revision. DEPRECATED:
     GitRepo is deprecated. To provision a container with a git repo, mount an
     EmptyDir into an InitContainer that clones the repo using git, then mount
     the EmptyDir into the Pod's container.

   glusterfs	<Object>
     Glusterfs represents a Glusterfs mount on the host that shares a pod's
     lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md

   hostPath	<Object>
     HostPath represents a pre-existing file or directory on the host machine
     that is directly exposed to the container. This is generally used for
     system agents or other privileged things that are allowed to see the host
     machine. Most containers will NOT need this. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

   iscsi	<Object>
     ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
     host machine and then exposed to the pod. More info:
     https://examples.k8s.io/volumes/iscsi/README.md

   name	<string> -required-
     Volume's name. Must be a DNS_LABEL and unique within the pod. More info:
     https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

   nfs	<Object>
     NFS represents an NFS mount on the host that shares a pod's lifetime More
     info: https://kubernetes.io/docs/concepts/storage/volumes#nfs

   persistentVolumeClaim	<Object>
     PersistentVolumeClaimVolumeSource represents a reference to a
     PersistentVolumeClaim in the same namespace. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

   photonPersistentDisk	<Object>
     PhotonPersistentDisk represents a PhotonController persistent disk attached
     and mounted on kubelets host machine

   portworxVolume	<Object>
     PortworxVolume represents a portworx volume attached and mounted on
     kubelets host machine

   projected	<Object>
     Items for all in one resources secrets, configmaps, and downward API

   quobyte	<Object>
     Quobyte represents a Quobyte mount on the host that shares a pod's lifetime

   rbd	<Object>
     RBD represents a Rados Block Device mount on the host that shares a pod's
     lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md

   scaleIO	<Object>
     ScaleIO represents a ScaleIO persistent volume attached and mounted on
     Kubernetes nodes.

   secret	<Object>
     Secret represents a secret that should populate this volume. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#secret

   storageos	<Object>
     StorageOS represents a StorageOS volume attached and mounted on Kubernetes
     nodes.

   vsphereVolume	<Object>
     VsphereVolume represents a vSphere volume attached and mounted on kubelets
     host machine

PersistentVolume（PV）：是集群中由管理员配置的一段网络存储。 它是集群中的资源，就像节点是集群资源一样。 PV是容量插件，如Volumes，但其生命周期独立于使用PV的任何单个pod。 此API对象捕获存储实现的详细信息，包括NFS，iSCSI或特定于云提供程序的存储系统。

**PersistentVolumeClaim（PVC）**是由用户进行存储的请求。 它类似于pod。 Pod消耗节点资源，PVC消耗PV资源。Pod可以请求特定级别的资源（CPU和内存）。声明可以请求特定的大小和访问模式（例如，可以一次读/写或多次只读）。

虽然PersistentVolumeClaims允许用户使用抽象存储资源，但是PersistentVolumes对于不同的问题，用户通常需要具有不同属性（例如性能）。群集管理员需要能够提供各种PersistentVolumes不同的方式，而不仅仅是大小和访问模式，而不会让用户了解这些卷的实现方式。对于这些需求，有StorageClass 资源。

StorageClass为管理员提供了一种描述他们提供的存储的“类”的方法。 不同的类可能映射到服务质量级别，或备份策略，或者由群集管理员确定的任意策略。 Kubernetes本身对于什么类别代表是不言而喻的。 这个概念有时在其他存储系统中称为“配置文件”。

persistentVolumeClaim -->PVC(存储卷创建申请)
当你需要创建一个存储卷时，只需要进行申请对应的存储空间即可使用，这就是PVC。注意：PVC和PV是一一对应的。

上图解析：在Pod上定义一个PVC，该PVC要关联到当前名称空间的PVC资源，该PVC只是一个申请，PVC需要和PV进行关联。PV属于存储上的一部分存储空间。但是该方案存在的问题是，我们无法知道用户是什么时候去创建Pod，也不知道创建Pod时定义多大的PVC，那么如何实现按需创建呢？？？

不需要PV层，把所有存储空间抽象出来，这一个抽象层称为存储类，当用户创建PVC需要用到PV时，可以向存储类申请对应的存储空间，存储类会按照需求创建对应的存储空间，这就是PV的动态供给，如图：


总结：
k8s要使用存储卷，需要2步：
1、在pod定义volume，并指明关联到哪个存储设备
2、在容器使用volume mount进行挂载

二、emptyDir存储卷演示

一个emptyDir 第一次创建是在一个pod被指定到具体node的时候，并且会一直存在在pod的生命周期当中，正如它的名字一样，它初始化是一个空的目录，pod中的容器都可以读写这个目录，这个目录可以被挂在到各个容器相同或者不相同的的路径下。当一个pod因为任何原因被移除的时候，这些数据会被永久删除。注意：一个容器崩溃了不会导致数据的丢失，因为容器的崩溃并不移除pod.
emptyDir 磁盘的作用：
（1）普通空间，基于磁盘的数据存储
（2）作为从崩溃中恢复的备份点
（3）存储那些那些需要长久保存的数据，例web服务中的数据
默认的，emptyDir 磁盘会存储在主机所使用的媒介上，可能是SSD，或者网络硬盘，这主要取决于你的环境。当然，我们也可以将emptyDir.medium的值设置为Memory来告诉Kubernetes 来挂在一个基于内存的目录tmpfs，因为
tmpfs速度会比硬盘块度了，但是，当主机重启的时候所有的数据都会丢失。

[root@master ~]# kubectl explain pods.spec.volumes.emptyDir
KIND:     Pod
VERSION:  v1

RESOURCE: emptyDir <Object>

DESCRIPTION:
     EmptyDir represents a temporary directory that shares a pod's lifetime.
     More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir

     Represents an empty directory for a pod. Empty directory volumes support
     ownership management and SELinux relabeling.

FIELDS:
   medium	<string>
     What type of storage medium should back this directory. The default is ""
     which means to use the node's default medium. Must be an empty string
     (default) or Memory. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#emptydir

   sizeLimit	<string>
     Total amount of local storage required for this EmptyDir volume. The size
     limit is also applicable for memory medium. The maximum usage on memory
     medium EmptyDir would be the minimum value between the SizeLimit specified
     here and the sum of memory limits of all containers in a pod. The default
     is nil which means that the limit is undefined. More info:
     http://kubernetes.io/docs/user-guide/volumes#emptydir

[root@master kube_manifest]# vim pod-vol-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  namespace: default
  labels:
    app: myapp
    tier: frontend
  #annotations:
   # chenwj.com/create-by:"cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    volumeMounts:    #在容器内定义挂载存储名称和挂载路径
    - name: html
      mountPath: /usr/share/nginx/html/
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: html
      mountPath: /data/    #在容器内定义挂载存储名称和挂载路径
    command: ['/bin/sh','-c','for i in 1 2 3;do echo $(date) >> /data/index.html;sleep 15;done']
  volumes:  #定义存储卷
  - name: html    #定义存储卷名称  
    emptyDir: {}  #定义存储卷类型
                                

[root@master kube_manifest]# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
......
pod-demo                              2/2     Running   1          76s   10.244.2.37   node2   <none>           <none>
......

通过上面的资源清单配置，我们定义了2个容器，其中一个容器是输入日期到index.html中，然后访问nginx的html页面是否可以获取日期。以验证两个容器之间挂载的emptyDir实现共享:
[root@master kube_manifest]# curl 10.244.2.37
Sun Jun 21 08:29:36 UTC 2020
Sun Jun 21 08:29:51 UTC 2020
Sun Jun 21 08:30:06 UTC 2020
Sun Jun 21 08:30:23 UTC 2020
Sun Jun 21 08:30:38 UTC 2020
Sun Jun 21 08:30:53 UTC 2020

三、hostPath存储卷演示

hostPath宿主机路径，就是把pod所在的宿主机之上的脱离pod中的容器名称空间的之外的宿主机的文件系统的某一目录和pod建立关联关系，在pod删除时，存储数据不会丢失。

1、查看hostPath存储类型定义

[root@master kube_manifest]# kubectl explain pods.spec.volumes.hostPath 
KIND:     Pod
VERSION:  v1

RESOURCE: hostPath <Object>

DESCRIPTION:
     HostPath represents a pre-existing file or directory on the host machine
     that is directly exposed to the container. This is generally used for
     system agents or other privileged things that are allowed to see the host
     machine. Most containers will NOT need this. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

     Represents a host path mapped into a pod. Host path volumes do not support
     ownership management or SELinux relabeling.

FIELDS:
   path	<string> -required-
     Path of the directory on the host. If the path is a symlink, it will follow
     the link to the real path. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

   type	<string>
     Type for HostPath Volume Defaults to "" More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

  type：   
    DirectoryOrCreate  宿主机上不存在则创建此目录  
    Directory 必须存在挂载目录  
    FileOrCreate 宿主机上不存在挂载文件则创建  
    File 必须存在文件  

2、定义资源清单

[root@k8s-master volumes]# vim pod-hostpath-vol.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-vol-hostpath
  namespace: default
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  volumes:
    - name: html
      hostPath:
        path: /data/pod/volume1
        type: DirectoryOrCreate
        
[root@master kube_manifest]# kubectl apply -f pod-hostpath-vol.yaml 
pod/pod-vol-hostpath created

3、创建pod，挂载目录会自动在pod所运行的节点上被创建。

##pod调度到了node1节点上
[root@master kube_manifest]# kubectl get pod -o wide
......
pod-vol-hostpath                      1/1     Running   0          16h   10.244.1.48   node1   <none>           <none>
.....
##在node1上可以看到挂载目录被自动创建出来了
[root@node1 ~]# cd /data/pod/volume1/
[root@node1 volume1]# pwd
/data/pod/volume1
##在目录下创建测试页面
[root@node1 volume1]# vim index.html
node1.chenwj.com          

（4）访问测试

[root@node2 ~]# curl  10.244.1.48
node1.chenwj.com
##即使删除了pod，主机节点上的挂载目录依然存在，只要下次pod再被调度到该节点上时就可继续访问。
[root@node1 volume1]# cat index.html 
node1.chenwj.com
[root@node1 volume1]# pwd
/data/pod/volume1

注意：hostPath可以实现持久存储，但是在node节点故障时，也会导致数据的丢失

四、nfs共享存储卷演示

我们可以在pod中使用nfs服务挂载共享卷，和emptyDir不同的是，emptyDir会被删除当我们的Pod被删除的时候，但是nfs不会被删除，仅仅是解除挂在状态而已。这就意味着NFS能够允许我们提前对数据进行处理，而且这些数据可以在Pod之间相互传递。并且，nfs可以同时被多个pod挂载并同时进行读写操作（nfs支持多用户读写操作）。

注意：必须先保证NFS服务器正常运行，然后再在pod中定义nfs挂载
（1）在集群外部节点上安装nfs，并配置nfs服务

[root@centos7 ~]# yum install -y nfs-utils  ##nfs服务器的地址为192.168.147.131
[root@centos7 ~]# mkdir -pv /data/volumes 
[root@centos7 ~]# vim /etc/exports
[root@centos7 data]# vim /etc/exports
/data/volumes  192.168.147.0/24(rw,no_root_squash)
[root@centos7 data]# systemctl start nfs
[root@centos7 data]# systemctl enabled nfs
[root@centos7 data]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

[root@centos7 data]# showmount -e
Export list for centos7.cwj.com:
/data/volumes 192.168.147.0/24

（2）在node1和node2节点上安装nfs-utils，并测试挂载

[root@node1 ~]# yum install -y nfs-utils
[root@node2 ~]# yum install -y nfs-utils
[root@node2 ~]# mkdir /mnt/volume
[root@node2 ~]# mount -t nfs 192.168.147.131:/data/volumes /mnt/volume
[root@node2 ~]# mount
......
192.168.147.131:/data/volumes on /mnt/volume type nfs4 (rw,relatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.147.134,local_lock=none,addr=192.168.147.131)

[root@node2 ~]# umount /mnt/

（3）创建nfs存储卷的pod资源清单

[root@master kube_manifest]# vim pod-nfs-vol.yaml 

apiVersion: v1
kind: Pod
metadata:
  name: pod-vol-nfs
  namespace: default
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  volumes:
    - name: html
      nfs:
        path: /data/volumes
        server: 192.168.147.131

[root@master kube_manifest]# kubectl apply -f pod-nfs-vol.yaml 
pod/pod-vol-nfs created
[root@master kube_manifest]# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
pod-vol-nfs                           1/1     Running   0          31s   10.244.2.43   node2   <none>           <none>

（4）在nfs服务器上创建index.html

[root@centos7 ~]# cd /data/volumes/
[root@centos7 volumes]# vim index.html
<h1> nfs server 131</h1>
##访问测试
[root@master kube_manifest]# curl 10.244.2.43
<h1> nfs server 131</h1>
#删除pod，再重新创建，任然可以得到数据的持久化存储
[root@master kube_manifest]#  kubectl delete -f pod-nfs-vol.yaml
pod "pod-vol-nfs" deleted
[root@master kube_manifest]# kubectl apply -f pod-nfs-vol.yaml 
pod/pod-vol-nfs created
[root@master kube_manifest]# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
......
pod-vol-nfs                           1/1     Running   0          67s   10.244.2.44   node2   <none>           <none>
[root@master kube_manifest]# curl 10.244.2.44
<h1> nfs server 131</h1>

五、PVC和PV的概念

我们前面提到kubernetes提供那么多存储接口，但是首先kubernetes的各个Node节点能管理这些存储，但是各种存储参数也需要专业的存储工程师才能了解，由此我们的kubernetes管理变的更加复杂的。由此kubernetes提出了PV和PVC的概念，这样开发人员和使用者就不需要关注后端存储是什么，使用什么参数等问题。如下图：

PersistentVolume（PV）是集群中已由管理员配置的一段网络存储。 集群中的资源就像一个节点是一个集群资源。 PV是诸如卷之类的卷插件，但是具有独立于使用PV的任何单个pod的生命周期。 该API对象捕获存储的实现细节，即NFS，iSCSI或云提供商特定的存储系统。

PersistentVolumeClaim（PVC）是用户存储的请求。PVC的使用逻辑：在pod中定义一个存储卷（该存储卷类型为PVC），定义的时候直接指定大小，pvc必须与对应的pv建立关系，pvc会根据定义去pv申请，而pv是由存储空间创建出来的。pv和pvc是kubernetes抽象出来的一种存储资源。

虽然PersistentVolumeClaims允许用户使用抽象存储资源，但是常见的需求是，用户需要根据不同的需求去创建PV，用于不同的场景。而此时需要集群管理员提供不同需求的PV，而不仅仅是PV的大小和访问模式，但又不需要用户了解这些卷的实现细节。 对于这样的需求，此时可以采用StorageClass资源。这个在前面就已经提到过此方案。

PV是集群中的资源。 PVC是对这些资源的请求，也是对资源的索赔检查。 PV和PVC之间的相互作用遵循这个生命周期：

Provisioning（配置）—> Binding（绑定）—>Using（使用）—> Releasing（释放） —> Recycling（回收）

Provisioning

这里有两种PV的提供方式:静态或者动态

静态–>直接固定存储空间：
    集群管理员创建一些 PV。它们携带可供集群用户使用的真实存储的详细信息。 它们存在于Kubernetes API中，可用于消费。

动态–>通过存储类进行动态创建存储空间：
    当管理员创建的静态 PV 都不匹配用户的 PVC 时，集群可能会尝试动态地为 PVC 配置卷。此配置基于 StorageClasses：PVC 必须请求存储类，并且管理员必须已创建并配置该类才能进行动态配置。 要求该类的声明有效地为自己禁用动态配置。

Binding

在动态配置的情况下，用户创建或已经创建了具有特定数量的存储请求和特定访问模式的PersistentVolumeClaim。 主机中的控制回路监视新的PVC，找到匹配的PV（如果可能），并将 PVC 和 PV 绑定在一起。 如果为新的PVC动态配置PV，则循环将始终将该PV绑定到PVC。 否则，用户总是至少得到他们要求的内容，但是卷可能超出了要求。 一旦绑定，PersistentVolumeClaim绑定是排他的，不管用于绑定它们的模式。

如果匹配的卷不存在，PVC将保持无限期。 随着匹配卷变得可用，PVC将被绑定。 例如，提供许多50Gi PV的集群将不匹配要求100Gi的PVC。 当集群中添加100Gi PV时，可以绑定PVC。

Using

Pod使用PVC作为卷。 集群检查声明以找到绑定的卷并挂载该卷的卷。 对于支持多种访问模式的卷，用户在将其声明用作pod中的卷时指定所需的模式。

一旦用户有声明并且该声明被绑定，绑定的PV属于用户，只要他们需要它。 用户通过在其Pod的卷块中包含PersistentVolumeClaim来安排Pods并访问其声明的PV。

Releasing

当用户完成卷时，他们可以从允许资源回收的API中删除PVC对象。 当声明被删除时，卷被认为是“释放的”，但是它还不能用于另一个声明。 以前的索赔人的数据仍然保留在必须根据政策处理的卷上.

Reclaiming

PersistentVolume的回收策略告诉集群在释放其声明后，该卷应该如何处理。 目前，卷可以是保留，回收或删除。 保留可以手动回收资源。 对于那些支持它的卷插件，删除将从Kubernetes中删除PersistentVolume对象，以及删除外部基础架构（如AWS EBS，GCE PD，Azure Disk或Cinder卷）中关联的存储资产。 动态配置的卷始终被删除

Recycling

如果受适当的卷插件支持，回收将对卷执行基本的擦除（rm -rf / thevolume / *），并使其再次可用于新的声明。

六、NFS使用PV和PVC

查看pv的定义方式

[root@k8s-master ~]# kubectl explain pv 
KIND:     PersistentVolume
VERSION:  v1 
FIELDS:
	apiVersion
	kind
	metadata
	spec
	status
[root@k8s-master ~]# kubectl explain pv.spec    #查看pv定义的规格
spec:
  nfs                    ##定义存储类型
    path                 ##定义挂载卷路径
    server               ##定义服务器名称
  accessModes            ##定义访问模型，有以下三种访问模型，以列表的方式存在，也就是说可以定义多个访问模式
    ReadWriteOnce（RWO）   ##单节点读写
	ReadOnlyMany（ROX）    ##多节点只读
	ReadWriteMany（RWX）   ##多节点读写
  capacity                ##定义PV空间的大小
    storage               ##指定大小

查看PVC的定义方式

[root@k8s-master volumes]# kubectl explain pvc   
KIND:     PersistentVolumeClaim
VERSION:  v1
FIELDS:
   apiVersion	<string>
   kind	<string>  
   metadata	<Object>
   spec	<Object>
[root@k8s-master volumes]# kubectl explain pvc.spec
spec:
  accessModes   ##定义访问模式，必须是PV的访问模式的子集
  resources     ##定义申请资源的大小
    requests:
      

1、配置nfs存储

[root@centos7 volumes]# mkdir v{1,2,3,4,5}

[root@centos7 volumes]# vim /etc/exports
/data/volumes/v1 192.168.147.0/24(rw,no_root_squash)
/data/volumes/v2 192.168.147.0/24(rw,no_root_squash)
/data/volumes/v3 192.168.147.0/24(rw,no_root_squash)
/data/volumes/v4 192.168.147.0/24(rw,no_root_squash)
/data/volumes/v5 192.168.147.0/24(rw,no_root_squash)
                                                
[root@centos7 volumes]# exportfs -arv
exporting 192.168.147.0/24:/data/volumes/v5
exporting 192.168.147.0/24:/data/volumes/v4
exporting 192.168.147.0/24:/data/volumes/v3
exporting 192.168.147.0/24:/data/volumes/v2
exporting 192.168.147.0/24:/data/volumes/v1

[root@centos7 volumes]# showmount -e
Export list for centos7.cwj.com:

/data/volumes/v5 192.168.147.0/24
/data/volumes/v4 192.168.147.0/24
/data/volumes/v3 192.168.147.0/24
/data/volumes/v2 192.168.147.0/24
/data/volumes/v1 192.168.147.0/24

2、定义PV
这里定义5个PV，并且定义挂载的路径以及访问模式，还有PV划分的大小。
注意：定义PV的时候不要指定名称空间，因为PV是属于集群级别的，每个名称空间都可以调用，PVC是属于名称空间级别的

[root@master kube_manifest]# cat pv-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv001
  labels:
    name: pv001
spec:
  nfs:
    path: /data/volumes/v1
    server: 192.168.147.131
  accessModes: ["ReadWriteMany","ReadWriteOnce"]
  capacity:
    storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv002
  labels:
    name: pv002
spec:
  nfs:
    path: /data/volumes/v2
    server: 192.168.147.131
  accessModes: ["ReadWriteOnce"]
  capacity:
    storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv003
  labels:
    name: pv003
spec:
  nfs:
    path: /data/volumes/v3
    server: 192.168.147.131
  accessModes: ["ReadWriteMany","ReadWriteOnce"]
  capacity:
    storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv004
  labels:
    name: pv004
spec:
  nfs:
    path: /data/volumes/v4
    server: 192.168.147.131
  accessModes: ["ReadWriteMany","ReadWriteOnce"]
  capacity:
    storage: 4Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv005
  labels:
    name: pv005
spec:
  nfs:
    path: /data/volumes/v5
    server: 192.168.147.131
  accessModes: ["ReadWriteMany","ReadWriteOnce"]
  capacity:
    storage: 5Gi

[root@master kube_manifest]# kubectl apply -f pv-demo.yaml 
persistentvolume/pv001 created
persistentvolume/pv002 created
persistentvolume/pv003 created
persistentvolume/pv004 created
persistentvolume/pv005 created
[root@master kube_manifest]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                   23s
pv002   2Gi        RWO            Retain           Available                                   23s
pv003   2Gi        RWO,RWX        Retain           Available                                   23s
pv004   4Gi        RWO,RWX        Retain           Available                                   23s
pv005   5Gi        RWO,RWX        Retain           Available                                   23s

3、定义PVC
这里定义了pvc的访问模式为多路读写，该访问模式必须属于前面pv定义的访问模式之中（子集）。定义PVC申请的大小为2Gi，此时PVC会自动去匹配多路读写且大小为2Gi的PV，匹配成功获取PVC的状态即为Bound。

[root@master kube_manifest]# vim pod-vol-pvc.yaml 

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  namespace: default
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-vol-pvc
  namespace: default
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  volumes:
    - name: html
      persistentVolumeClaim:
        claimName: mypvc
                        

[root@master kube_manifest]# kubectl apply -f pod-vol-pvc.yaml 
persistentvolumeclaim/mypvc created
pod/pod-vol-pvc created
[root@master kube_manifest]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM           STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                           25m
pv002   2Gi        RWO            Retain           Available                                           25m
pv003   2Gi        RWO,RWX        Retain           Bound       default/mypvc                           25m
pv004   4Gi        RWO,RWX        Retain           Available                                           25m
pv005   5Gi        RWO,RWX        Retain           Available                                           25m
[root@master kube_manifest]# kubectl get pvc
NAME    STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc   Bound    pv003    2Gi        RWO,RWX                       4m55s

4、测试访问
在存储服务器上创建index.html，并写入数据，通过访问Pod进行查看，可以获取到相应的页面。

[root@centos7 volumes]# cd ./v3
[root@centos7 v3]#  echo "pv3 storage" > index.html
[root@master kube_manifest]# kubectl get pods -o wide | grep pod-vol-pvc
pod-vol-pvc                           1/1     Running   0          14m    10.244.2.45   node2   <none>           <none>
[root@master kube_manifest]# curl 10.244.2.45
pv3 storage

七、StorageClass

在pv和pvc使用过程中存在的问题，在pvc申请存储空间时，未必就有现成的pv符合pvc申请的需求，上面nfs在做pvc可以成功的因素是因为我们做了指定的需求处理。那么当PVC申请的存储空间不一定有满足PVC要求的PV事，又该如何处理呢？？？为此，Kubernetes为管理员提供了描述存储"class（类）"的方法（StorageClass）。举个例子，在存储系统中划分一个1TB的存储空间提供给Kubernetes使用，当用户需要一个10G的PVC时，会立即通过restful发送请求，从而让存储空间创建一个10G的image，之后在我们的集群中定义成10G的PV供给给当前的PVC作为挂载使用。在此之前我们的存储系统必须支持restful接口，比如ceph分布式存储，而glusterfs则需要借助第三方接口完成这样的请求。如图：

[root@master kube_manifest]#  kubectl explain storageclass
KIND:     StorageClass
VERSION:  storage.k8s.io/v1

DESCRIPTION:
     StorageClass describes the parameters for a class of storage for which
     PersistentVolumes can be dynamically provisioned. StorageClasses are
     non-namespaced; the name of the storage class according to etcd is in
     ObjectMeta.Name.

FIELDS:
   allowVolumeExpansion	<boolean>
     AllowVolumeExpansion shows whether the storage class allow volume expand

   allowedTopologies	<[]Object>
     Restrict the node topologies where volumes can be dynamically provisioned.
     Each volume plugin defines its own supported topology specifications. An
     empty TopologySelectorTerm list means there is no topology restriction.
     This field is only honored by servers that enable the VolumeScheduling
     feature.

   apiVersion	<string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind	<string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata	<Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   mountOptions	<[]string>    #挂载选项
     Dynamically provisioned PersistentVolumes of this storage class are created
     with these mountOptions, e.g. ["ro", "soft"]. Not validated - mount of the
     PVs will simply fail if one is invalid.

   parameters	<map[string]string>
     Parameters holds the parameters for the provisioner that should create
     volumes of this storage class.

   provisioner	<string> -required-  #存储分配器，用来决定使用哪个卷插件分配 PV。该字段必须指定。
     Provisioner indicates the type of the provisioner.

   reclaimPolicy	<string>   #回收策略，可以是 Delete 或者 Retain。如果 StorageClass 对象被创建时没有指定 reclaimPolicy ，它将默认为 Delete。 
     Dynamically provisioned PersistentVolumes of this storage class are created
     with this reclaimPolicy. Defaults to Delete.

   volumeBindingMode	<string>  #卷的绑定模式
     VolumeBindingMode indicates how PersistentVolumeClaims should be
     provisioned and bound. When unset, VolumeBindingImmediate is used. This
     field is only honored by servers that enable the VolumeScheduling feature.

StorageClass 中包含 provisioner、parameters 和 reclaimPolicy 字段，当 class 需要动态分配 PersistentVolume 时会使用到。由于StorageClass需要一个独立的存储系统，此处就不再演示。从其他资料查看定义StorageClass的方式如下：

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug

八、配置容器应用：Secret和configMap

在日常单机甚至集群状态下，我们需要对一个应用进行配置，只需要修改其配置文件即可。那么在容器中又该如何提供配置 信息呢？？？例如，为Nginx配置一个指定的server_name或worker进程数，为Tomcat的JVM配置其堆内存大小。传统的实践过程中通常有以下几种方式：

启动容器时，通过命令传递参数

将定义好的配置文件通过镜像文件进行写入

通过环境变量的方式传递配置数据

挂载Docker卷传送配置文件

而在Kubernetes系统之中也存在这样的组件，就是特殊的存储卷类型。其并不是提供pod存储空间，而是给管理员或用户提供从集群外部向Pod内部的应用注入配置信息的方式。这两种特殊类型的存储卷分别是：configMap和secret

Secret：用于向Pod传递敏感信息，比如密码，私钥，证书文件等，这些信息如果在容器中定义容易泄露，Secret资源可以让用户将这些信息存储在急群众，然后通过Pod进行挂载，实现敏感数据和系统解耦的效果。

ConfigMap：主要用于向Pod注入非敏感数据，使用时，用户将数据直接存储在ConfigMap对象当中，然后Pod通过使用ConfigMap卷进行引用，实现容器的配置文件集中定义和管理。

[root@master ~]# kubectl explain pods.spec.volumes
KIND:     Pod
VERSION:  v1

RESOURCE: volumes <[]Object>

DESCRIPTION:
     List of volumes that can be mounted by containers belonging to the pod.
     More info: https://kubernetes.io/docs/concepts/storage/volumes

     Volume represents a named volume in a pod that may be accessed by any
     container in the pod.

FIELDS:
 
   configMap	<Object>
     ConfigMap represents a configMap that should populate this volume

   secret	<Object>
     Secret represents a secret that should populate this volume. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#secret

8.1、ConifgMap解析
configmap是让配置文件从镜像中解耦，让镜像的可移植性和可复制性。许多应用程序会从配置文件、命令行参数或环境变量中读取配置信息。这些配置信息需要与docker image解耦，你总不能每修改一个配置就重做一个image吧？ConfigMap API给我们提供了向容器中注入配置信息的机制，ConfigMap可以被用来保存单个属性，也可以用来保存整个配置文件或者JSON二进制大对象。
ConfigMap API资源用来保存key-value pair配置数据，这个数据可以在pods里使用，或者被用来为像controller一样的系统组件存储配置数据。虽然ConfigMap跟Secrets类似，但是ConfigMap更方便的处理不含敏感信息的字符串。 注意：ConfigMaps不是属性配置文件的替代品。ConfigMaps只是作为多个properties文件的引用。可以把它理解为Linux系统中的/etc目录，专门用来存储配置文件的目录。下面举个例子，使用ConfigMap配置来创建Kuberntes Volumes，ConfigMap中的每个data项都会成为一个新文件。

[root@master ~]#  kubectl explain configmap
KIND:     ConfigMap
VERSION:  v1

DESCRIPTION:
     ConfigMap holds configuration data for pods to consume.

FIELDS:
   apiVersion	<string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   binaryData	<map[string]string>
     BinaryData contains the binary data. Each key must consist of alphanumeric
     characters, '-', '_' or '.'. BinaryData can contain byte sequences that are
     not in the UTF-8 range. The keys stored in BinaryData must not overlap with
     the ones in the Data field, this is enforced during validation process.
     Using this field will require 1.10+ apiserver and kubelet.

   data	<map[string]string>
     Data contains the configuration data. Each key must consist of alphanumeric
     characters, '-', '_' or '.'. Values with non-UTF-8 byte sequences must use
     the BinaryData field. The keys stored in Data must not overlap with the
     keys in the BinaryData field, this is enforced during validation process.

   immutable	<boolean>
     Immutable, if set to true, ensures that data stored in the ConfigMap cannot
     be updated (only object metadata can be modified). If not set to true, the
     field can be modified at any time. Defaulted to nil. This is an alpha field
     enabled by ImmutableEphemeralVolumes feature gate.

   kind	<string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata	<Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

ConfigMap 支持四种创建方式

[root@master ~]#  kubectl create configmap --help
......
Aliases:
configmap, cm

Examples:
  # Create a new configmap named my-config based on folder bar
  kubectl create configmap my-config --from-file=path/to/bar
  
  # Create a new configmap named my-config with specified keys instead of file basenames on disk
  kubectl create configmap my-config --from-file=key1=/path/to/bar/file1.txt --from-file=key2=/path/to/bar/file2.txt
  
  # Create a new configmap named my-config with key1=config1 and key2=config2
  kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
  
  # Create a new configmap named my-config from the key=value pairs in the file
  kubectl create configmap my-config --from-file=path/to/bar
  
  # Create a new configmap named my-config from an env file
  kubectl create configmap my-config --from-env-file=path/to/bar.env
......

1、 通过 --from-literal
每个 --from-literal 对应一个信息条目。

[root@master ~]#  kubectl create configmap nginx-config --from-literal=nginx_port=80 --from-literal=server_name=www.cwj.com
configmap/nginx-config created
[root@master ~]#  kubectl get cm 
NAME           DATA   AGE
nginx-config   2      27s
[root@master ~]# kubectl describe cm nginx-config
Name:         nginx-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
nginx_port:
----
80
server_name:
----
www.cwj.com
Events:  <none>

2、通过 --from-file：
每个文件内容对应一个信息条目。

[root@master ~]# mkdir configmap && cd configmap
[root@master configmap]#  vim www.conf

server {
        server_name myapp.cwj.com;
        listen 80;
        root /data/web/html;
}
[root@master configmap]#  kubectl get cm 
NAME           DATA   AGE
nginx-config   2      8m55s
nginx-www      1      10s
[root@master configmap]#  kubectl get cm nginx-www -o yaml
apiVersion: v1
data:
  www.conf: "server {\n\tserver_name myapp.cwj.com;\n\tlisten 80;\n\troot /data/web/html;\n}\n"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-17T09:36:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:www.conf: {}
    manager: kubectl
    operation: Update
    time: "2020-11-17T09:36:02Z"
  name: nginx-www
  namespace: default
  resourceVersion: "1958148"
  selfLink: /api/v1/namespaces/default/configmaps/nginx-www
  uid: d04335b9-7248-4f88-bd8d-d610e00899b5

使用configMap
1、环境变量方式注入到pod

[root@master kube_manifest]# vim pod-configmap.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-cm-1
  namespace: default
  labels: 
    app: myapp
    tier: frontend
  annotations:
    magedu.com/created-by: "cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80 
    env:
    - name: NGINX_SERVER_PORT
      valueFrom:
        configMapKeyRef:
          name: nginx-config  ##指明调用哪个configmap,这里调用nginx-config这个configmap
          key: nginx_port   ##指明调用configmap中的哪个key
    - name: NGINX_SERVER_NAME
      valueFrom:
        configMapKeyRef:
          name: nginx-config
          key: server_name
        
[root@master kube_manifest]# kubectl apply -f pod-configmap.yaml
pod/pod-cm-1 created
[root@master kube_manifest]# kubectl exec -it pod-cm-1 -- /bin/sh
/ # echo $NGINX_SERVER_PORT
80
/ # echo $NGINX_SERVER_NAME
www.cwj.com
/ # 
               

此时，修改nginx-config这个configmap的端口

[root@master kube_manifest]# kubectl edit cm nginx-config

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  nginx_port: "8080"  ##把端口改成8080
  server_name: www.cwj.com
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-17T09:27:17Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:nginx_port: {}
        f:server_name: {}
    manager: kubectl
    operation: Update
    time: "2020-11-17T09:27:17Z"
  name: nginx-config
  namespace: default
  resourceVersion: "1956809"
  selfLink: /api/v1/namespaces/default/configmaps/nginx-config
  uid: e24549e1-32c5-409c-95ec-ef36ab4720e6
  
[root@master kube_manifest]# kubectl edit cm nginx-config
configmap/nginx-config edited
[root@master kube_manifest]# kubectl describe cm nginx-config
Name:         nginx-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
nginx_port:
----
8080
server_name:
----
www.cwj.com
Events:  <none>

查看已经使用该configmap的pod的环境变量是否发生改变。可以发现使用环境变化注入pod中的端口不会根据配置的更改而变化。

/ # echo $NGINX_SERVER_PORT
80

2、存储卷方式挂载configmap：
Volume 形式的 ConfigMap 也支持动态更新

[root@master kube_manifest]# vim pod-configmap-2.yaml 

apiVersion: v1
kind: Pod
metadata:
  name: pod-cm-2
  namespace: default
  labels: 
    app: myapp
    tier: frontend
  annotations:
    magedu.com/created-by: "cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80 
    volumeMounts:
    - name: nginxconf  ##指明挂载哪个存储卷
      mountPath: /etc/nginx/config.d/
      readOnly: true
  volumes:
  - name: nginxconf  ##定义存储卷的名称
    configMap:        ##定义存储卷的类型为configmap
      name: nginx-config  ##指明把哪个configmap定义为存储卷
      
[root@master kube_manifest]# kubectl apply -f pod-configmap-2.yaml 
pod/pod-cm-2 created

[root@master kube_manifest]# kubectl exec -it pod-cm-2 -- /bin/sh
/ # cd /etc/nginx/config.d
/etc/nginx/config.d # cat nginx_port
8080/etc/nginx/config.d # cat server_name 
www.cwj.com/etc/nginx/config.d # ls
nginx_port   server_name

此时修改configmap端口，再查看pod的端口是否变化

[root@master kube_manifest]# kubectl edit cm nginx-config

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  nginx_port: "80888"  ##修改端口
  server_name: www.cwj.com
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-17T09:27:17Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:nginx_port: {}
        f:server_name: {}
    manager: kubectl
    operation: Update
    time: "2020-11-17T10:59:28Z"
  name: nginx-config
  namespace: default
  resourceVersion: "1970908"
  selfLink: /api/v1/namespaces/default/configmaps/nginx-config
  uid: e24549e1-32c5-409c-95ec-ef36ab4720e6
~                                            
[root@master kube_manifest]# kubectl edit cm nginx-config
configmap/nginx-config edited

再次查看发现端口已经变化

/etc/nginx/config.d # cat nginx_port
80888/etc/nginx/config.d #    

3、以nginx-www这个configmap配置pod

[root@master kube_manifest]# kubectl get cm 
NAME           DATA   AGE
nginx-config   2      14h
nginx-www      1      14h

[root@master kube_manifest]# vim pod-configmap-3.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-cm-3
  namespace: default
  labels: 
    app: myapp
    tier: frontend
  annotations:
    magedu.com/created-by: "cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80 
    volumeMounts:
    - name: nginxconf
      mountPath: /etc/nginx/conf.d/
      readOnly: true
  volumes:
  - name: nginxconf
    configMap:
      name: nginx-www
[root@master kube_manifest]# kubectl apply -f pod-configmap-3.yaml
pod/pod-cm-3 created
[root@master kube_manifest]#  kubectl get pods -o wide | grep pod-cm-3
pod-cm-3                              1/1     Running       0          37m     10.244.2.48   node2   <none>           <none>
[root@master kube_manifest]# kubectl exec -it pod-cm-3 -- /bin/sh
/ # cd /etc/nginx/conf.d/
/etc/nginx/conf.d # ls
www.conf
/etc/nginx/conf.d # cat www.conf 
server {
	server_name myapp.cwj.com;
	listen 80;
	root /data/web/html;
}
/etc/nginx/conf.d # 
/ # nginx -T
......
# configuration file /etc/nginx/conf.d/www.conf:
server {
	server_name myapp.cwj.com;
	listen 80;
	root /data/web/html;
}

##创建web页面
/ # mkdir -pv /data/web/html
created directory: '/data/'
created directory: '/data/web/'
created directory: '/data/web/html'
/ # echo "nginx server" >> /data/web/html/index.html

##在节点中访问myapp.cwj.com
[root@master kube_manifest]# echo "10.244.2.48 myapp.cwj.com" >> /etc/hosts
[root@master kube_manifest]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.147.132 master
10.10.20.207    master
10.10.20.210    node1
10.10.20.202    node2
192.168.147.133 node1
192.168.147.134 node2
10.244.2.48 myapp.cwj.com
[root@master kube_manifest]# curl myapp.cwj.com
nginx server

##此时修改nginx-www这个configmap，监听端口该为8088
[root@master kube_manifest]# kubectl edit cm nginx-www

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  www.conf: "server {\n\tserver_name myapp.cwj.com;\n\tlisten 8088;\n\troot /data/web/html;\n}\n"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-17T09:36:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:www.conf: {}
    manager: kubectl
    operation: Update
    time: "2020-11-17T09:36:02Z"
  name: nginx-www
  namespace: default
  resourceVersion: "1958148"
  selfLink: /api/v1/namespaces/default/configmaps/nginx-www
  uid: d04335b9-7248-4f88-bd8d-d610e00899b5
                  
###过几分钟查看pod的监听端口情况
/ # cat /etc/nginx/conf.d/www.conf 
server {
	server_name myapp.cwj.com;
	listen 8088;
	root /data/web/html;
}
##虽然修改了configmap，但nginx的监听端口不会改变
/ # netstat -ntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN 
##重载以下才能发生改变    
/ # nginx -s reload
2020/11/18 01:15:40 [notice] 25#25: signal process started
/ # netstat -ntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN    
##节点访问
[root@master kube_manifest]# curl myapp.cwj.com:8088
nginx server
                      

在configmap挂载为存储卷时，由于configmap中会存在多个键值，有时候我们可能只使用其中的某几个键值数据，因此可以通过items字段调用named字段中所指定的configmap的某几个键值。

[root@master kube_manifest]# kubectl explain pods.spec.volumes.configMap
KIND:     Pod
VERSION:  v1

RESOURCE: configMap <Object>

DESCRIPTION:
     ConfigMap represents a configMap that should populate this volume

     Adapts a ConfigMap into a volume. The contents of the target ConfigMap's
     Data field will be presented in a volume as files using the keys in the
     Data field as the file names, unless the items element is populated with
     specific mappings of keys to paths. ConfigMap volumes support ownership
     management and SELinux relabeling.

FIELDS:
   defaultMode	<integer>
     Optional: mode bits to use on created files by default. Must be a value
     between 0 and 0777. Defaults to 0644. Directories within the path are not
     affected by this setting. This might be in conflict with other options that
     affect the file mode, like fsGroup, and the result can be other mode bits
     set.

   items	<[]Object>
     If unspecified, each key-value pair in the Data field of the referenced
     ConfigMap will be projected into the volume as a file whose name is the key
     and content is the value. If specified, the listed keys will be projected
     into the specified paths, and unlisted keys will not be present. If a key
     is specified which is not present in the ConfigMap, the volume setup will
     error unless it is marked optional. Paths must be relative and may not
     contain the '..' path or start with '..'.

   name	<string>
     Name of the referent. More info:
     https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

   optional	<boolean>
     Specify whether the ConfigMap or its keys must be defined



[root@master kube_manifest]# kubectl explain pods.spec.volumes.configMap.items
KIND:     Pod
VERSION:  v1

RESOURCE: items <[]Object>

DESCRIPTION:
     If unspecified, each key-value pair in the Data field of the referenced
     ConfigMap will be projected into the volume as a file whose name is the key
     and content is the value. If specified, the listed keys will be projected
     into the specified paths, and unlisted keys will not be present. If a key
     is specified which is not present in the ConfigMap, the volume setup will
     error unless it is marked optional. Paths must be relative and may not
     contain the '..' path or start with '..'.

     Maps a string key to a path within a volume.

FIELDS:
   key	<string> -required-
     The key to project.

   mode	<integer>
     Optional: mode bits to use on this file, must be a value between 0 and
     0777. If not specified, the volume defaultMode will be used. This might be
     in conflict with other options that affect the file mode, like fsGroup, and
     the result can be other mode bits set.

   path	<string> -required-
     The relative path of the file to map the key to. May not be an absolute
     path. May not contain the path element '..'. May not start with the string
     '..'.

8.2、Secret解析
Secret对象存储数据的方式是以键值方式存储数据，在Pod资源进行调用Secret的方式是通过环境变量或者存储卷的方式进行访问数据，解决了密码、token、密钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec中。另外，Secret对象的数据存储和打印格式为Base64编码的字符串，因此用户在创建Secret对象时，也需要提供该类型的编码格式的数据。在容器中以环境变量或存储卷的方式访问时，会自动解码为明文格式。需要注意的是，如果是在Master节点上，Secret对象以非加密的格式存储在etcd中，所以需要对etcd的管理和权限进行严格控制。

Secret有4种类型：

Service Account ：用来访问Kubernetes API，由Kubernetes自动创建，并且会自动挂载到Pod的/run/secrets/kubernetes.io/serviceaccount目录中；

Opaque ：即generic，base64编码格式的Secret，用来存储密码、密钥、信息、证书等，类型标识符为generic；

kubernetes.io/dockerconfigjson ：用来存储私有docker registry的认证信息，类型标识为docker-registry。

kubernetes.io/tls：用于为SSL通信模式存储证书和私钥文件，命令式创建类型标识为tls。

创建 Secret的2种方式

命令式创建
1、通过 --from-literal：

[root@master kube_manifest]# kubectl create secret -h
Create a secret using specified subcommand.

Available Commands:
  docker-registry 创建一个给 Docker registry 使用的 secret
  generic         从本地 file, directory 或者 literal value 创建一个 secret
  tls             创建一个 TLS secret

Usage:
  kubectl create secret [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

##每个 --from-literal 对应一个信息条目。
[root@master kube_manifest]# kubectl create secret generic mysql-root-passwd --from-literal=passwd=123456
secret/mysql-root-passwd created
[root@master kube_manifest]# kubectl get secret 
NAME                  TYPE                                  DATA   AGE
default-token-z5gjj   kubernetes.io/service-account-token   3      180d
mysql-root-passwd     Opaque                                1      29s
[root@master kube_manifest]# kubectl describe secret 
Name:         default-token-z5gjj
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 6926d5b2-44f2-4aa2-aaec-ee37462cc79c

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Imh4cjVOQmVzamZna09LYzdJS1dWcWJQc0pEb3NJVDVPWVRZVjlQakhxLXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tejVnamoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY5MjZkNWIyLTQ0ZjItNGFhMi1hYWVjLWVlMzc0NjJjYzc5YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.k_QxNI8-RYZYJtwCWxkOI5vgUgfu-4lwWejUfqHQobyGOiXGBuGlehz5DdjD5ZRmuWdSbEpBWg8hweSKpHfyfzyXQmjz4OG6Ky1Q1vtaeEp2kROa9wIu7r-YBlsZphvQcNdbEVa7ZKz-1nvaWsmXhkeFtzh7vG7n13feN9-TXi-OHiA5GKOKQsL5E_LEmfgIj1XJXf2u35RdSo8wgJjT-7VK7DkuLUKmsfgmZyNlJS75m9K9sxx_CSTOap9V4gRNElArqwFSU8hyi1Bmwdg16vJOdIpOQypUqE9bRSNmxR9YNzad6CaiaOvgXfZutOOVSQl1tn2Ew3TomE9JeE9qvQ
ca.crt:     1025 bytes
namespace:  7 bytes


Name:         mysql-root-passwd
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
passwd:  6 bytes

[root@master kube_manifest]# kubectl get secret mysql-root-passwd -o yaml
apiVersion: v1
data:
  passwd: MTIzNDU2
kind: Secret
metadata:
  creationTimestamp: "2020-11-18T02:34:41Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:passwd: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-18T02:34:41Z"
  name: mysql-root-passwd
  namespace: default
  resourceVersion: "2112605"
  selfLink: /api/v1/namespaces/default/secrets/mysql-root-passwd
  uid: 012f288a-22e4-4485-9896-61817a3bb4aa
type: Opaque

##由于采用的是base64编码，因此可以轻松的解码
[root@master kube_manifest]# echo MTIzNDU2 | base64 -d
123456

2、通过 --from-file
每个文件内容对应一个信息条目，文件名是key，文件内容是value

[root@master kube_manifest]# mkdir secret 
[root@master kube_manifest]# cd secret/
[root@master secret]# vim username
root
[root@master secret]# vim passwd
123456
[root@master secret]# kubectl create secret generic mysecret --from-file=./username --from-file=./passwd
secret/mysecret created

[root@master secret]# kubectl get secret 
NAME                  TYPE                                  DATA   AGE
default-token-z5gjj   kubernetes.io/service-account-token   3      180d
mysecret              Opaque                                2      30s
mysql-root-passwd     Opaque                                1      3h14m

[root@master secret]# kubectl get secret mysecret -o yaml
apiVersion: v1
data:
  passwd: MTIzNDU2Cg==
  username: cm9vdAoxMjM0NTYK
kind: Secret
metadata:
  creationTimestamp: "2020-11-18T05:48:26Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:passwd: {}
        f:username: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-18T05:48:26Z"
  name: mysecret
  namespace: default
  resourceVersion: "2142242"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: a9e3f650-941e-40b9-af40-762743fe073f
type: Opaque

3、通过 --from-env-file：
文件 env.txt 中每行 Key=Value 对应一个信息条目。

[root@master secret]# vim env.txt

username=root
password=123456
[root@master secret]# kubectl create secret generic mysecret2 --from-env-file=env.txt
secret/mysecret2 created
[root@master secret]# kubectl get secret mysecret2 -o yaml
apiVersion: v1
data:
  password: MTIzNDU2
  username: cm9vdA==
kind: Secret
metadata:
  creationTimestamp: "2020-11-18T05:53:33Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:password: {}
        f:username: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-18T05:53:33Z"
  name: mysecret2
  namespace: default
  resourceVersion: "2143024"
  selfLink: /api/v1/namespaces/default/secrets/mysecret2
  uid: 82afe10d-9bbe-4606-8818-f98a7d51d777
type: Opaque

清单式创建
通过 YAML 配置文件

先对敏感数据进行Base64编码

[root@master kube_manifest]# echo -n root | base64
cm9vdA==
[root@master kube_manifest]# echo -n 123456 | base64
MTIzNDU2

[root@master kube_manifest]# vim secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret-4
data:
  username: cm9vdA==
  password: MTIzNDU2

[root@master kube_manifest]# kubectl apply -f secret.yaml
secret/mysecret-4 created

[root@master kube_manifest]# kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-z5gjj   kubernetes.io/service-account-token   3      180d
mysecret-4            Opaque                                2      62s
[root@master kube_manifest]# kubectl describe secret mysecret-4
Name:         mysecret-4
Namespace:    default
Labels:       <none>
Annotations:  
Type:         Opaque

Data
====
password:  6 bytes
username:  4 bytes

使用secret
Pod 可以通过 Volume 或者环境变量的方式使用 Secret

[root@master kube_manifest]# vim pod-secret-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret
spec:
  containers:
  - name: pod-secret
    image: busybox
    args:
      - /bin/sh
      - -c
      - sleep 10;touch /tmp/healthy;sleep 30000
    volumeMounts:   #将 sec mount 到容器路径 /etc/secret，可指定读写权限为 readOnly。
    - name: sec
      mountPath: "/etc/secret"
      readOnly: true
  volumes:    #定义 volume sec，来源为 secret mysecret-4。
  - name: sec
    secret:
      secretName: mysecret-4
[root@master kube_manifest]#  kubectl apply -f pod-secret-demo.yaml 
pod/pod-secret created
[root@master kube_manifest]# kubectl exec -it pod-secret -- /bin/sh
/ # ls /etc/secret
password  username
/ # cat /etc/secret/password 
123456/ # 
/ # cat /etc/secret/username 
root/ # 
                         

可以看出，pod 会在指定的路径 /etc/secret 下为每条敏感数据创建一个文件，文件名就是数据条目的 Key，也就是/etc/secret/username和/etc/secret/password， Value 则以明文存放在文件中。

也可以只调用secret中的某个key，并自定义存放数据路径

[root@master kube_manifest]# vim pod-secret-demo-2.yaml 

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret-2
  
spec:
  containers:
  - name: pod-secret-2
    image: busybox
    args:
      - /bin/sh
      - -c
      - sleep 10;touch /tmp/healthy;sleep 30000
    volumeMounts:
    - name: sec
      mountPath: "/etc/secret"
      readOnly: true
  volumes:
  - name: sec
    secret:
      secretName: mysecret-4
      items:    #自定义调用secret中的某个key
      - key: username
        path: my-secret/my-username  ##定义key的存放路径，这里用的是相对路径，相对于mountPath中定义的挂载路径
[root@master kube_manifest]#  kubectl apply -f pod-secret-demo-2.yaml 
pod/pod-secret-2 created
[root@master kube_manifest]# kubectl exec -it pod-secret-2 -- /bin/sh
/ # ls /etc/secret/my-secret/
my-username
/ # ls /etc/secret/my-secret/my-username 
/etc/secret/my-secret/my-username
/ # cat /etc/secret/my-secret/my-username 
root/ # 

以 Volume 方式使用的 Secret 支持动态更新，Secret 更新后，容器中的数据也会更新。

将 mysecret-4这个secret中的username的值改为admin(admin的base64编码为YWRtaW4K）

[root@master ~]# echo "admin" | base64 
YWRtaW4K
[root@master ~]# kubectl edit secret mysecret-4

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  password: MTIzNDU2
  username: YWRtaW4K  ##修改username的值为admin
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"password":"MTIzNDU2","username":"cm9vdA=="},"kind":"Secret","metadata":{"annotations":{},"name":"mysecret-4","namespace":"default"}}
  creationTimestamp: "2020-11-18T06:57:25Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:password: {}
        f:username: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-18T06:57:25Z"
  name: mysecret-4
  namespace: default
  resourceVersion: "2151693"
  selfLink: /api/v1/namespaces/default/secrets/mysecret-4
  uid: 94f94c89-2f96-4736-a4dd-5f4b0b7c1d4c
type: Opaque
[root@master ~]# kubectl edit secret mysecret-4
secret/mysecret-4 edited
[root@master kube_manifest]# kubectl exec -it pod-secret-2 -- /bin/sh
/ # ls /etc/secret/my-secret/my-username 
/etc/secret/my-secret/my-username
/ # cat /etc/secret/my-secret/my-username 
admin   ##pod中的调用的secret也修改为了admin
/ # 
       

还支持通过环境变量使用 Secret。

[root@master kube_manifest]# vim pod-secret-env-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret-env
spec:
  containers:
  - name: pod-secret-env
    image: busybox
    args:
      - /bin/sh
      - -c
      - sleep 10;touch /tmp/healthy;sleep 30000
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret-4
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret-4
            key: password
[root@master kube_manifest]#  kubectl apply -f pod-secret-env-demo.yaml
pod/pod-secret-env created
[root@master kube_manifest]# kubectl exec -it pod-secret-env -- /bin/sh
/ # echo $SECRET_USERNAME
admin
/ # echo $SECRET_PASSWORD
123456
/ # 

通过环境变量 SECRET_USERNAME 和 SECRET_PASSWORD 成功读取到 Secret 的数据。需要注意的是，环境变量读取 Secret 很方便，但无法支撑 Secret 动态更新。