一、Kubernetes e2e 测试简介

Kubernetes e2e(end-to-end) 测试是单元测试和集成测试的补充，其主要目标是确保 Kubernetes 行为的一致可靠，并在单元和集成测试不足时，在用户真正使用软件之前捕获难以测试的 Bug。Kubernetes e2e 测试基于 Ginkgo 和 Gomega 实现。

二、在本地运行 Kubernetes e2e 测试

要在本地运行 e2e 测试，方法有很多，官方建议基于 kubetest 启动，但相对来说环境要求较高。本文介绍一种比较简单且通用的方式：先编译出用于执行 e2e 测试的二进制可执行程序，然后放到目标集群内执行。

2.1 编译 e2e 测试可执行程序

a. 准备一个 Linux 操作系统（例如 CentOS Linux 7）

b. 安装 Go 语言

注意: 版本要与 K8s 代码库要求保持一致（可在 https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh 查找 minimum_go_version 以确认当前代码库需要的最低 Golang 版本，例如 K8s 1.19 需要 Golang 1.14.4 版本及以上）。

c. 克隆代码到本地

# mkdir $GOPATH/src/k8s.io
# cd $GOPATH/src/k8s.io
# git clone https://github.com/kubernetes/kubernetes.git

d. 编译 e2e.test 二进制

# cd $GOPATH/src/k8s.io/kubernetes
# make all WHAT=test/e2e/e2e.test
+++ [0708 09:42:01] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
+++ [0708 09:42:06] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0708 09:42:14] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0708 09:42:24] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0708 09:42:38] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
+++ [0708 09:42:52] Building go targets for linux/amd64:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0708 09:42:53] Building go targets for linux/amd64:
    test/e2e/e2e.test

编译好的可执行程序位于：$GOPATH/src/k8s.io/kubernetes/_output/bin/e2e.test

e. 编译对应版本的 ginkgo (可选)

e2e.test 自身已经可以用于执行测试，但为了更好地利用 ginkgo 的高级能力（例如指定执行哪些用例，跳过某些用例等），建议使用 ginkgo 来启动 e2e.test

# make all WHAT=vendor/github.com/onsi/ginkgo/ginkgo
+++ [0709 11:18:10] Building go targets for linux/amd64:
    vendor/github.com/onsi/ginkgo/ginkgo

编译好的可执行程序位于：$GOPATH/src/k8s.io/kubernetes/_output/bin/ginkgo

2.2 执行 e2e 测试

准备一个 Kubernetes 集群，作为目标测试集群，然后将 e2e.test 和 ginkgo 拷贝到能够通过 kubectl 访问集群的节点上，执行测试命令（这里以执行 NetworkPolicy 测试为例）：

# ./ginkgo --focus="NetworkPolicy" ./e2e.test -- --disable-log-dump --provider="skeleton" --kubeconfig="/root/.kube/config" > >(tee e2e.log)
I0709 12:48:04.233600   20566 test_context.go:427] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
I0709 12:48:04.233768   20566 e2e.go:129] Starting e2e run "b404a0ac-6071-481a-bf1c-4f721393a76a" on Ginkgo node 1
{"msg":"Test Suite starting","total":29,"completed":0,"skipped":0,"failed":0}
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1594270083 - Will randomize all specs
Will run 29 of 5208 specs

Jul  9 12:48:04.253: INFO: >>> kubeConfig: /root/.kube/config
Jul  9 12:48:04.256: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
Jul  9 12:48:04.268: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Jul  9 12:48:04.295: INFO: 10 / 10 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Jul  9 12:48:04.295: INFO: expected 2 pod replicas in namespace 'kube-system', 2 are Running and Ready.
Jul  9 12:48:04.295: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Jul  9 12:48:04.308: INFO: 2 / 2 pods ready in namespace 'kube-system' in daemonset 'fabric-node' (0 seconds elapsed)
Jul  9 12:48:04.308: INFO: 2 / 2 pods ready in namespace 'kube-system' in daemonset 'kube-proxy' (0 seconds elapsed)
Jul  9 12:48:04.308: INFO: e2e test version: v0.0.0-master+$Format:%h$
Jul  9 12:48:04.311: INFO: kube-apiserver version: v1.18.2
Jul  9 12:48:04.311: INFO: >>> kubeConfig: /root/.kube/config
Jul  9 12:48:04.321: INFO: Cluster IP family: ipv4
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client 
  should support a 'default-deny-ingress' policy [Feature:NetworkPolicy]
  /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:83
[BeforeEach] [sig-network] NetworkPolicy [LinuxOnly]
  /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:174
STEP: Creating a kubernetes client
Jul  9 12:48:04.322: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename network-policy
Jul  9 12:48:04.346: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-network] NetworkPolicy [LinuxOnly]
  /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:55
[BeforeEach] NetworkPolicy between server and client
  /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:61
STEP: Creating a simple server that serves on port 80 and 81.
STEP: Creating a server pod server in namespace network-policy-4845
Jul  9 12:48:04.353: INFO: Created pod server-6hfz8
STEP: Creating a service svc-server for pod server in namespace network-policy-4845
Jul  9 12:48:04.372: INFO: Created service svc-server
STEP: Waiting for pod ready
Jul  9 12:48:04.396: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:06.398: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:08.398: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:10.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
Jul  9 12:48:12.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
Jul  9 12:48:14.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
......

常用命令

1. 查看 e2e 测试将会使用的镜像

以便基于私有仓库提前准备好（或预分发到集群节点）

# ./e2e.test -list-images
gcr.io/k8s-staging-csi/nfs-provisioner:v2.2.2
gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.17
docker.io/library/busybox:1.29
k8s.gcr.io/etcd:3.4.7
gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0
gcr.io/kubernetes-e2e-test-images/regression-issue-74839-amd64:1.0
gcr.io/kubernetes-e2e-test-images/cuda-vector-add:1.0
gcr.io/kubernetes-e2e-test-images/echoserver:2.2
gcr.io/kubernetes-e2e-test-images/ipc-utils:1.0
gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0
docker.io/library/nginx:1.14-alpine
k8s.gcr.io/pause:3.2
docker.io/library/redis:5.0.5-alpine
gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0
gcr.io/authenticated-image-pulling/windows-nanoserver:v1
us.gcr.io/k8s-artifacts-prod/build-image/debian-iptables:v12.1.0
gcr.io/kubernetes-e2e-test-images/volume/iscsi:2.0
gcr.io/kubernetes-e2e-test-images/volume/rbd:1.0.1
docker.io/library/httpd:2.4.38-alpine
k8s.gcr.io/sd-dummy-exporter:v0.2.0
gcr.io/kubernetes-e2e-test-images/volume/gluster:1.0
gcr.io/kubernetes-e2e-test-images/apparmor-loader:1.0
gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0
docker.io/library/httpd:2.4.39-alpine
k8s.gcr.io/prometheus-dummy-exporter:v0.1.0
gcr.io/k8s-authenticated-test/agnhost:2.6
docker.io/gluster/glusterdynamic-provisioner:v1.0
gcr.io/kubernetes-e2e-test-images/nonroot:1.0
k8s.gcr.io/prometheus-to-sd:v0.5.0
invalid.com/invalid/alpine:3.1
gcr.io/kubernetes-e2e-test-images/kitten:1.0
gcr.io/kubernetes-e2e-test-images/nautilus:1.0
docker.io/library/nginx:1.15-alpine
gcr.io/kubernetes-e2e-test-images/resource-consumer:1.5
us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.20
gcr.io/authenticated-image-pulling/alpine:3.7
gcr.io/kubernetes-e2e-test-images/metadata-concealment:1.2
docker.io/library/perl:5.26

2. 查看一致性测试用例列表

# ./e2e.test -list-conformance-tests
- testname: Pod Lifecycle, post start exec hook
  codename: '[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook
    should execute poststart exec hook properly [NodeConformance] [Conformance]'
  description: When a post start handler is specified in the container lifecycle using
    a 'Exec' action, then the handler MUST be invoked after the start of the container.
    A server pod is created that will serve http requests, create a second pod with
    a container lifecycle specifying a post start that invokes the server pod using
    ExecAction to validate that the post start is executed.
  release: v1.9
  file: test/e2e/common/lifecycle_hook.go
.....