helm入门

简介

Helm是一个由CNCF孵化和管理的项目,用于对需要在Kubernetes 上部署的复杂应用进行定义、安装和更新。Helm以Chart的方式对应用软件进行描述,可以方便地创建、版本化、共享和发布复杂的应用软件。

helm3架构

在这里插入图片描述

helm3安装

#github地址
#https://github.com/helm/helm
#本文示例使用的是v3.7.0版本
wget https://get.helm.sh/helm-v3.7.0-linux-amd64.tar.gz
#解压->helm放入PATH一个路径下

helm中三大概念

  • Chart:一个Helm包,其中包含运行一个应用所需要的工具和资源定义,还可能包含Kubernetes集群中的服务定义,类似于Homebrew 中的formula、APT中的dpkg或者Yum中的RPM文件。
  • Release:在Kubernetes集群上运行的一个Chart实例。在同一个 集群上,一个Chart可以被安装多次。例如有一个MySQL Chart,如果想在服务器上运行两个MySQL数据库,就可以基于这个Chart安装两次。 每次安装都会生成新的Release,会有独立的Release名称。
  • Repository:用于存放和共享Chart仓库。 简单来说,Helm整个系统的主要任务就是,在仓库中查找需要的 Chart,然后将Chart以Release的形式安装到Kubernetes集群中。

Helm Chart的使用

下面将使用一个例子展示helm chart的使用。

创建

$ helm create nginx

该命令会创建一个nginx文件目录,tree查看目录结构

$ tree
.
├── charts  #包含chart依赖的其他chart
├── Chart.yaml #包含了chart信息的YAML文件
├── templates #模板目录, 当和values 结合时,可生成有效的Kubernetes manifest文件
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests #测试
│       └── test-connection.yaml
└── values.yaml #chart 默认的配置值

Chart.yaml

$ cat Chart.yaml 
apiVersion: v2 #在heml3中apiVersion必须是v2
name: nginx #chart名字
description: A Helm chart for Kubernetes #chart描述
type: application #chart类型 application(默认)、library
version: 0.1.0 #chart的版本
appVersion: "1.16.0" #应用的版本

values.yaml

$ cat values.yaml 
# Default values for nginx.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: nginx
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  className: ""
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

templates目录下存放了应用编排文件。

  • (_)开头的文件用来存储局部和辅助对象,供其他chart模板使用。模板命令都是嵌入在{{}}之间的。

    cat _helpers.tpl 
    {{/*
    Expand the name of the chart.
    */}}
    {{- define "nginx.name" -}}
    {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Create a default fully qualified app name.
    We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
    If release name contains chart name it will be used as a full name.
    */}}
    {{- define "nginx.fullname" -}}
    {{- if .Values.fullnameOverride }}
    {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- $name := default .Chart.Name .Values.nameOverride }}
    {{- if contains $name .Release.Name }}
    {{- .Release.Name | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
    {{- end }}
    {{- end }}
    {{- end }}
    
    {{/*
    Create chart name and version as used by the chart label.
    */}}
    {{- define "nginx.chart" -}}
    {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Common labels
    */}}
    {{- define "nginx.labels" -}}
    helm.sh/chart: {{ include "nginx.chart" . }}
    {{ include "nginx.selectorLabels" . }}
    {{- if .Chart.AppVersion }}
    app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
    {{- end }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    {{- end }}
    
    {{/*
    Selector labels
    */}}
    {{- define "nginx.selectorLabels" -}}
    app.kubernetes.io/name: {{ include "nginx.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    {{- end }}
    
    {{/*
    Create the name of the service account to use
    */}}
    {{- define "nginx.serviceAccountName" -}}
    {{- if .Values.serviceAccount.create }}
    {{- default (include "nginx.fullname" .) .Values.serviceAccount.name }}
    {{- else }}
    {{- default "default" .Values.serviceAccount.name }}
    {{- end }}
    {{- end }}
    
  • yaml格式的编排文件中将某些字段设置为“模板命令”,这些“模板命令”会在Helm部署应用时进行参数注入和模板的动态渲染。

    $ cat deployment.yaml 
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "nginx.fullname" . }}
      labels:
        {{- include "nginx.labels" . | nindent 4 }}
    spec:
      {{- if not .Values.autoscaling.enabled }}
      replicas: {{ .Values.replicaCount }}
      {{- end }}
      selector:
        matchLabels:
          {{- include "nginx.selectorLabels" . | nindent 6 }}
      template:
        metadata:
          {{- with .Values.podAnnotations }}
          annotations:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          labels:
            {{- include "nginx.selectorLabels" . | nindent 8 }}
        spec:
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          serviceAccountName: {{ include "nginx.serviceAccountName" . }}
          securityContext:
            {{- toYaml .Values.podSecurityContext | nindent 8 }}
          containers:
            - name: {{ .Chart.Name }}
              securityContext:
                {{- toYaml .Values.securityContext | nindent 12 }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              ports:
                - name: http
                  containerPort: 80
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /
                  port: http
              readinessProbe:
                httpGet:
                  path: /
                  port: http
              resources:
                {{- toYaml .Values.resources | nindent 12 }}
          {{- with .Values.nodeSelector }}
          nodeSelector:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          {{- with .Values.affinity }}
          affinity:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          {{- with .Values.tolerations }}
          tolerations:
            {{- toYaml . | nindent 8 }}
          {{- end }}
    

    可是使用helm template [NAME] [CHART] [flags]命令在本地渲染Helm Chart并打印最终的应用模板内容,values.yaml和(_)开头的文件中的值会被注入到yaml中,参数--set则会覆盖value.yaml文件中的值。

    $ helm template nginx  ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort
    ...
    ...
    ---
    # Source: nginx/templates/deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx
      labels:
        helm.sh/chart: nginx-0.1.0
        app.kubernetes.io/name: nginx
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/version: "1.16.0"
        app.kubernetes.io/managed-by: Helm
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: nginx
          app.kubernetes.io/instance: nginx
      template:
        metadata:
          labels:
            app.kubernetes.io/name: nginx
            app.kubernetes.io/instance: nginx
        spec:
          serviceAccountName: nginx
          securityContext:
            {}
          containers:
            - name: nginx
              securityContext:
                {}
              image: "docker.io/library/nginx:1.16.0"
              imagePullPolicy: IfNotPresent
              ports:
                - name: http
                  containerPort: 80
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /
                  port: http
              readinessProbe:
                httpGet:
                  path: /
                  port: http
              resources:
                {}
    ---
    ...
    ...
    
  • NOTES.txt在helm installhelm upgrade命令的最后,Helm会打印出对用户有用的信息,该文件是纯文本,但会像模板一样处理, 所有正常的模板函数和对象都是可用的。

    $ cat NOTES.txt 
    1. Get the application URL by running these commands:
    {{- if .Values.ingress.enabled }}
    {{- range $host := .Values.ingress.hosts }}
      {{- range .paths }}
      http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
      {{- end }}
    {{- end }}
    {{- else if contains "NodePort" .Values.service.type }}
      export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "nginx.fullname" . }})
      export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
      echo http://$NODE_IP:$NODE_PORT
    {{- else if contains "LoadBalancer" .Values.service.type }}
         NOTE: It may take a few minutes for the LoadBalancer IP to be available.
               You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "nginx.fullname" . }}'
      export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "nginx.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
      echo http://$SERVICE_IP:{{ .Values.service.port }}
    {{- else if contains "ClusterIP" .Values.service.type }}
      export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "nginx.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
      export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
    {{- end }}
    

部署

helm install [NAME] [CHART] [flags]命令安装部署,同样values.yaml和(_)开头的文件中的值会被注入到yaml中,参数--set则会覆盖value.yaml文件中的值。

$ helm -n default install nginx ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort
NAME: nginx
LAST DEPLOYED: Sat Oct  9 16:26:50 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
  • 查看部署的release

    $ helm -n default ls
    NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
    nginx	default  	1       	2021-10-09 16:26:50.820014356 +0800 CST	deployed	nginx-0.1.0	1.16.0 
    
  • release版本信息存储

    默认存储在release相应namespace下的secret(可配置configmap、sql),每一个REVISION都会生成一个secret。

    $ kubectl -n default get secret -l "owner=helm"
    NAME                          TYPE                 DATA   AGE
    sh.helm.release.v1.nginx.v1   helm.sh/release.v1   1      1m
    

    查看存储内容

    kubectl -n default get secret sh.helm.release.v1.nginx.v1 -ojsonpath={.data.release} |base64 -d |base64 -d |gzip -d
    
  • 测试release是否符合预期(需等待所有部署的资源对象创建成功),原理:通过在集群中部署templates/tests/下的yaml,根据容器退出状态判断(exit 0:通过)是否测试通过。测试的yaml必须含有注解:helm.sh/hook: test。

    $ helm test nginx
    NAME: nginx
    LAST DEPLOYED: Sat Oct  9 16:26:50 2021
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE:     nginx-test-connection
    Last Started:   Sat Oct  9 16:47:02 2021
    Last Completed: Sat Oct  9 16:47:20 2021
    Phase:          Succeeded
    NOTES:
    1. Get the application URL by running these commands:
      export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
      export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
      echo http://$NODE_IP:$NODE_PORT
    

升级、回滚、卸载

  • helm upgrade [RELEASE] [CHART] [flags]命令进行升级。

    升级之前查看nginx release当前版本:REVISION=1

    helm -n default ls
    NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
    nginx	default  	1       	2021-10-09 16:26:50.820014356 +0800 CST	deployed	nginx-0.1.0	1.16.0     
    

    使用升级命令,将deployment中的镜像tag升级到1.17.0,可使用--dry-run试运行测试是否可以升级成功,并打印升级清单。

    $ helm -n default upgrade nginx ./nginx/ --set image.repository=docker.io/library/nginx,service.type=NodePort,image.tag=1.17.0
    Release "nginx" has been upgraded. Happy Helming!
    NAME: nginx
    LAST DEPLOYED: Sun Oct 10 13:41:17 2021
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    NOTES:
    1. Get the application URL by running these commands:
      export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
      export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
      echo http://$NODE_IP:$NODE_PORT
    

    再次查看nginx release当前版本:REVISION=2

    $ helm -n default ls
    NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
    nginx	default  	2       	2021-10-10 13:41:17.486033411 +0800 CST	deployed	nginx-0.1.0	1.16.0     
    
  • helm rollback <RELEASE> [REVISION] [flags]命令进行回滚。

    回滚之前可使用helm history RELEASE_NAME [flags]查看nginx release历史版本。

    $ helm -n default history nginx
    REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
    1       	Sat Oct  9 16:26:50 2021	superseded	nginx-0.1.0	1.16.0     	Install complete
    2       	Sun Oct 10 13:41:17 2021	deployed  	nginx-0.1.0	1.16.0     	Upgrade complete
    

    将nginx release回滚到REVISION=1,回滚完成后可查看deployment image tag:1.16.0。

    $ helm -n default rollback nginx 1
    Rollback was a success! Happy Helming!
    
  • helm uninstall RELEASE_NAME [...] [flags]命令卸载release,卸载完成后release、release secret、k8s manifest都被删除。

    $ helm -n default uninstall nginx
    release "nginx" uninstalled
    

仓库使用

Harbor
  • 使用helm安装harbor

    $ helm repo add harbor https://helm.goharbor.io
    "harbor" has been added to your repositories
    #查看本地repo列表
    $ helm repo ls
    NAME  	URL                     
    harbor	https://helm.goharbor.io
    #查看harbor仓库helm chart版本
    $ helm search repo harbor
    NAME         	CHART VERSION	APP VERSION	DESCRIPTION                                       
    harbor/harbor	1.7.3        	2.3.3      	An open source trusted cloud native registry th...
    #部署harbor release
    #persistence.enabled=false 关闭持久化存储
    #expose.type=nodePort service类型设置为nodeport方便暴露服务
    #expose.tls.enabled=false 关闭tls
    #--create-namespace 没有harbor命名空间,则会创建
    #externalURL中的ip为本机网卡ip
    $ helm -n harbor install harbor harbor/harbor --set persistence.enabled=false,expose.type=nodePort,expose.tls.enabled=false,externalURL=http://10.23.18.211:30002 --create-namespace
    NAME: harbor
    LAST DEPLOYED: Mon Oct 11 18:38:47 2021
    NAMESPACE: harbor
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    Please wait for several minutes for Harbor deployment to complete.
    Then you should be able to visit the Harbor portal at http://10.23.18.211:30002
    For more details, please visit https://github.com/goharbor/harbor
    #等待所有pod running
    $ kubectl -n harbor get pod
    NAME                                    READY   STATUS    RESTARTS   AGE
    harbor-chartmuseum-6c9f9c84f8-ztpgp     1/1     Running   0          3m20s
    harbor-core-c6dc8c895-vn5kx             1/1     Running   0          3m19s
    harbor-database-0                       1/1     Running   0          3m20s
    harbor-jobservice-6c4d647bcb-nd64t      1/1     Running   0          3m20s
    harbor-nginx-7bc597c58b-67x6x           1/1     Running   0          3m19s
    harbor-notary-server-5f598b9555-4qjpm   1/1     Running   0          3m20s
    harbor-notary-signer-65b84d688d-g92tv   1/1     Running   0          3m19s
    harbor-portal-945d945f-ks6tx            1/1     Running   0          3m20s
    harbor-redis-0                          1/1     Running   0          3m20s
    harbor-registry-6d976587dc-dbrnt        2/2     Running   0          3m20s
    harbor-trivy-0                          1/1     Running   0          3m20s
    

    浏览器访问http://10.23.18.211:30002即可访问harbor页面,admin/Harbor12345为默认用户密码。

  • 推送nginx chart到仓库

    登录harbor页面,项目–>新建项目,创建一个名为helm的helm chart仓库。

    添加helm仓库。

    $ helm repo add my-harbor http://10.23.18.211:30002/chartrepo/helm --username=admin --password=Harbor12345
    "my-harbor" has been added to your repositories
    #查看当前repo
    $ helm repo ls
    NAME     	URL                                     
    harbor   	https://helm.goharbor.io                
    my-harbor	http://10.23.18.211:30002/chartrepo/helm
    

    由于原生helm没有推送命令,所以需要安装一个推送插件。

    $ helm plugin install https://github.com/chartmuseum/helm-push.git
    Downloading and installing helm-push v0.10.0 ...
    https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
    Installed plugin: cm-push
    #查看已安装的插件
    $ helm plugin ls
    NAME   	VERSION	DESCRIPTION                      
    cm-push	0.10.0 	Push chart package to ChartMuseum
    

    (备选方案)如果服务器不能访问外网。可以从能够访问外网的机器clone插件代码和插件压缩包,然后放到此服务器上。

    #在能够访问外网的机器上
    $ git clone https://github.com/chartmuseum/helm-push.git
    $curl https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
    #将插件代码上传到服务器/root/.cache/helm/plugins目录下,创建release目录
    $ mkdiir /root/.cache/helm/plugins/helm-push/release
    #将插件压缩包上传到服务器/root/.cache/helm/plugins/helm-push/releases下
    $ cd /root/.cache/helm/plugins
    #vim 编辑helm-push/scripts/install_plugin.sh,根据需要注释掉 `# Download with curl if possible.`下边代码,以注释掉curl为例:
    $ vim helm-push/scripts/install_plugin.sh
    $ cat helm-push/scripts/install_plugin.sh
    ...
    ...
    # Download with curl if possible.
    if [ -x "$(which curl 2>/dev/null)" ]; then
      #  curl -sSL "${url}" -o "releases/v${version}.tar.gz"
         echo "jump curl"
    else
        wget -q "${url}" -O "releases/v${version}.tar.gz"
    fi
    ...
    ...
    #安装推送插件
    $ helm plugin install ./helm-push/
    Downloading and installing helm-push v0.10.0 ...
    https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
    jump curl
    Installed plugin: cm-push
    [root@master01 plugins]# helm plugin ls
    NAME   	VERSION	DESCRIPTION                      
    cm-push	0.10.0 	Push chart package to ChartMuseum
    

    推送nginx chart。

    #cd到nginx目录,推送
    $ helm cm-push ./nginx my-harbor
    Pushing nginx-0.1.0.tgz to my-harbor...
    Done.
    #更新本地缓存的my-harbor仓库信息,可以确认nginx chart上传到harbor上,也可以在harbor页面上确认
    $ helm repo  update my-harbor 
    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "my-harbor" chart repository
    Update Complete. ⎈Happy Helming!⎈
    $ helm search repo my-harbor
    NAME           	CHART VERSION	APP VERSION	DESCRIPTION                
    my-harbor/nginx	0.1.0        	1.16.0     	A Helm chart for Kubernetes
    
  • 拉取、部署仓库中的chart

    helm pull [chart URL | repo/chartname] [...] [flags]拉取仓库中chart,pull只会拉取到本地当前目录下的.tgz压缩包,可使用tar xzvf xxx.tgz解压。

    #更新本地helm repo 缓存
    $ helm repo update
    #pull拉取nginx chart
    $ helm pull my-harbor/nginx
    

    helm install [NAME] [CHART] [flags]部署仓库中的chart,和上边部署篇幅类似,把使用本地文件改成使用仓库中的chart即可。

    $ helm -n test install nginx my-harbor/nginx --set image.repository=docker.io/library/nginx,service.type=NodePort --create-namespace
    NAME: nginx
    LAST DEPLOYED: Mon Oct 11 19:47:36 2021
    NAMESPACE: test
    STATUS: deployed
    REVISION: 1
    NOTES:
    1. Get the application URL by running these commands:
      export NODE_PORT=$(kubectl get --namespace test -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
      export NODE_IP=$(kubectl get nodes --namespace test -o jsonpath="{.items[0].status.addresses[0].address}")
      echo http://$NODE_IP:$NODE_PORT
    
Logo

开源、云原生的融合云平台

更多推荐