​
kubernetes官方指定的ingress-nginx的官方地址：Welcome - NGINX Ingress Controller

官网说明的大致过程：

1.创建新的namespace。

2.确保每个ingress controller的--controller-class= 和 --ingress-class 是不同的。

# ingress-nginx Deployment/Statfulset 
spec:
  template:
    spec:
      containers:
      - name: ingress-nginx-internal-controller
        args:
        - /nginx-ingress-controller
        - --controller-class=k8s.io/internal-ingress-nginx
        - --ingress-class=internal-nginx
        ...

3. 定义相应值的ingressClass

# ingress-nginx IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: internal-nginx
spec:
  controller: k8s.io/internal-ingress-nginx
  ...

4.在ingress中指定IngressClass

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  ingressClassName: internal-nginx
  ...

注意：When running multiple ingress-nginx controllers, it will only process an unset class annotation if one of the controllers uses the default `--controller-class` value (see `IsValid` method in `internal/ingress/annotations/class/main.go`), otherwise the class annotation becomes required. If `--controller-class` is set to the default value of `k8s.io/ingress-nginx`, the controller will monitor Ingresses with no class annotation *and* Ingresses with annotation class set to `nginx`. Use a non-default value for `--controller-class`, to ensure that the controller only satisfied the specific class of Ingresses.

所以，当有多个controller时，为了保证ingress controller只满足特定的ingressClass，‘--controller-class’ 不要设成默认值(k8s.io/ingress-nginx)。

ingressClass的默认值为nginx。

部署实践步骤：

1.ingress Controller使用nodePort方式暴露端口

官方提供的ingress.yaml中将所有的“ingress-nginx”换成新的名称（包括namespace、service、label等等），并且每个ingress controller使用的nodePort要设成不一样的值。

我是使用的deployment的方式部署的，先给每组controller的node添加不同的label（如 给4台node节点设置label为kubernetes.io/ingressgroup=group59，在deployment的nodeSelector中设置kubernetes.io/ingressgroup: group59），并且设置relipcas的值为该组controller要使用的node的数量。

示例：

ingressgroup59.yaml:由于文章篇幅限制，只展示需要修改的地方。

apiVersion: v1

kind: Namespace

metadata:

name: ingress-nginx-group59

labels:

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

---

# Source: ingress-nginx/templates/controller-serviceaccount.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-group59

namespace: ingress-nginx-group59

automountServiceAccountToken: true

---

# Source: ingress-nginx/templates/controller-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-controller-group59

namespace: ingress-nginx-group59

data:

allow-snippet-annotations: 'true'

---

# Source: ingress-nginx/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

name: ingress-nginx-group59

rules:

...#省略

---

# Source: ingress-nginx/templates/clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

name: ingress-nginx-group59

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: ingress-nginx-group59

subjects:

- kind: ServiceAccount

name: ingress-nginx-group59

namespace: ingress-nginx-group59

---

# Source: ingress-nginx/templates/controller-role.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-group59

namespace: ingress-nginx-group59

rules:

...#省略

---

# Source: ingress-nginx/templates/controller-rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-group59

namespace: ingress-nginx-group59

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: ingress-nginx-group59

subjects:

- kind: ServiceAccount

name: ingress-nginx-group59

namespace: ingress-nginx-group59

---

# Source: ingress-nginx/templates/controller-service.yaml

apiVersion: v1

kind: Service

metadata:

annotations:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-controller-group59

namespace: ingress-nginx-group59

spec:

type: NodePort

ipFamilyPolicy: SingleStack

ipFamilies:

- IPv4

ports:

- name: http

port: 80

protocol: TCP

targetPort: http

appProtocol: http

nodePort: 30275

selector:

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/component: controller

---

# Source: ingress-nginx/templates/controller-deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

#app.kubernetes.io/ingress-conttroller: controller59

name: ingress-nginx-controller59

namespace: ingress-nginx-group59

spec:

replicas: 4 #group59组主机的数量

selector:

matchLabels:

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/component: controller

#app.kubernetes.io/ingress-conttroller: controller59

revisionHistoryLimit: 10

minReadySeconds: 0

template:

metadata:

labels:

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/component: controller

#app.kubernetes.io/ingress-conttroller: controller59

spec:

dnsPolicy: ClusterFirst

# hostNetwork: true

containers:

- name: controller59

image: ingress-nginx-controller:v1.0.5 #我把镜像改成了私有仓库地址了，请自行更改

imagePullPolicy: IfNotPresent

lifecycle:

preStop:

exec:

command:

- /wait-shutdown

args:

- /nginx-ingress-controller

- --election-id=ingress-controller-leader

- --controller-class=k8s.io/group59-ingress-nginx

- --ingress-class=group59-ingress-nginx

- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller-group59

- --validating-webhook=:8443

- --validating-webhook-certificate=/usr/local/certificates/cert

- --validating-webhook-key=/usr/local/certificates/key

securityContext:

...#省略
resources:

requests:

cpu: 100m

memory: 90Mi

nodeSelector:

kubernetes.io/ingressgroup: group59

serviceAccountName: ingress-nginx-group59

terminationGracePeriodSeconds: 300

volumes:

- name: webhook-cert

secret:

secretName: ingress-nginx-admission-group59

---

# Source: ingress-nginx/templates/controller-ingressclass.yaml

apiVersion: networking.k8s.io/v1

kind: IngressClass

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: group59-ingress-nginx

namespace: ingress-nginx-group59

spec:

controller: k8s.io/group59-ingress-nginx

---

# Source: ingress-nginx/templates/controller-service-webhook.yaml

apiVersion: v1

kind: Service

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: controller

name: ingress-nginx-controller-admission-group59

namespace: ingress-nginx-group59

spec:

type: ClusterIP

...#省略

selector:

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/component: controller

---

# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml

apiVersion: admissionregistration.k8s.io/v1

kind: ValidatingWebhookConfiguration

metadata:

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

name: ingress-nginx-admission-group59

webhooks:

...#省略
service:

namespace: ingress-nginx-group59

name: ingress-nginx-controller-admission-group59

path: /networking/v1/ingresses

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

name: ingress-nginx-admission-group59

namespace: ingress-nginx-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: ingress-nginx-admission-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

rules:

...#省略
---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: ingress-nginx-admission-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: ingress-nginx-admission-group59

subjects:

- kind: ServiceAccount

name: ingress-nginx-admission-group59

namespace: ingress-nginx-group59

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

name: ingress-nginx-admission-group59

namespace: ingress-nginx-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

rules:

...#省略

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: ingress-nginx-admission-group59

namespace: ingress-nginx-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: ingress-nginx-admission-group59

subjects:

- kind: ServiceAccount

name: ingress-nginx-admission-group59

namespace: ingress-nginx-group59

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml

apiVersion: batch/v1

kind: Job

metadata:

name: ingress-nginx-admission-create-group59

namespace: ingress-nginx-group59

annotations:

helm.sh/hook: pre-install,pre-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

spec:

template:

metadata:

name: ingress-nginx-admission-create-group59

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

spec:

containers:

- name: create

image: k8s/kube-webhook-certgen:v1.1.1 #请自行更改镜像地址

imagePullPolicy: IfNotPresent

args:

- create

- --host=ingress-nginx-controller-admission-group59,ingress-nginx-controller-admission-group59.$(POD_NAMESPACE).svc

- --namespace=$(POD_NAMESPACE)

- --secret-name=ingress-nginx-admission-group59

env:

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

restartPolicy: OnFailure

serviceAccountName: ingress-nginx-admission-group59

nodeSelector:

kubernetes.io/os: linux

securityContext:

runAsNonRoot: true

runAsUser: 2000

---

# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml

apiVersion: batch/v1

kind: Job

metadata:

name: ingress-nginx-admission-patch-group59

namespace: ingress-nginx-group59

annotations:

helm.sh/hook: post-install,post-upgrade

helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

spec:

template:

metadata:

name: ingress-nginx-admission-patch-group59

labels:

helm.sh/chart: ingress-nginx-4.0.7

app.kubernetes.io/name: ingress-nginx-group59

app.kubernetes.io/instance: ingress-nginx-group59

app.kubernetes.io/version: 1.0.5

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/component: admission-webhook

spec:

containers:

- name: patch

image: k8s/kube-webhook-certgen:v1.1.1 #请自行更改镜像地址

imagePullPolicy: IfNotPresent

args:

- patch

- --webhook-name=ingress-nginx-admission-group59

- --namespace=$(POD_NAMESPACE)

- --patch-mutating=false

- --secret-name=ingress-nginx-admission-group59

- --patch-failure-policy=Fail

env:

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

restartPolicy: OnFailure

serviceAccountName: ingress-nginx-admission-group59

nodeSelector:

kubernetes.io/os: linux

securityContext:

runAsNonRoot: true

runAsUser: 2000

ingressgroup63.yaml中，将ingressgroup59.yaml中所有的xxx-group59改成相应的xxx-group63。nodePort设置为30276，其余内容和ingressgroup59.yaml相同。

2.使用hostNetwork的方式部署

如果使用hostNetwork的方式部署的话，是直接使用主机的网络，把ingress controller的80和443端口映射成主机的80和443端口。

但是这种方式可能会存在node间无法通信和集群内域名解析的问题，所以我用nodePort的方式部署的。

这种方式也是利用上面的部署文件进行部署，只是不再需要定义service了，而且要把controller的deployment中的dnsPolicy: ClusterFirst改成hostNetwork: true。这里不详细举例了。

多个ingress controller部署好，并且配置了ingress之后，就可以通过IP+对应ingress controller开放的端口/ingress path访问应用了。

ingress示例：

testgroup59.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: iptest

name: iptest

namespace: default

spec:

replicas: 1

selector:

matchLabels:

app: iptest

template:

metadata:

labels:

app: iptest

spec:

containers:

- name: iptest

image: library/nginx:stable #自定义镜像地址

ports:

- containerPort: 80

nodeSelector:

kubernetes.io/os: linux

---
apiVersion: v1

kind: Service

metadata:

name: iptest

labels:

app: iptest

namespace: default

spec:

ports:

- port: 8190

targetPort: 80

#nodePort: 30009

#type: NodePort

selector:

app: iptest

---

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: nginx-web-test59

namespace: default

annotations:

nginx.ingress.kubernetes.io/rewrite-target: /

spec:

ingressClassName: group59-ingress-nginx

rules:

- http:

paths:

- path: /test

pathType: Prefix

backend:

service:

name: iptest

port:

number: 8190

testgroup63.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web-test63
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: group63-ingress-nginx
  rules:
  - http:
      paths:
      - path: /test
        pathType: Prefix
        backend:
          service:
            name: iptest
            port:
              number:  8190

配置同testgroup59.yaml，只是ingressClassName: group63-ingress-nginx不同。

通过http://任意node的ip:(controller group59开放的端口30275 或者controller group63开放发端口30276)/test/都可以访问到nginx的欢迎页面了。

​

# kubernetes
Logo
Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。 一、云计算历经十余年

avatar Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松 今天的内容会从以下几个方面展开： 负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点 一、产品简介​ 1. 以NGINX为例​ 提到负载均衡，我们以

avatar Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。 单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这

avatar Cloudpods