k8s ingress 配置问题

Ingress-Nginx Deploymentcontainers:- name: nginx-ingress-controllerimage: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0args:- /nginx-ingress-controller- --configmap=$...

一杯咖啡的时间

3141人浏览 · 2021-08-31 17:22:18

一杯咖啡的时间 · 2021-08-31 17:22:18 发布

Ingress-Nginx Deployment

通过configmap nginx-configuration定义一些全局常规参数

Ingress-Nginx Deployment

      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io

--ingress-class：声明ingress入口名称，如果ingress资源要绑该ingress controller，需要在 annotation 中定义 kubernetes.io/ingress.class: "nginx"

开启TLS/ HTTPS

创建ssl证书 secret

kubectl -n default create secret tls tls-https --key ./tls.key --cert ./tls.crt
secret/tls-https created

默认情况下，如果ingress对象入口启用了TLS，则ingress-controller将使用308永久重定向响应将HTTP客户端重定向到HTTPS端口443

[root@test-k8s-wuwjg static]# curl -I   http://test.haha.com/
 HTTP/1.1 308 Permanent Redirect

可以在特定ingress资源的metadata.annotations中通过配置nginx.ingress.kubernetes.io/ssl-redirect: "false" 使用注释禁用此功能

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx" # 绑定ingress-class
    nginx.ingress.kubernetes.io/ssl-redirect: "false" #关闭SSL跳转
spec:
  tls:
  - hosts:
    - test.haha.com
    secretName: tls-https 
  rules:
    - host: test.haha.com
      http:
        paths:
          - path: /
            backend:
              serviceName: test-front
              servicePort: 80

四层反向代理配置

Ingress Controller启动时会去watch两个configmap(一个tcp，一个 udp),即开头deployment模板中args字段配置 --tcp-services-configmap --udp-services-configmap所定义,以tcp代理相关configmap如下


kubectl edit cm  tcp-services -o yaml -n ingress-nginx 
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: tcp-services
  namespace: ingress-nginx
data:     #通过data字段添加四层反向代理的service
  "6666": default/nginx-06:1992  #key为代理端口
 
wq!
 
[root@test-k8s-01]#telnet 192.168.1.18 6666
Trying 192.168.2.18...
Connected to 192.168.2.18.
Escape character is '^]'.

客户端地址记录

可以在特定ingress资源下通过metadata.annotations字段下通过nginx.ingress.kubernetes.io/configuration-snippet 参数来定义

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

ingress匹配优先级

首先在nginx中我们知道location的匹配优先级大致为精准匹配 > 前缀匹配 > 正则匹配> /

其中,前缀匹配:^~，精准匹配 =，正则匹配细分为：

~ 区分大小写（大小写敏感）匹配成功；~* 不区分大小写匹配成功；!~ 区分大小写匹配失败；!~* 不区分大小写匹配失败

而ingress资源对象中，spec.rules.http.paths.path字段默认只支持不区分大小写的正则匹配,但前提需要设置nginx.ingress.kubernetes.io/use-regex注释设置为true(默认值为false)来启用此功能

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: test.haha.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nginx-front
              servicePort: 80
          - path: /wifi
            backend:
              serviceName: nginx-wifi
              servicePort: 80

进入到ingress-controller的pod中，观察nginx配置文件，发现location的正则匹配已生效

[root@test-k8s-wuwjg static]# kubectl exec -it nginx-ingress-controller-79886bd49b-zf5h5 -n ingress-nginx /bin/sh -c "ca nginx.conf"
   ...
   ...
        ## start server test.haha.com                                                                                                                                                                
        server {                                                                                                                                                                                          
                server_name test.haha.com ;                                                                                                                                                          
                                                                                                                                                                                                          
                listen 80  ;                                                                                                                                                                              
                listen [::]:80  ;                                                                                                                                                                        
                                                                                                                                                                                                          
                set $proxy_upstream_name "-";                                                                                                                                                             
                                                                                                                                                                                                          
                ssl_certificate_by_lua_block {                                                                                                                                                            
                        certificate.call()                                                                                                                                                                
                }                                                                                                                                                                                         
                                                                                                                                                                                                          
                location ~* "^/wifi" {                                                                                                                                                                    
                                                                                                                                                                                                          
                        set $namespace      "default";                                                                                                                                                    
                        set $ingress_name   "ingress-dt";                                                                                                                                                 
                        set $service_name   "nginx-wifi";                                                                                                                                            
                        set $service_port   "80";                                                                                                                                                         
                        set $location_path  "/wifi";

跨域配置

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-headers："DNT,X-CustomHeader,Keep-Alive,User- 
Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"

白名单及请求速率限制

设置 test.haha.com/login 登陆页为每秒100个连接数，192.168.1.0/24,192.168.2.8 IP段不在限速范围

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.1.0/24,192.168.2.8
    nginx.ingress.kubernetes.io/limit-rps: '100'
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - test.haha.com
    secretName: tls-https
  rules:
    - host: test.haha.com
      http:
        paths:
          - path: /login
            backend:
              serviceName: nginx-front
              servicePort: 80

支持websocket配置

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_set_header Upgrade "websocket";
    proxy_set_header Connection "Upgrade";
    nginx.ingress.kubernetes.io/proxy-read-timeout 3600; 
    nginx.ingress.kubernetes.io/proxy-send-timeout 3600;

七层负载均衡算法

默认为round-robin，在具体ingress资源中通过ingress metadata.annotations字段可具体设置

通过会话cookie进行一致性hash均衡算法

ingress.kubernetes.io/affinity: "cookie"
ingress.kubernetes.io/session-cookie-name: "route"
ingress.kubernetes.io/session-cookie-hash: "sha1"

通过客户端ip进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${remote_addr}"

通过请求uri进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${request_uri}"

通过configmap nginx-configuration定义一些全局常规参数

[root@test-k8s-01]# kubectl get cm nginx-configuration -n ingress-nginx -o yaml |grep data |grep -v "metadata" -A 500
data:
  multi_accept: on;
  use: epoll;
  user: www;
  worker_connections: 65535;
  worker_cpu_affinity: auto;
  worker_processes: auto;
  worker_rlimit_nofile: 300000;
  
  # 把真实IP地址传给后端
  compute-full-forwarded-for: "true"
  forwarded-for-header: "X-Forwarded-For"
  use-forwarded-headers: "true"
  # 关闭版本显示
  server-tokens: "false"
  # 客户端请求头的缓冲区大小 
  client-header-buffer-size: "512k"
  # 设置用于读取大型客户端请求标头的最大值number和size缓冲区
  large-client-header-buffers: "16 512k"
  # 读取客户端请求body的缓冲区大小
  client-body-buffer-size: "968k"
  # 代理缓冲区大小
  proxy-buffer-size: "1024k"
  # 代理body大小
  proxy-body-size: "50m"
  # 服务器名称哈希大小
  server-name-hash-bucket-size: "128"
  # map哈希大小
  map-hash-bucket-size: "128"
  # SSL加密套件
  ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
  # ssl 协议
  ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"
#定义json 访问日志格式
log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent"}'

Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。一、云计算历经十余年

Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松今天的内容会从以下几个方面展开：负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点一、产品简介 1. 以NGINX为例提到负载均衡，我们以

Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这