​ 由于公司K8s集群环境并不是很大的集群，所以日志收集使用的是Loki+Grafana的方案，使用helm的方式安装部署loki启动前修改配置文件已应用自定义配置来实现日志的收集，文件方式的日志收集等。

一、Loki的部署

  1. 部署helm；
https://www.yuque.com/kkkfree/ilecfh/tf6dai
  1. helm拉取loki-stack；
# 加源
~] helm repo add grafana https://grafana.github.io/helm-charts
~] helm repo update
# 拉取2.0.0版本的loki，因为现在已有的grafana版本是5.0与最新版本的loki不兼容，所以使用老版本
~] helm fetch grafana/loki-stack --version v2.0.0 --untar --untardir .
~] cd loki-stack
# 生成 k8s 配置
loki-stack ] helm template loki . > loki.yaml
  1. 自定义部署文件
loki-stack ] mkdir conf
loki-stack ] cp loki.yaml{,.bak}
loki-stack ] sed -i 's/namespace: logs/namespace: default/g' loki.yaml
loki-stack ] vim loki.yaml
...
apiVersion: v1
kind: Service
metadata:
  name: loki
  namespace: logs
  labels:
    app: loki
    chart: loki-2.0.0
    release: loki
    heritage: Helm
  annotations:
    {}
spec:
# 端口类型修改为NodePort绑定到节点上的3100，我们这边使用的Nginx代理，所以不改也可以
  type: NodePort
  ports:
    - port: 3100
      protocol: TCP
      name: http-metrics
      targetPort: http-metrics
      nodePort: 3100
  selector:
    app: loki
    release: loki

# 搜索"storage"将emptyDir修改为pvc的名称，这里做日志持久化
      volumes:
        - name: config
          secret:
            secretName: loki
        - name: storage
          persistentVolumeClaim:
            claimName: logs-pvc

# 搜索"hostPath"找到定义的volumes在volumes下添加下列内容,文件方式记录日志的日志目录
# volumeMounts也要添加      
      - mountPath: /gensee/nginx
        name: nginx

      - hostPath:
          path: /gensee/nginx
          type: ""
        name: nginx


# 搜索kubernetes-pods-static,在这个ConfigMap里添加下列自定义的日志文件标签处理方式，在kubernetes-pods-static这一组的下列新增job_name
    - job_name: kubernetes-pods-name-nginx
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_label_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /gensee/nginx/$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: __path__
    - job_name: kubernetes-pods-app-eschol-nginx
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        source_labels:
        - __meta_kubernetes_pod_label_name
      - source_labels:
        - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /gensee/nginx/$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: __path__
    - job_name: kubernetes-pods-direct-controllers-nginx
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        separator: ''
        source_labels:
        - __meta_kubernetes_pod_label_name
        - __meta_kubernetes_pod_label_app
      - action: drop
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
      - source_labels:
        - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /gensee/nginx/$1/*.log
        separator: /
        source_labels:
        - __meta_kuberenetes_pod_name
        target_label: __path__
    - job_name: kubernetes-pods-indirect-controller-nginx
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        separator: ''
        source_labels:
        - __meta_kubernetes_pod_label_name
        - __meta_kubernetes_pod_label_app
      - action: keep
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
      - action: replace
        regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /gensee/nginx/$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: __path__
    - job_name: kubernetes-pods-static-nginx
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: ''
        source_labels:
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_label_component
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /gensee/nginx/$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: __path__
...
loki-stack ] mkdir pvc && cd pvc
pvc ] vim log-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-pvc
  namespace: logs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
pvc ] kubectl apply -f log-pvc.yaml
  1. 自定义配置文件
# 获取loki的配置文件
~] kubectl get secret loki -o json -n logs | jq  -r '.data."loki.yaml"' | base64 -d > /root/loki-stack/conf/loki.yaml
~] vim /root/loki-stack/conf/loki.yaml
auth_enabled: false
chunk_store_config:
  max_look_back_period: 0s
compactor:
  shared_store: filesystem
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  # 旧日志是否拒绝接收
  reject_old_samples: true
  # 多久以前的数据不接收
  reject_old_samples_max_age: 72h
  # 当你搭建完成promtail，并且启动发送日志到loki的时候很有可能会碰到这个错误，
  # 因为你要收集的日志太多了，超过了loki的限制，所以会报429，如果你要增加限制可以修改loki的配置文件,在limits_config中添加
  ingestion_rate_mb: 100
  # 每个用户允许摄入的突发大小(样本大小)。 单位MB。  
  # 突发大小指的是每个分发器本地速率限制器，即使在  
  # case的“全局”策略，并且应该至少设置为最大日志  
  # 单个推请求中期望的大小  
  ingestion_burst_size_mb: 200
schema_config:
  configs:
  - from: "2020-10-24"
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v11
    store: boltdb-shipper
server:
  http_listen_port: 3100
  grpc_server_max_recv_msg_size: 10388608
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
  1. 部署loki-stack
# 部署(如果要修改默认配置必须要修改一下yaml)
~] kubectl apply -f loki.yaml
# 使用自定义的配置文件
~] kubectl delete secret loki -n logs
  1. 自定义的Pod日志输出路径
...          
          volumeMounts:
          - name: nginx-log
            mountPath: /gensee/openresty/nginx/logs/
            subPathExpr: $(POD_NAME)
      volumes:
      - name: nginx-log
        hostPath:
          path: /gensee/nginx
          type: DirectoryOrCreate
...
  1. 反向代理Loki以让外部Grafana可连接Loki
~] vim nginx/conf/conf.d/loki.conf
server {
        listen       80;
        server_name  loki.general.com; # 替换成你的域名
        access_log  logs/loki.log;
        location / {
                proxy_pass http://loki.logs.svc.cluster.local:3100;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
         }
}
~] kubectl delete cm nginx-confd -n nginx
~] kubectl create cm nginx-confd --from-file=nginx/conf/conf.d/loki.conf -n nginx

二、配置Grafana

  1. 增加DataSource；

img

  1. 测试通过后在下列位置查看日志，筛选标签即可查看日志

img

# kubernetes # nginx # docker # linux # 容器
Logo
Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。 一、云计算历经十余年

avatar Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松 今天的内容会从以下几个方面展开： 负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点 一、产品简介​ 1. 以NGINX为例​ 提到负载均衡，我们以

avatar Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。 单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这

avatar Cloudpods