Kubernetes部署Prometheus operator
部署Prometheus
·
部署Prometheus
我们使用kube-prometheus 在K8S中部署Prometheus ,我们直接使用开源的 mainfest 文件即可。我们创建单独的 namespace 进行监控
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
官网如下:
## 参考1
https://github.com/prometheus-operator/kube-prometheus
###
https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests/setup
## 参考2
https://github.com/camilb/prometheus-kubernetes
### 告警配置
https://www.qikqiak.com/post/prometheus-operator-custom-alert/
安装部署如下:
###先查看k8s 是哪个版本,切到那个版本下
git checkout -b 本地分支 origi/远程分支
###
# Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/
组件分析
(1)kube-state-metrics与metrics-server对比?
我们服务在运行过程中,我们想了解服务运行状态,pod有没有重启,伸缩有没有成功,pod的状态是怎么样的等,这时就需要kube-state-metrics,它主要关注deployment,、node 、 pod等内部对象的状态。而metrics-server 主要用于监测node,pod等的CPU,内存,网络等系统指标。
最后查看是否都起来了:
$ kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 38m
alertmanager-main-1 2/2 Running 0 38m
alertmanager-main-2 2/2 Running 0 38m
blackbox-exporter-776596fdf8-82qj7 3/3 Running 0 39m
grafana-667874d57-xvvpt 1/1 Running 0 39m
kube-state-metrics-584858f6fc-24jlx 3/3 Running 0 12m
node-exporter-hn88p 2/2 Running 0 39m
node-exporter-jt7b8 2/2 Running 0 39m
prometheus-adapter-544596c9f5-gsbzp 1/1 Running 0 42s
prometheus-adapter-544596c9f5-rsb7d 1/1 Running 0 42s
prometheus-k8s-0 2/2 Running 0 38m
prometheus-k8s-1 2/2 Running 0 38m
prometheus-operator-7ddc6877d5-d58rd 2/2 Running 0 39m
开启对外访问
(1)修改proms的svc
# vi prometheus-service.yaml
##
[root@k8s-01 manifests]# cat prometheus-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.36.1
name: prometheus-k8s
namespace: monitoring
spec:
type: NodePort
ports:
- name: web
port: 9090
targetPort: web
nodePort: 30100 # 外部访问
# - name: reloader-web
# port: 8080
# targetPort: reloader-web
selector:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
sessionAffinity: ClientIP
[root@k8s-01 manifests]#
(2)修改grafana的svc
[root@k8s-01 manifests]# cat grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: grafana
app.kubernetes.io/name: grafana
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 8.5.5
name: grafana
namespace: monitoring
spec:
type: NodePort
ports:
- name: http
port: 3000
targetPort: http
nodePort: 30200
selector:
app.kubernetes.io/component: grafana
app.kubernetes.io/name: grafana
app.kubernetes.io/part-of: kube-prometheus
(3)访问:
##
http://xx.cn:30200
#
http://xx.cn:30100
### grafana的默认账号和密码为
admin/admin
proms查询
# 查询指定命名空间信息
container_cpu_usage_seconds_total{namespace="car-stg"}
告警规则编写
规则文章可参考如下:
## 参考1
https://awesome-prometheus-alerts.grep.to/rules.html
## 参考2
https://github.com/camilb/prometheus-kubernetes/blob/master/manifests/prometheus/prometheus-k8s-rules.yaml
如何修改alert rule?
#### 方式1: 通过rule规则修改
## edit
kubectl edit cm prometheus-k8s-rulefiles-0 -n monitoring
#### 方式2: 修改配置文件方式
cd /opt/proms-k8s/kube-prometheus/manifests
vim kubePrometheus-prometheusRule.yaml
###
kubectl apply kubePrometheus-prometheusRule.yaml
(1)Kubernetes Node ready
- alert: KubernetesNodeReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 10m
labels:
severity: critical
annotations:
summary: Kubernetes Node ready (instance {{ $labels.instance }})
description: "Node {{ $labels.node }} has been unready for a long time\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
(2)Kubernetes 内存告警
- alert: KubernetesMemoryPressure
expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
for: 2m
labels:
severity: critical
annotations:
summary: Kubernetes memory pressure (instance {{ $labels.instance }})
description: "{{ $labels.node }} has MemoryPressure condition\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
(3)Kubernetes out of disk
- alert: KubernetesOutOfDisk
expr: kube_node_status_condition{condition="OutOfDisk",status="true"} == 1
for: 2m
labels:
severity: critical
annotations:
summary: Kubernetes out of disk (instance {{ $labels.instance }})
description: "{{ $labels.node }} has OutOfDisk condition\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
配置告警推送到媒介
##
更多推荐
已为社区贡献1条内容
所有评论(0)