溫馨提示×
您好，登錄后才能下訂單哦！
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶服務(wù)條款》
您好，登錄后才能下訂單哦！
今天就跟大家聊聊有關(guān)Prometheus中怎么配置自定義告警規(guī)則，可能很多人都不太了解，為了讓大家更加了解，小編給大家總結(jié)了以下內(nèi)容，希望大家根據(jù)這篇文章可以有所收獲。
從下圖我們將看到所有組成Prometheus生態(tài)的組件：
以下是與本文相關(guān)的術(shù)語(yǔ)，大家可以快速了解：
Prometheus Server：在時(shí)序數(shù)據(jù)庫(kù)中抓取和存儲(chǔ)指標(biāo)的主要組件
抓?。阂环N拉取方法以獲取指標(biāo)。它通常以10-60秒的時(shí)間間隔抓取。
Target：檢索數(shù)據(jù)的server客戶端
服務(wù)發(fā)現(xiàn)：?jiǎn)⒂肞rometheus，使其能夠識(shí)別它需要監(jiān)控的應(yīng)用程序并在動(dòng)態(tài)環(huán)境中拉取指標(biāo)
Alert Manager：負(fù)責(zé)處理警報(bào)的組件（包括silencing、inhibition、聚合告警信息，并通過(guò)郵件、PagerDuty、Slack等方式發(fā)送告警通知）。
數(shù)據(jù)可視化：抓取的數(shù)據(jù)存儲(chǔ)在本地存儲(chǔ)中，并使用PromQL直接查詢，或通過(guò)Grafana dashboard查看。
根據(jù)Prometheus Operator的項(xiàng)目所有者CoreOS稱，Prometheus Operator可以配置原生Kubernetes并且可以管理和操作Prometheus和Alertmanager集群。
該Operator引入了以下Kubernetes自定義資源定義（CRDs）：Prometheus、ServiceMonitor、PrometheusRule和Alertmanager。如果你想了解更多內(nèi)容可以訪問(wèn)鏈接：
https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md
在我們的演示中，我們將使用PrometheusRule來(lái)定義自定義規(guī)則。
首先，我們需要使用 stable/prometheus-operator Helm chart來(lái)安裝Prometheus Operator，下載鏈接：
https://github.com/helm/charts/tree/master/stable/prometheus-operator
默認(rèn)安裝程序?qū)?huì)部署以下組件：prometheus-operator、prometheus、alertmanager、node-exporter、kube-state-metrics以及grafana。默認(rèn)狀態(tài)下，Prometheus將會(huì)抓取Kubernetes的主要組件：kube-apiserver、kube-controller-manager以及etcd。
前期準(zhǔn)備
要順利執(zhí)行此次demo，你需要準(zhǔn)備以下內(nèi)容：
一個(gè)Google Cloud Platform賬號(hào)（免費(fèi)套餐即可）。其他任意云也可以
Rancher v2.3.5（發(fā)布文章時(shí)的最新版本）
運(yùn)行在GKE（版本1.15.9-gke.12.）上的Kubernetes集群（使用EKS或AKS也可以）
在計(jì)算機(jī)上安裝好Helm binary
啟動(dòng)一個(gè)Rancher實(shí)例
直接按照這一直觀的入門指南進(jìn)行操作即可：
https://rancher.com/quick-start
使用Rancher部署一個(gè)GKE集群
使用Rancher來(lái)設(shè)置和配置你的Kubernetes集群：
https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/
部署完成后，并且為kubeconfig文件配置了適當(dāng)?shù)腸redential和端點(diǎn)信息，就可以使用kubectl指向該特定集群。
部署Prometheus 軟件
首先，檢查一下我們所運(yùn)行的Helm版本
$ helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}當(dāng)我們使用Helm 3時(shí)，我們需要添加一個(gè)stable 鏡像倉(cāng)庫(kù)，因?yàn)槟J(rèn)狀態(tài)下不會(huì)設(shè)置該倉(cāng)庫(kù)。
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com "stable" has been added to your repositories
$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "stable" chart repository Update Complete. ? Happy Helming!?
$ helm repo list NAME URL stable https://kubernetes-charts.storage.googleapis.com
Helm配置完成后，我們可以開(kāi)始安裝prometheus-operator
$ kubectl create namespace monitoring namespace/monitoring created
$ helm install --namespace monitoring demo stable/prometheus-operator manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" NAME: demo LAST DEPLOYED: Sat Mar 14 09:40:35 2020 NAMESPACE: monitoring STATUS: deployed REVISION: 1 NOTES: The Prometheus Operator has been installed. Check its status by running: kubectl --namespace monitoring get pods -l "release=demo" Visit https://github.com/coreos/prometheus-operator for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
除了監(jiān)控之外，Prometheus還讓我們創(chuàng)建觸發(fā)告警的規(guī)則。這些規(guī)則基于Prometheus的表達(dá)式語(yǔ)言。只要滿足條件，就會(huì)觸發(fā)告警并將其發(fā)送到Alertmanager。之后，我們會(huì)看到規(guī)則的具體形式。
我們回到demo。Helm完成部署之后，我們可以檢查已經(jīng)創(chuàng)建了什么pod：
$ kubectl -n monitoring get pods NAME READY STATUS RESTARTS AGE alertmanager-demo-prometheus-operator-alertmanager-0 2/2 Running 0 61s demo-grafana-5576fbf669-9l57b 3/3 Running 0 72s demo-kube-state-metrics-67bf64b7f4-4786k 1/1 Running 0 72s demo-prometheus-node-exporter-ll8zx 1/1 Running 0 72s demo-prometheus-node-exporter-nqnr6 1/1 Running 0 72s demo-prometheus-node-exporter-sdndf 1/1 Running 0 72s demo-prometheus-operator-operator-b9c9b5457-db9dj 2/2 Running 0 72s prometheus-demo-prometheus-operator-prometheus-0 3/3 Running 1 50s
為了從web瀏覽器中訪問(wèn)Prometheus和Alertmanager，我們需要使用port轉(zhuǎn)發(fā)。
由于本例中使用的是GCP實(shí)例，并且所有的kubectl命令都從該實(shí)例運(yùn)行，因此我們使用實(shí)例的外部IP地址訪問(wèn)資源。
$ kubectl port-forward --address 0.0.0.0 -n monitoring prometheus-demo-prometheus-operator-prometheus-0 9090 >/dev/null 2>&1 &
$ kubectl port-forward --address 0.0.0.0 -n monitoring alertmanager-demo-prometheus-operator-alertmanager-0 9093 >/dev/null 2>&1 &
“Alert”選項(xiàng)卡向我們展示了所有當(dāng)前正在運(yùn)行/已配置的告警。也可以通過(guò)查詢名稱為prometheusrules的CRD從CLI進(jìn)行檢查：
$ kubectl -n monitoring get prometheusrules NAME AGE demo-prometheus-operator-alertmanager.rules 3m21s demo-prometheus-operator-etcd 3m21s demo-prometheus-operator-general.rules 3m21s demo-prometheus-operator-k8s.rules 3m21s demo-prometheus-operator-kube-apiserver-error 3m21s demo-prometheus-operator-kube-apiserver.rules 3m21s demo-prometheus-operator-kube-prometheus-node-recording.rules 3m21s demo-prometheus-operator-kube-scheduler.rules 3m21s demo-prometheus-operator-kubernetes-absent 3m21s demo-prometheus-operator-kubernetes-apps 3m21s demo-prometheus-operator-kubernetes-resources 3m21s demo-prometheus-operator-kubernetes-storage 3m21s demo-prometheus-operator-kubernetes-system 3m21s demo-prometheus-operator-kubernetes-system-apiserver 3m21s demo-prometheus-operator-kubernetes-system-controller-manager 3m21s demo-prometheus-operator-kubernetes-system-kubelet 3m21s demo-prometheus-operator-kubernetes-system-scheduler 3m21s demo-prometheus-operator-node-exporter 3m21s demo-prometheus-operator-node-exporter.rules 3m21s demo-prometheus-operator-node-network 3m21s demo-prometheus-operator-node-time 3m21s demo-prometheus-operator-node.rules 3m21s demo-prometheus-operator-prometheus 3m21s demo-prometheus-operator-prometheus-operator 3m21s
我們也可以檢查位于prometheus容器中prometheus-operator Pod中的物理文件。
$ kubectl -n monitoring exec -it prometheus-demo-prometheus-operator-prometheus-0 -- /bin/sh Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-demo-prometheus-operator-prometheus-0 -n monitoring' to see all of the containers in this pod.
在容器中，我們可以檢查規(guī)則的存儲(chǔ)路徑：
/prometheus $ ls /etc/prometheus/rules/prometheus-demo-prometheus-operator-prometheus-rulefiles-0/ monitoring-demo-prometheus-operator-alertmanager.rules.yaml monitoring-demo-prometheus-operator-kubernetes-system-apiserver.yaml monitoring-demo-prometheus-operator-etcd.yaml monitoring-demo-prometheus-operator-kubernetes-system-controller-manager.yaml monitoring-demo-prometheus-operator-general.rules.yaml monitoring-demo-prometheus-operator-kubernetes-system-kubelet.yaml monitoring-demo-prometheus-operator-k8s.rules.yaml monitoring-demo-prometheus-operator-kubernetes-system-scheduler.yaml monitoring-demo-prometheus-operator-kube-apiserver-error.yaml monitoring-demo-prometheus-operator-kubernetes-system.yaml monitoring-demo-prometheus-operator-kube-apiserver.rules.yaml monitoring-demo-prometheus-operator-node-exporter.rules.yaml monitoring-demo-prometheus-operator-kube-prometheus-node-recording.rules.yaml monitoring-demo-prometheus-operator-node-exporter.yaml monitoring-demo-prometheus-operator-kube-scheduler.rules.yaml monitoring-demo-prometheus-operator-node-network.yaml monitoring-demo-prometheus-operator-kubernetes-absent.yaml monitoring-demo-prometheus-operator-node-time.yaml monitoring-demo-prometheus-operator-kubernetes-apps.yaml monitoring-demo-prometheus-operator-node.rules.yaml monitoring-demo-prometheus-operator-kubernetes-resources.yaml monitoring-demo-prometheus-operator-prometheus-operator.yaml monitoring-demo-prometheus-operator-kubernetes-storage.yaml monitoring-demo-prometheus-operator-prometheus.yaml
為了詳細(xì)了解如何將這些規(guī)則加載到Prometheus中，請(qǐng)檢查Pod的詳細(xì)信息。我們可以看到用于prometheus容器的配置文件是etc/prometheus/config_out/prometheus.env.yaml。該配置文件向我們展示了文件的位置或重新檢查yaml的頻率設(shè)置。
$ kubectl -n monitoring describe pod prometheus-demo-prometheus-operator-prometheus-0
完整命令輸出如下：
Name: prometheus-demo-prometheus-operator-prometheus-0 Namespace: monitoring Priority: 0 Node: gke-c-7dkls-default-0-c6ca178a-gmcq/10.132.0.15 Start Time: Wed, 11 Mar 2020 18:06:47 +0000 Labels: app=prometheus controller-revision-hash=prometheus-demo-prometheus-operator-prometheus-5ccbbd8578 prometheus=demo-prometheus-operator-prometheus statefulset.kubernetes.io/pod-name=prometheus-demo-prometheus-operator-prometheus-0 Annotations: <none> Status: Running IP: 10.40.0.7 IPs: <none> Controlled By: StatefulSet/prometheus-demo-prometheus-operator-prometheus Containers: prometheus: Container ID: docker://360db8a9f1cce8d72edd81fcdf8c03fe75992e6c2c59198b89807aa0ce03454c Image: quay.io/prometheus/prometheus:v2.15.2 Image ID: docker-pullable://quay.io/prometheus/prometheus@sha256:914525123cf76a15a6aaeac069fcb445ce8fb125113d1bc5b15854bc1e8b6353 Port: 9090/TCP Host Port: 0/TCP Args: --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://demo-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/ State: Running Started: Wed, 11 Mar 2020 18:07:07 +0000 Last State: Terminated Reason: Error Message: caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-03-11T18:07:02.185Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-03-11T18:07:02.192Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-03-11T18:07:02.192Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-03-11T18:07:02.194Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-03-11T18:07:02.194Z caller=main.go:664 msg="TSDB started" level=info ts=2020-03-11T18:07:02.194Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml level=info ts=2020-03-11T18:07:02.194Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-03-11T18:07:02.194Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-03-11T18:07:02.194Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-03-11T18:07:02.194Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-03-11T18:07:02.194Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-03-11T18:07:02.194Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-03-11T18:07:02.194Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-03-11T18:07:02.194Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-03-11T18:07:02.197Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-03-11T18:07:02.197Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-03-11T18:07:02.197Z caller=main.go:727 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory" Exit Code: 1 Started: Wed, 11 Mar 2020 18:07:02 +0000 Finished: Wed, 11 Mar 2020 18:07:02 +0000 Ready: True Restart Count: 1 Liveness: http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6 Readiness: http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120 Environment: <none> Mounts: /etc/prometheus/certs from tls-assets (ro) /etc/prometheus/config_out from config-out (ro) /etc/prometheus/rules/prometheus-demo-prometheus-operator-prometheus-rulefiles-0 from prometheus-demo-prometheus-operator-prometheus-rulefiles-0 (rw) /prometheus from prometheus-demo-prometheus-operator-prometheus-db (rw) /var/run/secrets/kubernetes.io/serviceaccount from demo-prometheus-operator-prometheus-token-jvbrr (ro) prometheus-config-reloader: Container ID: docker://de27cdad7067ebd5154c61b918401b2544299c161850daf3e317311d2d17af3d Image: quay.io/coreos/prometheus-config-reloader:v0.37.0 Image ID: docker-pullable://quay.io/coreos/prometheus-config-reloader@sha256:5e870e7a99d55a5ccf086063efd3263445a63732bc4c04b05cf8b664f4d0246e Port: <none> Host Port: <none> Command: /bin/prometheus-config-reloader Args: --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml State: Running Started: Wed, 11 Mar 2020 18:07:04 +0000 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 25Mi Requests: cpu: 100m memory: 25Mi Environment: POD_NAME: prometheus-demo-prometheus-operator-prometheus-0 (v1:metadata.name) Mounts: /etc/prometheus/config from config (rw) /etc/prometheus/config_out from config-out (rw) /var/run/secrets/kubernetes.io/serviceaccount from demo-prometheus-operator-prometheus-token-jvbrr (ro) rules-configmap-reloader: Container ID: docker://5804e45380ed1b5374a4c2c9ee4c9c4e365bee93b9ccd8b5a21f50886ea81a91 Image: quay.io/coreos/configmap-reload:v0.0.1 Image ID: docker-pullable://quay.io/coreos/configmap-reload@sha256:e2fd60ff0ae4500a75b80ebaa30e0e7deba9ad107833e8ca53f0047c42c5a057 Port: <none> Host Port: <none> Args: --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-demo-prometheus-operator-prometheus-rulefiles-0 State: Running Started: Wed, 11 Mar 2020 18:07:06 +0000 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 25Mi Requests: cpu: 100m memory: 25Mi Environment: <none> Mounts: /etc/prometheus/rules/prometheus-demo-prometheus-operator-prometheus-rulefiles-0 from prometheus-demo-prometheus-operator-prometheus-rulefiles-0 (rw) /var/run/secrets/kubernetes.io/serviceaccount from demo-prometheus-operator-prometheus-token-jvbrr (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: Secret (a volume populated by a Secret) SecretName: prometheus-demo-prometheus-operator-prometheus Optional: false tls-assets: Type: Secret (a volume populated by a Secret) SecretName: prometheus-demo-prometheus-operator-prometheus-tls-assets Optional: false config-out: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> prometheus-demo-prometheus-operator-prometheus-rulefiles-0: Type: ConfigMap (a volume populated by a ConfigMap) Name: prometheus-demo-prometheus-operator-prometheus-rulefiles-0 Optional: false prometheus-demo-prometheus-operator-prometheus-db: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> demo-prometheus-operator-prometheus-token-jvbrr: Type: Secret (a volume populated by a Secret) SecretName: demo-prometheus-operator-prometheus-token-jvbrr Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m51s default-scheduler Successfully assigned monitoring/prometheus-demo-prometheus-operator-prometheus-0 to gke-c-7dkls-default-0-c6ca178a-gmcq Normal Pulling 4m45s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Pulling image "quay.io/prometheus/prometheus:v2.15.2" Normal Pulled 4m39s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Successfully pulled image "quay.io/prometheus/prometheus:v2.15.2" Normal Pulling 4m36s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Pulling image "quay.io/coreos/prometheus-config-reloader:v0.37.0" Normal Pulled 4m35s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Successfully pulled image "quay.io/coreos/prometheus-config-reloader:v0.37.0" Normal Pulling 4m34s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Pulling image "quay.io/coreos/configmap-reload:v0.0.1" Normal Started 4m34s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Started container prometheus-config-reloader Normal Created 4m34s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Created container prometheus-config-reloader Normal Pulled 4m33s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Successfully pulled image "quay.io/coreos/configmap-reload:v0.0.1" Normal Created 4m32s (x2 over 4m36s) kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Created container prometheus Normal Created 4m32s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Created container rules-configmap-reloader Normal Started 4m32s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Started container rules-configmap-reloader Normal Pulled 4m32s kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Container image "quay.io/prometheus/prometheus:v2.15.2" already present on machine Normal Started 4m31s (x2 over 4m36s) kubelet, gke-c-7dkls-default-0-c6ca178a-gmcq Started container prometheus
讓我們清理默認(rèn)規(guī)則，使得我們可以更好地觀察我們將要?jiǎng)?chuàng)建的那個(gè)規(guī)則。以下命令將刪除所有規(guī)則，但會(huì)留下monitoring-demo-prometheus-operator-alertmanager.rules。
$ kubectl -n monitoring delete prometheusrules $(kubectl -n monitoring get prometheusrules | grep -v alert)
$ kubectl -n monitoring get prometheusrules NAME AGE demo-prometheus-operator-alertmanager.rules 8m53s
請(qǐng)注意：我們只保留一條規(guī)則是為了讓demo更容易。但是有一條規(guī)則，你絕對(duì)不能刪除，它位于monitoring-demo-prometheus-operator-general.rules.yaml中，被稱為看門狗。該告警總是處于觸發(fā)狀態(tài)，其目的是確保整個(gè)告警流水線正常運(yùn)轉(zhuǎn)。
讓我們從CLI中檢查我們留下的規(guī)則并將其與我們將在瀏覽器中看到的進(jìn)行比較。
$ kubectl -n monitoring describe prometheusrule demo-prometheus-operator-alertmanager.rules
Name: demo-prometheus-operator-alertmanager.rules
Namespace: monitoring
Labels: app=prometheus-operator
chart=prometheus-operator-8.12.1
heritage=Tiller
release=demo
Annotations: prometheus-operator-validated: true
API Version: monitoring.coreos.com/v1
Kind: PrometheusRule
Metadata:
Creation Timestamp: 2020-03-11T18:06:25Z
Generation: 1
Resource Version: 4871
Self Link: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules/demo-prometheus-operator-alertmanager.rules
UID: 6a84dbb0-feba-4f17-b3dc-4b6486818bc0
Spec:
Groups:
Name: alertmanager.rules
Rules:
Alert: AlertmanagerConfigInconsistent
Annotations:
Message: The configuration of the instances of the Alertmanager cluster `{{$labels.service}}` are out of sync.
Expr: count_values("config_hash", alertmanager_config_hash{job="demo-prometheus-operator-alertmanager",namespace="monitoring"}) BY (service) / ON(service) GROUP_LEFT() label_replace(max(prometheus_operator_spec_replicas{job="demo-prometheus-operator-operator",namespace="monitoring",controller="alertmanager"}) by (name, job, namespace, controller), "service", "$1", "name", "(.*)") != 1
For: 5m
Labels:
Severity: critical
Alert: AlertmanagerFailedReload
Annotations:
Message: Reloading Alertmanager's configuration has failed for {{ $labels.namespace }}/{{ $labels.pod}}.
Expr: alertmanager_config_last_reload_successful{job="demo-prometheus-operator-alertmanager",namespace="monitoring"} == 0
For: 10m
Labels:
Severity: warning
Alert: AlertmanagerMembersInconsistent
Annotations:
Message: Alertmanager has not found all other members of the cluster.
Expr: alertmanager_cluster_members{job="demo-prometheus-operator-alertmanager",namespace="monitoring"}
!= on (service) GROUP_LEFT()
count by (service) (alertmanager_cluster_members{job="demo-prometheus-operator-alertmanager",namespace="monitoring"})
For: 5m
Labels:
Severity: critical
Events: <none>
讓我們移除所有默認(rèn)告警并創(chuàng)建一個(gè)我們自己的告警：
$ kubectl -n monitoring edit prometheusrules demo-prometheus-operator-alertmanager.rules prometheusrule.monitoring.coreos.com/demo-prometheus-operator-alertmanager.rules edited
我們的自定義告警如下所示：
$ kubectl -n monitoring describe prometheusrule demo-prometheus-operator-alertmanager.rules
Name: demo-prometheus-operator-alertmanager.rules
Namespace: monitoring
Labels: app=prometheus-operator
chart=prometheus-operator-8.12.1
heritage=Tiller
release=demo
Annotations: prometheus-operator-validated: true
API Version: monitoring.coreos.com/v1
Kind: PrometheusRule
Metadata:
Creation Timestamp: 2020-03-11T18:06:25Z
Generation: 3
Resource Version: 18180
Self Link: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules/demo-prometheus-operator-alertmanager.rules
UID: 6a84dbb0-feba-4f17-b3dc-4b6486818bc0
Spec:
Groups:
Name: alertmanager.rules
Rules:
Alert: PodHighCpuLoad
Annotations:
Message: Alertmanager has found {{ $labels.instance }} with CPU too high
Expr: rate (container_cpu_usage_seconds_total{pod_name=~"nginx-.*", image!="", container!="POD"}[5m]) > 0.04
For: 1m
Labels:
Severity: critical
Events: <none>
以下是我們創(chuàng)建的告警的選項(xiàng)：
annotation：描述告警的信息標(biāo)簽集。
expr：由PromQL寫的表達(dá)式
for：可選參數(shù)，設(shè)置了之后會(huì)告訴Prometheus在定義的時(shí)間段內(nèi)告警是否處于active狀態(tài)。僅在此定義時(shí)間后才會(huì)觸發(fā)告警。
label：可以附加到告警的額外標(biāo)簽。如果你想了解更多關(guān)于告警的信息，可以訪問(wèn)： https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
現(xiàn)在我們已經(jīng)完成了Prometheus告警的設(shè)置，讓我們配置Alertmanager，使得我們能夠通過(guò)電子郵件獲得告警通知。Alertmanager的配置位于Kubernetes secret對(duì)象中。
$ kubectl get secrets -n monitoring NAME TYPE DATA AGE alertmanager-demo-prometheus-operator-alertmanager Opaque 1 32m default-token-x4rgq kubernetes.io/service-account-token 3 37m demo-grafana Opaque 3 32m demo-grafana-test-token-p6qnk kubernetes.io/service-account-token 3 32m demo-grafana-token-ff6nl kubernetes.io/service-account-token 3 32m demo-kube-state-metrics-token-vmvbr kubernetes.io/service-account-token 3 32m demo-prometheus-node-exporter-token-wlnk9 kubernetes.io/service-account-token 3 32m demo-prometheus-operator-admission Opaque 3 32m demo-prometheus-operator-alertmanager-token-rrx4k kubernetes.io/service-account-token 3 32m demo-prometheus-operator-operator-token-q9744 kubernetes.io/service-account-token 3 32m demo-prometheus-operator-prometheus-token-jvbrr kubernetes.io/service-account-token 3 32m prometheus-demo-prometheus-operator-prometheus Opaque 1 31m prometheus-demo-prometheus-operator-prometheus-tls-assets Opaque 0 31m
我們只對(duì)alertmanager-demo-prometheus-operator-alertmanager感興趣。讓我們看一下：
kubectl -n monitoring get secret alertmanager-demo-prometheus-operator-alertmanager -o yaml apiVersion: v1 data: alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwiCg== kind: Secret metadata: creationTimestamp: "2020-03-11T18:06:24Z" labels: app: prometheus-operator-alertmanager chart: prometheus-operator-8.12.1 heritage: Tiller release: demo name: alertmanager-demo-prometheus-operator-alertmanager namespace: monitoring resourceVersion: "3018" selfLink: /api/v1/namespaces/monitoring/secrets/alertmanager-demo-prometheus-operator-alertmanager uid: 6baf6883-f690-47a1-bb49-491935956c22 type: Opaque
alertmanager.yaml字段是由base64編碼的，讓我們看看：
$ echo 'Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwiCg==' | base64 --decode global: resolve_timeout: 5m receivers: - name: "null" route: group_by: - job group_interval: 5m group_wait: 30s receiver: "null" repeat_interval: 12h routes: - match: alertname: Watchdog receiver: "null"
正如我們所看到的，這是默認(rèn)的Alertmanager配置。你也可以在Alertmanager UI的Status選項(xiàng)卡中查看此配置。接下來(lái)，我們來(lái)對(duì)它進(jìn)行一些更改——在本例中為發(fā)送郵件：
$ cat alertmanager.yaml global: resolve_timeout: 5m route: group_by: [Alertname] # Send all notifications to me. receiver: demo-alert group_wait: 30s group_interval: 5m repeat_interval: 12h routes: - match: alertname: DemoAlertName receiver: 'demo-alert' receivers: - name: demo-alert email_configs: - to: your_email@gmail.com from: from_email@gmail.com # Your smtp server address smarthost: smtp.gmail.com:587 auth_username: from_email@gmail.com auth_identity: from_email@gmail.com auth_password: 16letter_generated token # you can use gmail account password, but better create a dedicated token for this headers: From: from_email@gmail.com Subject: 'Demo ALERT'
首先，我們需要對(duì)此進(jìn)行編碼：
$ cat alertmanager.yaml | base64 -w0
我們獲得編碼輸出后，我們需要在我們將要應(yīng)用的yaml文件中填寫它：
cat alertmanager-secret-k8s.yaml apiVersion: v1 data: alertmanager.yaml: <paste here de encoded content of alertmanager.yaml> kind: Secret metadata: name: alertmanager-demo-prometheus-operator-alertmanager namespace: monitoring type: Opaque
$ kubectl apply -f alertmanager-secret-k8s.yaml Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply secret/alertmanager-demo-prometheus-operator-alertmanager configured
該配置將會(huì)自動(dòng)重新加載并在UI中顯示更改。
接下來(lái)，我們部署一些東西來(lái)對(duì)其進(jìn)行監(jiān)控。對(duì)于本例而言，一個(gè)簡(jiǎn)單的nginx deployment已經(jīng)足夠：
$ cat nginx-deployment.yaml apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 3 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
$ kubectl apply -f nginx-deployment.yaml deployment.apps/nginx-deployment created
根據(jù)配置yaml，我們有3個(gè)副本：
$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-5754944d6c-7g6gq 1/1 Running 0 67s nginx-deployment-5754944d6c-lhvx8 1/1 Running 0 67s nginx-deployment-5754944d6c-whhtr 1/1 Running 0 67s
在Prometheus UI中，使用我們?yōu)楦婢渲玫南嗤磉_(dá)式：
rate (container_cpu_usage_seconds_total{pod_name=~"nginx-.*", image!="", container!="POD"}[5m])我們可以為這些Pod檢查數(shù)據(jù)，所有Pod的值應(yīng)該為0。
讓我們?cè)谄渲幸粋€(gè)pod中添加一些負(fù)載，然后來(lái)看看值的變化，當(dāng)值大于0.04時(shí)，我們應(yīng)該接收到告警：
$ kubectl exec -it nginx-deployment-5754944d6c-7g6gq -- /bin/sh # yes > /dev/null
該告警有3個(gè)階段：
Inactive：不滿足告警觸發(fā)條件
Pending：條件已滿足
Firing：觸發(fā)告警
我們已經(jīng)看到告警處于inactive狀態(tài)，所以在CPU上添加一些負(fù)載，以觀察到剩余兩種狀態(tài)：
告警一旦觸發(fā)，將會(huì)在Alertmanager中顯示：
Alertmanger配置為當(dāng)我們收到告警時(shí)發(fā)送郵件。所以此時(shí)，如果我們檢查收件箱，會(huì)看到類似以下內(nèi)容：
看完上述內(nèi)容，你們對(duì)Prometheus中怎么配置自定義告警規(guī)則有進(jìn)一步的了解嗎？如果還想了解更多知識(shí)或者相關(guān)內(nèi)容，請(qǐng)關(guān)注億速云行業(yè)資訊頻道，感謝大家的支持。
免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。
億速云公眾號(hào)
手機(jī)網(wǎng)站二維碼