如何在alertmanager報(bào)警中添加企業(yè)微信監(jiān)控報(bào)警

Prometheus機(jī)器：172.27.143.155
alertmanager機(jī)器：172.27.143.150

Prometheus 告警分為兩部分。Prometheus 服務(wù)器中的告警規(guī)則會(huì)向Alertmanager發(fā)送告警。

然后，Alertmanager 處理這些告警信息，包括沉默，禁止，聚合和發(fā)送到電子郵件、PagerDuty和HipChat等。

設(shè)置告警和通知的主要步驟是：

設(shè)置并配置Alertmanager

配置Prometheus和Alertmanager通信

在Prometheus中創(chuàng)建告警規(guī)則

Alertmanager處理客戶端應(yīng)用程序（如Prometheus服務(wù)器）發(fā)送的警報(bào)。

它負(fù)責(zé)重復(fù)數(shù)據(jù)刪除，分組并將它們路由到正確的接收方集成，例如電子郵件，PagerDuty或OpsGenie。 它還處理警報(bào)的沉默和抑制。

一、上面配置了Prometheus和grafana服務(wù)在155機(jī)器上面
接下來配置 alermanager服務(wù)
1、wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz

2、tar zxf alertmanager-0.20.0.linux-amd64.tar.gz  
3、mv alertmanager-0.20.0.linux-amd64 /usr/local/alertmanager
4、vim alertmanager.yml

5、vim /etc/alertmanager/template/wechat.tmpl

{{ define "wechat.default.message" }}
{{ range $i, $alert :=.Alerts }}
========監(jiān)控報(bào)警==========
告警狀態(tài)：{{   .Status }}
告警級(jí)別：{{ $alert.Labels.severity }}
告警類型：{{ $alert.Labels.alertname }}
告警應(yīng)用：{{ $alert.Annotations.summary }}
告警主機(jī)：{{ $alert.Labels.instance }}
告警詳情：{{ $alert.Annotations.description }}
觸發(fā)閥值：{{ $alert.Annotations.value }}
告警時(shí)間：{{ $alert.StartsAt.Format "2006-01-02 15:04:05" }}
========end==========
{{ end }}
{{ end }}

6、完成之后啟動(dòng)服務(wù)
nohup ./alertmanager &

接下來配置Prometheus服務(wù)
修改配置文件

需要?jiǎng)?chuàng)建一個(gè) rules目錄
里面又2個(gè)文件，一個(gè)時(shí)主機(jī)監(jiān)控，一個(gè)時(shí)容器監(jiān)控
1）cat host_sys.yml

groups:

• name: Host
  rules:
  • alert: Memory Usage
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 2
    for: 1m
    labels:
    name: Memory
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: "宿主機(jī)內(nèi)存使用率超過80%."
    value: "{{ $value }}"
  • alert: CPU Usage
    expr: sum(avg without (cpu)(irate(node_cpu_seconds_total{mode!='idle'}[5m]))) by (instance,appname) > 0.05
    for: 1m
    labels:
    name: CPU
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: "宿主機(jī)CPU使用率超過65%."
    value: "{{ $value }}"
  • alert: HostLoad
    expr: node_load5 > 4
    for: 1m
    labels:
    name: Load
    severity: Warning
    annotations:
    summary: "{{ $labels.appname }} "
    description: " 主機(jī)負(fù)載5分鐘超過4."
    value: "{{ $value }}"
  • alert: Filesystem Usage
    expr: 1-(node_filesystem_free_bytes / node_filesystem_size_bytes) >  0.3
    for: 1m
    labels:
    name: Disk
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: " 宿主機(jī) [ {{ $labels.mountpoint }} ]分區(qū)使用超過80%."
    value: "{{ $value }}%"
  • alert: Diskio writes
    expr: irate(node_disk_writes_completed_total{job=~"Host"}[1m]) > 50
    for: 1m
    labels:
    name: Diskio
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: " 宿主機(jī) [{{ $labels.device }}]磁盤1分鐘平均寫入IO負(fù)載較高."
    value: "{{ $value }}iops"
  • alert: Diskio reads
    expr: irate(node_disk_reads_completed_total{job=~"Host"}[1m]) > 5
    for: 1m
    labels:
    name: Diskio
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: " 宿主機(jī) [{{ $labels.device }}]磁盤1分鐘平均讀取IO負(fù)載較高."
    value: "{{ $value }}iops"
  • alert: Network_receive
    expr: irate(node_network_receive_bytes_total{device!~"lo|bond[0-9]|cbr[0-9]|veth.|virbr.|ovs-system"}[5m]) / 1048576  > 5
    for: 1m
    labels:
    name: Network_receive
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: " 宿主機(jī) [{{ $labels.device }}] 網(wǎng)卡5分鐘平均接收流量超過5Mbps."
    value: "{{ $value }}Mbps"
  • alert: Network_transmit
    expr: irate(node_network_transmit_bytes_total{device!~"lo|bond[0-9]|cbr[0-9]|veth.|virbr.|ovs-system"}[5m]) / 1048576  > 5
    for: 1m
    labels:
    name: Network_transmit
    severity: Warning
    annotations:
    summary: " {{ $labels.appname }} "
    description: " 宿主機(jī) [{{ $labels.device }}] 網(wǎng)卡5分鐘內(nèi)平均發(fā)送流量超過5Mbps."
    value: "{{ $value }}Mbps"

2） cat container_sys.yml

groups:

• name: Container
  rules:
  • alert: CPU Usage
    expr: (sum by(name,instance) (rate(container_cpu_usage_seconds_total{image!=""}[5m]))*100) > 80
    for: 1m
    labels:
    name: CPU
    severity: Warning
    annotations:
    summary: "{{ $labels.name }} "
    description: " 容器CPU使用超過80%"
    value: "{{ $value }}%"
  • alert: Memory Usage
    expr: (container_memory_usage_bytes{name=~".+"} - container_memory_cache{name=~".+"})  / container_spec_memory_limit_bytes{name=~".+"}   * 100 > 80
    for: 1m
    labels:
    name: Memory
    severity: Warning
    annotations:
    summary: "{{ $labels.name }} "
    description: " 容器內(nèi)存使用超過80%."
    value: "{{ $value }}%"
  • alert: Network_receive
    expr: irate(container_network_receive_bytes_total{name=~".+",interface=~"eth.+"}[5m]) / 1048576  > 5
    for: 1m
    labels:
    name: Network_receive
    severity: Warning
    annotations:
    summary: "{{ $labels.name }} "
    description: "容器 [{{ $labels.device }}] 網(wǎng)卡5分鐘平均接收流量超過5Mbps."
    value: "{{ $value }}Mbps"
  • alert: Network_transmit
    expr: irate(container_network_transmit_bytes_total{name=~".+",interface=~"eth.+"}[5m]) / 1048576  > 5
    for: 1m
    labels:
    name: Network_transmit
    severity: Warning
    annotations:
    summary: "{{ $labels.name }} "
    description: "容器 [{{ $labels.device }}] 網(wǎng)卡5分鐘平均發(fā)送流量超過5Mbps."
    value: "{{ $value }}Mbps"

配置完成之后重啟一下Prometheus服務(wù)

等一會(huì)驗(yàn)證效果

恢復(fù)是下面這個(gè)

監(jiān)控容器就完成了