Prometheus 向多条路由发出警报，但接收器未发送一条规则

Question

我有此警报配置并期望此行为。

如果

destination: bloom

和

severity: info

发送至

slack-alert-info

- 成功了

那里有错误。如果

destination: bloom

和

severity: warning|critical

发送到

slack-alert-multi

- 这会出错。 严重性：警告按预期发送到 Slack 的两个通道，但严重仅发送到默认通道。

有人可以帮助我理解我的错误吗？

Amtool 没有报错。

amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=warning destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi


amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=critical destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi

警报配置

...
    labels:
      alerttype: infrastructure
      severity: warning
      destination: bloom
...

---
global:
  resolve_timeout: 30m

route:
  group_by: [ 'alertname', 'cluster', 'severity' ]
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 300s
  receiver: 'slack'

  routes:
  - receiver: 'slack-alert-multi'
    match_re:
      destination: bloom
      severity: warning|critical
  - receiver: 'slack-alert-info'
    match_re:
      destination: bloom
      severity: info

receivers:
- name: 'slack-alert-multi'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/V2'
    channel: '#alert-upload'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'
  - api_url: 'https://hooks.slack.com/services/T0/B0/J1'
    channel: '#alert-exports'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

# Default receiver
- name: 'slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/2x'
    channel: '#aws-notification'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

- name: 'slack-alert-info'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/EA'
    channel: '#alert-info'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

templates:
- '/opt/alertmanager_notifications.tmpl'

Answer 1

尝试添加

continue: true

进入

  - receiver: 'slack-alert-info'
    match_re:
      destination: bloom
      severity: info
      continue: true

Answer 2

来自官方文档，“路由相关设置 路由相关设置允许配置警报如何根据时间路由、聚合、限制和静音。

路由块定义路由树中的节点及其子节点。如果未设置，其可选配置参数将从其父节点继承。

每个警报都会在配置的顶级路由处进入路由树，该路由必须匹配所有警报（即没有任何配置的匹配器）。然后遍历子节点。如果 continue 设置为 false，则在第一个匹配的子项之后停止。如果匹配节点上的 continue 为 true，则警报将继续与后续兄弟节点匹配。”

Route-related settings

<route>

# Whether an alert should continue matching subsequent sibling nodes.
[ continue: <boolean> | default = false ]

--工作代码示例-

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1m
  receiver: teams

  routes:
    - receiver: 'fix_webhook'
      match_re:
        severity: critical
        continue: true
    - receiver: 'teams'
      match_re:
        severity: critical

receivers:
  - name: 'fix_webhook'
    webhook_configs:
      - url: 'http://host.docker.internal:5001/webhook'
        send_resolved: true
  - name: 'teams'
    webhook_configs:
      - url: 'https://green-pilot-49.webhook.cool'
        send_resolved: true

参考： https://prometheus.io/docs/alerting/latest/configuration/