Prometheus AlertManager Production Practice - Send alerts directly to the corresponding mailbox according to the to_email label

overview

By previous article - Summary of the pitfalls that Prometheus Alertmanager has trodden through production configurationsWe already know that AlertManager, as an alerting platform, is very powerful and can deduplicate (deduplicating), grouping, and routing them to the right receiver integration, such as email, WeChat, or DingTalk. It is also responsible for handling silencing, timed send/do not send (Mute), and inhibition issues of alarms.

The normal AlertManager process of handling alerts goes through the Alerts -> Route -> Receivers step

1. Alerts include tags such as env, team, job, etc
2. According to the pre-edited route, route alerts, such as which receivers for env=prod and team=db to whom to send it.
3. These receivers mailboxes that need to process prod and db alerts have been entered in advance in receivers. The alert is sent to the corresponding recipient.

However, if I bring my own recipient information (such as email) in Alerts, can I use it directly? There is no need to enter all receivers.

The answer, of course, is yes! This requirement is achieved through a template. Let’s GO！ 💪💪💪

Introduction to Template

The original purpose of the AlertManager template was to customize the alert message.
For example of the same Alerts, I:

• Sent via SMS, expect plain text;
• Sent via email, expect to be in HTML format;
• Send through DingTalk, Qiwei, expect to be in Markdown format;
• And in these channels,
  • Headings are different permutations
  • The content of the alarm is also different paragraph formats and words (for example, through DingTalk, Qwei will add more emojis)

The AlertManager template is the same as the Prometheus template, using the same Go template. Of course, there will be subtle differences between the specific data and functions, because the main treatment here is alarmsgrouprather than individual alarms.

Examples are as follows:

receivers:
  - name: emergency
    slack_configs:
    - api_url: https://hooks.slack.com/services/XXXXXXXX
      channel: '#emergency'
      title: Prometheus AlertManager Production Practice - Send alerts directly to the corresponding mailbox according to the to_email label

AlertManager advanced

In addition to templating txt fields, the definition of notifications (e.g. to whom) can also be templated. Usually each team has its own route tree and corresponding receivers. If another team (not a monitoring team, not an operations team, but a testing team, etc.) wants to send alerts to their own team, they need to set up labels from start to finish, set up a routing tree that matches their team’s labels, and configure the recipient information within the team into the receiver in AlertManager.

Then if you are a monitoring team, you use AlertManager to make an alarm platform for external teams and even customers to use, every time you have to do this, there will be “hundreds of millions of points” trouble.

What to do? 🤔🤔🤔

solution

The solution is:

• Label
• AlertManager notification template

First, provide the relevant recipient information directly in the label, and then through the AlertManager template, will receiver -> to Just write the corresponding template.

The specific demonstration is as follows:

Scenario demonstration

First, there are alerts that contain the recipient information label, as follows:

[
  {
    "labels": {
      "alertname": "<requiredAlertName>",
      "<labelname>": "<labelvalue>",
      "email_to": "[email protected],[email protected]",
      ...
    },
    "annotations": {
      "<labelname>": "<labelvalue>",
    },
    "startsAt": "<rfc3339>",
    "endsAt": "<rfc3339>",
    "generatorURL": "<generator_url>"
  },
  ...
]

Every alert is provided email_to Such a label.

Then, in AlertManager, you can set it up as follows route and receiverAs follows:

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: '[email protected]'
route:
  group_by: [email_to, alertname]
  receiver: customer_email
receivers:
  - name: customer_email
    email_configs:
      - to: '{{ .GroupLabels.email_to }}'
    headers:
      subject: 'Alert: {{ .GroupLabels.alertname }}'

notegroup_by Must include: email_to label, so that it counts .GroupLabels. Next member.

When alerts come, such as "email_to": "[email protected],[email protected]", will route to customer_email, whose recipient is {{ .GroupLabels.email_to }}, will be templated as: [email protected],[email protected], the warning email will naturally be sent.

Finish! 🎉🎉🎉