Use Jiralert to implement AlertManager alarm docking with Jira

This article was last updated on: February 7, 2024 pm

Brief introduction

Alertmanager Handles alerts sent by client applications, such as Prometheus server. It is responsible for deduplicating, grouping, and routing them to the correct receiver integration, such as email, WeChat, or DingTalk. It is also responsible for handling silencing, timed send/do not send (Mute), and inhibition of alarms.

As an open source alerting application designed for Prometheus, AlertManager already has a variety of rich, flexible, and customizable functions of alerting applications:

Jiralert

Prometheus Alertmanager Webhook Receiver for JIRA。

JIRAlert implements Alertmanager’s webhook HTTP API and connects to one or more JIRA instances to create highly configurable JIRA Issues. Each different Groupkey creates an issue – by the Alertmanager’s Routing Configuration sectiongroup_byParameter definition - but does not close when the alarm is resolved (default parameter, adjustable). Our expectation is that people will look at this issue. , take any necessary action, and then close it. If human interaction is not necessary, then it probably shouldn’t have called the police in the first place. However, this behavior can be set throughauto_resolvesection is modified and it will resolve the JIRA issue in the desired state.

If a corresponding JIRA issue. Already exists, but is solved and it will be reopened. There must be one between the resolved state and the reopened stateJIRA transition–asreopen_state–Otherwise, the reopening will fail. You can optionally define a “won’t fix” resolution (resolution–Bywont_fix_resolutionDefinition: JIRA issues with this resolution will not be reopened by JIRARlert.

Install Jiralert

The installation of Jiralert is relatively simple, mainly consisting of Deployment, Secret (Jiralert’s configuration) and Service. Typical examples are as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jiralert
spec:
  selector:
    matchLabels:
      app: jiralert
  template:
    metadata:
      labels:
        app: jiralert
    spec:
      containers:
      - name: jiralert
        image: quay.io/jiralert/jiralert-linux-amd64:latest
        imagePullPolicy: IfNotPresent
        args:
        - "--config=/jiralert-config/jiralert.yml"
        - "--log.level=debug"
        - "--listen-address=:9097"
        readinessProbe:
          tcpSocket:
            port: 9097
          initialDelaySeconds: 15
          periodSeconds: 15
          timeoutSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 9097
          initialDelaySeconds: 15
          periodSeconds: 15
          timeoutSeconds: 5
        ports:
        - containerPort: 9091
          name: metrics
        volumeMounts:
        - mountPath: /jiralert-config
          name: jiralert-config
          readOnly: true
      volumes:
      - name: jiralert-config
        secret:
          secretName: jiralert-config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: jiralert-config
stringData:
  jiralert.tmpl: |-
    {{ define "jira.summary" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join "," }}{{ end }}
    
    {{ define "jira.description" }}{{ range .Alerts.Firing }}Labels:
    {{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
    {{ end }}
    
    Annotations:
    {{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
    {{ end }}
    
    Source: {{ .GeneratorURL }}
    {{ end }}
    
    CommonLabels:
    {{ range .CommonLabels.SortedPairs }} - {{ .Name }} = {{ .Value}}
    {{ end }}
    
    GroupLabels:
    {{ range .GroupLabels.SortedPairs }} - {{ .Name }} = {{ .Value}}
    {{ end }}
    {{ end }}
  jiralert.yml: |-
    # Global defaults, applied to all receivers where not explicitly overridden. Optional.
    template: jiralert.tmpl
    defaults:
      # API access fields.
      api_url: https://jira.example.com
      user: foo
      password: bar
      # The type of JIRA issue to create. Required.
      issue_type: Bug
      # Issue priority. Optional.
      priority: Major
      # Go template invocation for generating the summary. Required.
      summary: '{{ template "jira.summary" . }}'
      # Go template invocation for generating the description. Optional.
      description: '{{ template "jira.description" . }}'
      # State to transition into when reopening a closed issue. Required.
      reopen_state: "REOPENED"
      # Do not reopen issues with this resolution. Optional.
      wont_fix_resolution: "Won't Fix"
      # Amount of time after being closed that an issue should be reopened, after which, a new issue is created.
      # Optional (default: always reopen)
      # reopen_duration: 30d
    
    # Receiver definitions. At least one must be defined.
    # Receiver names must match the Alertmanager receiver names. Required.
    receivers:
    - name: 'jiralert'
      project: 'YOUR-JIRA-PROJECT'
1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: Service
metadata:
  name: jiralert
spec:
  selector:
    app: jiralert
  ports:
  - port: 9097
    targetPort: 9097

Configuration of the corresponding AlertManager:

1
2
3
4
5
6
7
8
9
10
11
12
...
receivers:
- name: jiralert
  webhook_configs:
  - send_resolved: true
    url: http://jiralert:9097/alert
routes:
- receiver: jiralert
  matchers:
  - severity = critical
  continue: true
...

📝 Illustrate:

  • Official JirAlert image address: https://quay.io/repository/jiralert/jiralert-linux-amd64?tab=tags
  • jiralert.tmpl Similar to AlertManager’s template, issues sent to Jira use this as a template
  • jiralert.yml Jiralert’s configuration file
    • defaults Basic configuration
    • receivers Multiple receivers can be set up, and which Jira receiver AlertManager wants to send to needs to have the same name as the jiralert receiver. (Like the example above, both of them.)jiralert)

Jiralert configuration

The complete production Jiralert configuration is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Global defaults, applied to all receivers where not explicitly overridden. Optional.
template: jiralert.tmpl
defaults:
  # API access fields.
  api_url: https://example.atlassian.net
  user: <your-account-email>
  password: '<your-account-api-token>'
  # The type of JIRA issue to create. Required.
  issue_type: Support
  # Issue priority. Optional.
  priority: High
  # Go template invocation for generating the summary. Required.
  summary: '{{ template "jira.summary" . }}'
  # Go template invocation for generating the description. Optional.
  description: '{{ template "jira.description" . }}'
  # State to transition into when reopening a closed issue. Required.
  reopen_state: "Back to in progress"
  # Do not reopen issues with this resolution. Optional.
  wont_fix_resolution: "Won't Do"
  # Amount of time after being closed that an issue should be reopened, after which, a new issue is created.
  # Optional (default: always reopen)
  reopen_duration: 30d

# Receiver definitions. At least one must be defined.
# Receiver names must match the Alertmanager receiver names. Required.
receivers:
- name: 'jiralert'
  project: <your-project-code>
  add_group_labels: true
  auto_resolve:
    state: 'Resolve this issue'

📝 The detailed description is as follows:

  1. api_url: Jira’s address, if using Jira’s SaaS service, ishttps://<tenant>.atlassian.net
  2. Authentication:
    1. For the public cloud version of Jira, it can only be used user and passwordThereinto:
      1. user Fill in your account email address;
      2. password Need to be in first API Token | Atlassian account Apply for API Token. (🐾 Note: The password used for login cannot be authenticated)
    2. For other versions, it can also be filled in for use personal_access_token Conduct certification. Its values are: [email protected]:api_token_string The base64 encoded string. For specific instructions, see: Basic auth for REST APIs (atlassian.com)
  3. issue_type: Depending on your Jira Issue Type, it could be: Alert Support Bug New Feature And so on or so on
  4. priority Depending on your Issue priority, it may be: Critical High Medium Low And so on or so on
  5. reopen_state: Jira’s issue has been closed, to reopen, needed transitionAs: Back to in progress. (🐾 Note: What needs to be filled in here is your custom.) transition, and not status)
  6. wont_fix_resolution: With this resolution The problem (solution) will not be reopened. As: Won't Do Won't Fix, need to be according to their own resolution Define the content to fill in.
  7. reopen_duration: How long does it take for the issue to reopen, default is always reopen, which can be set to such as: 30d, indicating that if the same issue was 30 days ago, open a new issue instead of reopening the old issue.
  8. receivers: Multiple receivers can be defined, pointing to different ones project
  9. project: Jira’s Project ID, which is the initial capital of the project’s detailed name. Such as Project is For Example, fill in here FE
  10. add_group_labels: Whether you want to add AlertManager’s Group Labels to Jira’s Labels. (🐾 Note: Jira Labels cannot have spaces in their Value, so if your AlertManager’s Group Label Value has spaces.) NoTurn on this feature)
  11. auto_resolve: The latest 1.2 version adds a new feature, when the alarm is restored, you can automatically resolve the corresponding Jira Issue.
    1. state: 'Resolve this issue' Here is also to fill in your predefined Jira to solve the problem transition And not statusas'Resolve this issue'.

Other difficult situations

If you encounter all kinds of weird logs, most of the reasons are caused by not properly authenticating and logging in, typical such as this error:

1
The value 'XXX' does not exist for the field 'project'.

In fact, it is caused by not properly authenticating the login.

For details, please refer to here: Solved: REST error "The value ‘XXX’ does not exist for the… (atlassian.com)

There is also a category of errors that tell you that you can’t transition an issueThis is often due to the following reasons:

  1. Jiralertreopen_state or auto_resolve target state Not filled in correctly transition
  2. The account you are using does not have the corresponding permissions
  3. The status the Issue is in now (eg Closed) is not allowed to proceed again transition

For details, please refer to here: I can’t transition an issue in my Jira project - W… - Atlassian Community

The final effect

As shown in the following figure:

Jiralert 效果

You can create an issue, update Summary, update Description, Update Resolution, Update Status; The same problem recurred, reopening the previous issue…

🎉🎉🎉

📚️ Reference documentation