This article was last updated on: July 24, 2024 am
overview
Usually when we want to silence the alert of an AlertManager, we need to operate through the UI interface, as shown below:
It’s a bit inefficient and not automated enough, so is there a way to quickly create an AlertManager silence?
– Yes, via API.
API Payload
v1
As follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| {
"matchers": [
{
"name": "alername1",
"value": ".*",
"isRegex": true
}
],
"startsAt": "2022-04-29T22:12:33.533Z",
"endsAt": "2022-04-29T23:11:44.603Z",
"createdBy": "api",
"comment": "Silence",
"status": {
"state": "active"
}
}
|
v2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| {
"matchers": [
{
"name": "service",
"value": "rancher",
"isRegex": false,
"isEqual": true
},
{
"name": "alertname",
"value": "TargetDown",
"isRegex": false,
"isEqual": true
}
],
"startsAt": "2022-04-29T10:11:35.656Z",
"endsAt": "2022-04-29T12:11:35.656Z",
"createdBy": "Casey Cui",
"comment": "配置错误导致的误报",
"id": null
}
|
Concrete implementation
curl implementation
📝 Notes:
Take the v1 API as an example
As follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| curl https://alertmanager.e-whisper.com/api/v1/silences -d '{
"matchers": [
{
"name": "alername1",
"value": ".*",
"isRegex": true
}
],
"startsAt": "2022-04-29T22:12:33.533Z",
"endsAt": "2022-04-29T23:11:44.603Z",
"createdBy": "api",
"comment": "Silence",
"status": {
"state": "active"
}
}'
|
📝Notes:
In a K8S cluster, the address is typically:http://alertmanager:9093
Python implementation
As follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
import requests
import socket
import datetime
import time
res = requests.post("http://alertmanager:9093/api/v2/silences", json={
"matchers": [
{"name": "job", "value": "myjob", "isRegex": False},
{"name": "instance", "value": "{}:1234".format(socket.gethostname()), "isRegex": False},
],
"startsAt": datetime.datetime.utcfromtimestamp(time.time()).isoformat(),
"endsAt": datetime.datetime.utcfromtimestamp(time.time() + 4*3600).isoformat(),
"comment": "Backups on {}".format(socket.gethostname()),
"createdBy": "My backup script",
},
)
res.raise_for_status()
silenceId = res.json()["silenceID"]
|
Script to remove silence:
1
2
| res = requests.delete("http://alertmanager:9093/api/v2/silence/{}".format(silenceId))
res.raise_for_status()
|
EOF