Zabbix Alertmanager
Recently, we released the Zabbix Alertmanager integration to the open source, which can be downloaded from the GitHub page. And, in this post, we are going to dive deeper into how the Zabbix Alertmanager integration works.
When we first began working on this integration, we looked at a similar project made by Gmauleon. It’s a really great project that we took a lot of inspiration from, but we quickly realized that we needed to make some major changes. Our project is written in Go and released as a standalone binary, which we called zal. The integration consists of 2 separate commands:
zal prov
command, which converts Prometheus Alerting rules into Zabbix Triggers.zal send
command, which listens for Alert requests from Alertmanager and sends them to Zabbix.
Alert provisioning
The zal prov
command is used to create Zabbix Triggers from Prometheus Alerting rule definitions. It’s a simple executable binary. Almost like a shell script, it runs to completion and returns exits, with status code 0 upon success. Otherwise, it will fail and print an error message. This feature opens up many different deployment options. You can deploy alert provisioning using Cron Job, which periodically checks configuration and creates trigger. CI Job, which would create triggers on alert configuration change or a regular bash script.
For our customers, we recommend Alert provisioning as a part of a Gitlab Continuous Integration job. So, we decided to store all Prometheus alerting configuration in one git repository. Developers create a change in alerting rules via Pull Request in Git, then CI job runs promtool check rules
command, which validates configuration. Once Pull Request is merged, we will automatically provision alerting rules into Zabbix. Here is an example of .gitlab-ci.yml
:
stages:
check
provision
check-alerts:
stage: check
image:
name: prom/prometheus:v2.8.0
entrypoint: ["/bin/sh", "-c"]
script:
- promtool check rules /.yml
provision-rules:
stage: provision
image:
name: devopyio/zabbix-alertmanager:v1.1.1
entrypoint: ["/bin/sh", "-c"]
script:
- zal prov --log.level=info --config-path=zal-config.yaml
--url=http://ZABBIX_URL/api_jsonrpc.php
--prometheus-url=http://PROMETHEUS_URL
only:
- master
Getting started
In order to run zal prov
, you will need to set up Zabbix User. This Zabbix User has to have access to Zabbix API. Also this user requires elevated permissions to update Hosts, create Host Items, Triggers & Zabbix Applications. You can read more about Zabbix API & user permissions in Zabbix API manual.
When you first try it out, we suggest that you manually create a Host Group along with some empty Hosts. After that, create a Zabbix user, and allow this user to access that Host Group and enable Zabbix API access. Be sure to make note of your configuration, though, as you will need to provide these values to zal prov
via --user
, --password
, --url
flags or ZABBIX_USER
, ZABBIX_PASSWORD
, ZABBIX_URL
environment variables. We recommend setting user credential data via environment variables, in order to keep them secret.
Configuring Hosts
After empty hosts are created and a user is set up, you need to specify Host configurations. It’s a simple YAML configuration file. Take a look at this example:
- name: infrahost
hostGroups: INFRA
tag: prometheus
deploymentStatus: 0
itemDefaultApplication: prometheus
itemDefaultHistory: 5d
itemDefaultTrends: 5d
itemDefaultTrapperHosts: 0.0.0.0/0
triggerTags:
INFRA: ""
alertsDir: ./infra
- name: webhost
hostGroups: WEBTEAM
tag: prometheus
deploymentStatus: 0
itemDefaultApplication: prometheus
itemDefaultHistory: 5d
itemDefaultTrends: 5d
itemDefaultTrapperHosts: 0.0.0.0/0
triggerTags:
WEBT: ""
alertsDir: ./web
In this example, we create two Zabbix hosts. One is named infrahost
and placed in the host group,INFRA
. Additionally, we add a prometheus
tag to this host, and store only 5 days worth of history. This configuration will be shown in Zabbix web UI. Forinfrahost
we will provision Zabbix triggers from Prometheus alerts in the./infra
directory. Lastly, we add the INFRA
tag on those triggers.
Similarly, we do the same for any host named webhost
in the host group WEBTEAM
, and provision alerts from the ./web
directory. Multiple hosts and multiple alert directories allows us to separate teams as well as their alerts. In this case, we have an Infrastructure team, which will see their alerts in infrahost
host, along with a Web developer team, which will get their alerts in the webhost
Zabbix host.
Alerting configuration
In alertsDir
we expect Prometheus Alerting rules to be saved in files ending with .yml
or .yaml
extensions. There are some special rules when creating Alerts in Zabbix, which provide a more native Zabbix experience for Prometheus Alerts.
Here are the rules:
- If the Prometheus URL is configured, we setup a Trigger URL to link to Prometheus Query (Only if URL is shorter than 255 symbols, as Zabbix doesn’t support longer URLs).
- Trigger’s
Comment
field is set from Alert’s summary
, message
or description
annotation. - Trigger’s
Severity
is configured via severity
label. Severity label can have one of information
, warning
, average
, high
, critical
values. - If Alerting rule has special
zabbix_trigger_nodata
annotation, we set up a special Zabbix nodata trigger expression. Annotation’s value must be a number, which is the evaluation period in seconds.
Use configuration file via zal prov
--config-path
flag. In order to make triggers, link to Prometheus and add the --prometheus-url
flag. You can get more information by executing zal --help
and zal prov --help
commands.
Alert sending
Once Alert provisioning has successfully completed, you can start sending alerts Zabbix. zal send
command listens for alerts from Alertmanager, via webhook receiver and sends them into Zabbix, via Zabbix Sender Protocol. You can read more about the protocol and how it works in th Zabbix Trapper items section.
In order to run zal send
, you will need to set --zabbix-addr
to point to the Zabbix server trapper port. By default it is listening on Zabbix server’s 10051
port. You then need to configure--addr
, which is addressed to listen for Alertmanager’s Webhook requests (default is 0.0.0.0:9095). Also, you will need to provide --hosts-path
, which is pointing to the zal send
host configuration file.
Hosts configuration file is used to route alerts to correct Zabbix hosts. Let’s say you have two hosts,infrahost
for infrastructure alerts, and webhost
for web developer alerts. This would give you two mappings:
# Resolver name to zabbix host mapping
infra: infrahost
web: webhost
The first part of this configuration is actually alertmanager’s receiver name. In this example, alerts coming from Alertmanager’s infra
receiver will go to infrahost
Zabbix host, and web
receiver’s alerts will go to webhost
host. This configuration needs to be inline with Alertmanager’s configuration. Let’s take a look at this Alertmanager’s configuration example:
global:
route:
group_by: ['alertname', 'team']
group_wait: 30s
group_interval: 2m
repeat_interval: 3m
receiver: infra
routes:
- receiver: web
match_re:
team: web
- receiver: infra
match_re:
team: infra
receivers:
- name: 'infra'
webhook_configs:
- url: http://ZAL_SENDER_ADDR/alerts
send_resolved: true
- name: 'web'
webhook_configs:
- url: http://ZAL_SENDER_ADDR/alerts
send_resolved: true
This configuration routes alerts with label team: web
to web
receiver, and team: infra
to infra
receiver. Note that receivers must be configured via webhook_configs
and, for each team, there must be a separate receiver configuration. In this example, we have one receiver for the Web developer team and one receiver for the Infrastructure team. ZAL_SENDER_ADDR
is the address of zal send
, which we configured via --addr
flag.
If we fail to correctly route an alert (in the case where alert doesn’t have a team label), it will end up in infra
receiver. If we forget to configure the Hosts configuration file, it will default to the value specified in --default-host
flag.
Conclusion
These are the main things you need to know in order to successfully run your Zabbix Alertmanager integration. In the next post, we will take a look at deployment considerations of Zal sender, and see how we can ensure the whole system runs reliably.
Need help integrating Prometheus with Zabbix? Contact us.