Zabbix Alertmanager
Recently, we released the Zabbix Alertmanager integration to the open source, which can be downloaded from the GitHub page. And, in this post, we are going to dive deeper into how the Zabbix Alertmanager integration works.
When we first began working on this integration, we looked at a similar project made by
zal prov
command, which converts Prometheus Alerting rules into Zabbix Triggers.zal send
command, which listens for Alert requests from Alertmanager and sends them to Zabbix.
Alert provisioning
zal prov
For our customers, we recommend Alert provisioning as a part of a Gitlab Continuous Integration job. So, we decided to store all Prometheus alerting configuration in one git repository. Developers create a change in alerting rules via Pull Request in Git, then promtool check rules
.gitlab-ci.yml
:
stages:
check
provision
check-alerts:
stage: check
image:
name: prom/prometheus:v2.8.0
entrypoint: ["/bin/sh", "-c"]
script:
- promtool check rules /.yml
provision-rules:
stage: provision
image:
name: devopyio/zabbix-alertmanager:v1.1.1
entrypoint: ["/bin/sh", "-c"]
script:
- zal prov --log.level=info --config-path=zal-config.yaml
--url=http://ZABBIX_URL/api_jsonrpc.php
--prometheus-url=http://PROMETHEUS_URL
only:
- master
Getting started
In order to run zal prov
, you will need to set up Zabbix User. This Zabbix User has to have access to Zabbix API. Also this user requires elevated permissions to update Hosts, create Host Items, Triggers & Zabbix Applications. You can read more about Zabbix API & user permissions in Zabbix API manual.
When you first try it out, we suggest that you manually create a Host Group along with some empty Hosts. After that, create a Zabbix user, and allow this user to access that Host Group and enable Zabbix API access. Be sure to make note of your configuration, though, as you will need to provide these values zal prov
--user
--password
--url
flags ZABBIX_USER
ZABBIX_PASSWORD
ZABBIX_URL
environment variables. We recommend setting user credential data via environment variables, in order to keep them secret.
Configuring Hosts
After empty hosts are created and a user is set up, you need to specify Host configurations. It’s a simple YAML configuration file. Take a look at this example:
- name: infrahost
hostGroups: INFRA
tag: prometheus
deploymentStatus: 0
itemDefaultApplication: prometheus
itemDefaultHistory: 5d
itemDefaultTrends: 5d
itemDefaultTrapperHosts: 0.0.0.0/0
triggerTags:
INFRA: ""
alertsDir: ./infra
- name: webhost
hostGroups: WEBTEAM
tag: prometheus
deploymentStatus: 0
itemDefaultApplication: prometheus
itemDefaultHistory: 5d
itemDefaultTrends: 5d
itemDefaultTrapperHosts: 0.0.0.0/0
triggerTags:
WEBT: ""
alertsDir: ./web
In this example, we create two Zabbix hosts. One is infrahost
INFRA
prometheus
tag to this infrahost
we will provision Zabbix triggers from Prometheus alerts in the./infra
directory. Lastly, we add INFRA
Similarly, we do the same for any host webhost
WEBTEAM
./web
infrahost
webhost
Zabbix host.
Alerting configuration
In alertsDir
we expect Prometheus Alerting rules to be saved in files ending with .yml
or .yaml
extensions. There are some special rules when creating Alerts in Zabbix, which provide a more native Zabbix experience for Prometheus Alerts.
Here are the rules:
- If the Prometheus URL is configured, we
setup a Trigger URL to link to Prometheus Query (Only if URL is shorter than 255 symbols, as Zabbix doesn’t support longer URLs). Trigger’s Comment
field is set fromAlert’s summary
, message
or description
annotation .Trigger’s Severity
is configuredvia severity
label of information
, warning
, average
, high
, critical
values.- If Alerting rule has special
zabbix_trigger_nodata
annotation, we set up a special Zabbixnodata trigger expression. Annotation’s value must be a number, which is the evaluation period in seconds.
Use zal prov
--config-path
--prometheus-url
zal --help
zal prov --help
Alert sending
Once Alert provisioning has successfully completed, you can start sending alerts Zabbix. zal send
command listens for alerts from Alertmanager, via webhook receiver and sends them into Zabbix, via Zabbix Sender Protocol. You can read more about the protocol and how it works in th
In order to zal send
--zabbix-addr
10051
port. You then need to configure--addr
, which is addressed to listen for Alertmanager’s Webhook requests (default is 0.0.0.0:9095). Also, you will need to --hosts-path
zal send
infrahost
for infrastructure alerts, and webhost
for web developer alerts. This would give you two mappings:
# Resolver name to zabbix host mapping
infra: infrahost
web: webhost
The first part of this configuration is actually infra
infrahost
web
webhost
global:
route:
group_by: ['alertname', 'team']
group_wait: 30s
group_interval: 2m
repeat_interval: 3m
receiver: infra
routes:
- receiver: web
match_re:
team: web
- receiver: infra
match_re:
team: infra
receivers:
- name: 'infra'
webhook_configs:
- url: http://ZAL_SENDER_ADDR/alerts
send_resolved: true
- name: 'web'
webhook_configs:
- url: http://ZAL_SENDER_ADDR/alerts
send_resolved: true
This configuration team: web
web
team: infra
infra
webhook_configs
ZAL_SENDER_ADDR
is the address of zal send
, which we configured via --addr
flag.
If we fail to correctly route an alert (in the case where alert doesn’t have a team label), it will end up infra
--default-host
Conclusion
These are the main things you need to know in order to successfully run your Zabbix
Need help integrating Prometheus with Zabbix? Contact us.