Scope/Description

This article will show how to add custom alerting rules to Prometheus.

Prerequisites

• Monitoring, alerting and metric stack deployed
• Custom alerts to be added to Prometheus
• Access to the admin node if deploying on a Ceph Cluster

Steps

• From the command line, run podman ps to list running containers, note the container ID of Prometheus.

[root@octosd1 ~]# podman ps

• Edit the rules file by running vim /etc/prometheus/alerting/alertfilename.yml replacing alertfilename with your proper alerts file name.

For Ceph users:

[root@octosd1 ~]# vim /etc/prometheus/alerting/ceph_dashboard.yml

For Single Server users:

[root@octosd1 ~]# vim /etc/prometheus/alerting/zfs-alerts.yml

• Add your desired alerts to the list and Save.

Verification

• Verify the syntax and formatting of your added alerts by running podman exec ID promtool check rules /etc/prometheus/alerting/alertfilename.yml replacing ID with the Prometheus container ID, and alertfilename.yml with your proper alerts file name.

[root@octosd1 ~]# podman exec ID promtool check rules /etc/prometheus/alerting/alertfilename.yml 

• If it returns with SUCCESS. Then you have successfully added your alerts.

Troubleshooting

• If you do not return a SUCCESS when verifying, compare the syntax of your added alerts with the alerts already existing. Make sure the spacing and syntax is identical. If there is even one too many spaces somewhere, your alert will not work properly.