Did you know  45Drives offers free  public and private  webinars ? Click here to learn more  & register! Build & Price

KB450407 – Increase Grafana Logging Period

You are here:

Scope/Description

  • This article covers increasing the Prometheus/Grafana logging period.

Prerequisites

  • A working Ceph cluster running Ubuntu 20.04 with Ceph Octopus.
  • Access to the cli of the admin/metrics node.

Steps

Extending Logging Period

  • Open /etc/systemd/system/prometheus.service in your text editor for choice and add the following line:
--storage.tsdb.retention.time=x
  • Your config should now look like the below screenshot, substituting 1y for your desired time.

  • Your retention period is now set though the actual log files are by default outputted to /var/lib/prometheus which pulls from the boot drives.
  • To avoid having data on your boot drives it’s recommended to mount an rbd image to /var/lib/prometheus for additional storage
  • We will also need to reload the systemd daemon to apply this change, you can do so with the commands below:
systemctl daemon-reload
systemctl restart prometheus

Providing Additional Log Storage

  • Go to the Ceph dashboard and create a new block image.

  • We will now need to map the newly created rbd image through the cli with the following command
rbd map rbd/promlogs --id admin --keyring /etc/ceph/ceph.client.admin.keyring
  • Then we mount a filesystem with the following command
mkfs.ext4 -m0 /dev/rbd/rbd/promlogs
  • Then mount this new rbd to /var/lib/prometheus. Note that any files inside /var/lib/prometheus will be removed in the process so make sure to back them up if you want them
mount /dev/rbd/rbd/promlogs /var/lib/prometheus/
  • To make these changes permanent add the following line to /etc/ceph/rbdmap
rbd/promlogs    id=admin,keyring=/etc/ceph/ceph.client.admin.keyring
  • We will also need to edit /etc/fstab with the following line
/dev/rbd/rbd/promlogs /var/lib/prometheus ext4 name=admin,_netdev 0 0

Verification

To verify that the mount has been successful we can run the ‘df’ command. The output should show the block device mounted to /var/lib/prometheus

Troubleshooting

  • Ensure that the owner user/group of the /var/lib/prometheus directory matches up to the UID specified in the Prometheus service file.
Was this article helpful?
Dislike 0
Views: 55
Unboxing Racking Storage Drives Cable Setup Power UPS Sizing Remote Access