This blog post discusses the ‘Prometheus’ plugin, which implements a prometheus.io compatible metrics endpoint. But, first, for a more general introduction to ‘drbd-reactor,’ please read the overview blog post.
Proper monitoring of DRBD resources has been a messy task for too long. There are various monitoring solutions, and as DRBD is widely used, there are plugins/agents/importers that can monitor DRBD resources. Many of these plugins I’ve seen in the wild use creative shell scripts to get information out of ‘/proc/drbd.’ Using ‘/proc/drbd’ has multiple disadvantages:
There are alternatives to ‘/proc/drbd’ that are less popular. One of these is the JSON output of ‘drbdsetup status’ introduced some time ago for DRBD 9. There are also ‘drbdsetup events2’ that produce a stream of DRBD events. While the format for ‘events2’ is basically simple key/value pairs per line for several event types, it still needs some work to parse it and correlate its output to DRBD resources, their volumes, connections, peers, etc. In fact, DRBD Reactor uses ‘events2’ as its DRBD event source.
Given these existing solutions where some need custom parsers for ‘proprietary’ text formats, we saw room for improvement for input that does not even have a defined stable ‘API.’
We first had to decide which monitoring solution we wanted to support first. That was relatively simple, as we saw that Prometheus is the standard nowadays, and there have been requests for Prometheus support from customers and the FLOSS community. Even if there are users who don’t use Prometheus/Grafana directly, there is a plugin for their favourite monitoring solution that can consume the Prometheus metrics format. Even if not, it is then easier and more reasonable to write such a plugin for their platform that consumes the easy-to-handle Prometheus format than to start ‘grep’ing around in ‘/proc/drbd.’
The plugin is simple to use as there is not much need for configuration:
$ drbd-reactorctl edit –type prometheus prometheus
id = “prometheus”
enums = true
# address = “0.0.0.0:9942”
The shown command opens an editor with the displayed content. Save it, and you are done. After saving the file and DRBD Reactor picking up the changes, one can query the collected metrics via ‘curl -s localhost:9942/metrics’ and configure Prometheus/Grafana as usual. The plugin exposes various DRBD states and statistics like DRBD resource states, volume states, connection states, out-of-sync statistics, etc. For a complete list of metrics, please check the plugin’s documentation.
The DRBD Reactor GitHub repository contains an example Grafana dashboard that might be a good starting point for some users. The dashboard looks as follows:
The Prometheus plugin is easy to set up and use. It avoids ‘grep’ing around in ‘/proc/drbd’ or ‘drbdsetup’ output and provides a standardized output format (i.e., the Prometheus metrics format). This output can then be used by various monitoring and graphing solutions. Adapters for different monitoring solutions usually should not be needed as most of them can already consume Prometheus metrics or have a generic Prometheus metric import plugin. Even if that should not be the case, writing such a plugin should be trivial.
Share this post
LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.
By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.
Here’s the new LINSTOR Operator 1.8.2.
This release updates some components, notably the latest LINSTOR 1.18.2.
– LINSTOR 1.18.2
– LINSTOR CSI 0.19.1: fixes issues around cluster with quorum disabled
Access the update here: https://bit.ly/3PECaYF