DRBD Reactor – Prometheus

DRBD Reactor Prometheus

This blog post discusses the ‘Prometheus’ plugin, which implements a prometheus.io compatible metrics endpoint. But, first, for a more general introduction to ‘drbd-reactor,’ please read the overview blog post. 

Motivation 

Proper monitoring of DRBD® resources has been a messy task for too long. There are various monitoring solutions, and as DRBD is widely used, there are plugins/agents/importers that can monitor DRBD resources. Many of these plugins I’ve seen in the wild use creative shell scripts to get information out of ‘/proc/drbd.’ Using ‘/proc/drbd’ has multiple disadvantages: 

  • while we don’t break it intentionally, there is no stable ‘API’ (e.g., it changed from DRBD 8.4 to 9.0). 
  • it was meant to be consumed by humans, not programs. One could consider it a proprietary format that mainly consists of colon-separated key/value pairs. 

There are alternatives to ‘/proc/drbd’ that are less popular. One of these is the JSON output of ‘drbdsetup status’ introduced some time ago for DRBD 9. There are also ‘drbdsetup events2’ that produce a stream of DRBD events. While the format for ‘events2’ is basically simple key/value pairs per line for several event types, it still needs some work to parse it and correlate its output to DRBD resources, their volumes, connections, peers, etc. In fact, DRBD Reactor uses ‘events2’ as its DRBD event source. 

Given these existing solutions where some need custom parsers for ‘proprietary’ text formats, we saw room for improvement for input that does not even have a defined stable ‘API.’ 

Implementation 

We first had to decide which monitoring solution we wanted to support first. That was relatively simple, as we saw that Prometheus is the standard nowadays, and there have been requests for Prometheus support from customers and the FLOSS community. Even if there are users who don’t use Prometheus/Grafana directly, there is a plugin for their favourite monitoring solution that can consume the Prometheus metrics format. Even if not, it is then easier and more reasonable to write such a plugin for their platform that consumes the easy-to-handle Prometheus format than to start ‘grep’ing around in ‘/proc/drbd.’ 

The plugin is simple to use as there is not much need for configuration: 

”’

$ drbd-reactorctl edit –type prometheus prometheus 

[[prometheus]] 

id = “prometheus” 

enums = true 

# address = “0.0.0.0:9942” 

”’

The shown command opens an editor with the displayed content. Save it, and you are done. After saving the file and DRBD Reactor picking up the changes, one can query the collected metrics via ‘curl -s localhost:9942/metrics’ and configure Prometheus/Grafana as usual. The plugin exposes various DRBD states and statistics like DRBD resource states, volume states, connection states, out-of-sync statistics, etc. For a complete list of metrics, please check the plugin’s documentation. 

The DRBD Reactor GitHub repository contains an example Grafana dashboard that might be a good starting point for some users. The dashboard looks as follows:

Conclusion 

The Prometheus plugin is easy to set up and use. It avoids ‘grep’ing around in ‘/proc/drbd’ or ‘drbdsetup’ output and provides a standardized output format (i.e., the Prometheus metrics format). This output can then be used by various monitoring and graphing solutions. Adapters for different monitoring solutions usually should not be needed as most of them can already consume Prometheus metrics or have a generic Prometheus metric import plugin. Even if that should not be the case, writing such a plugin should be trivial. 

Roland Kammerer

Roland Kammerer

Roland Kammerer studied technical computer science at the Vienna University of Technology and graduated with distinction. Currently, he is a PhD candidate with a research focus on time-triggered realtime-systems and works for LINBIT in the DRBD development team.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.