This blog post discusses the User Mode Helper (UMH) plugin. Read the overview blog post first for a more general introduction to drbd-reactor.
Motivation
Experienced DRBD® users will know that DRBD (the part in the Linux kernel) can call user-defined helper scripts if several important events happen, like before a resource becomes a sync target. Such a script would then create a snapshot of the backing device. Over time, users found creative new ideas for executing scripts. Still, these ideas can be user/customer-specific, and adding tens or hundreds of new event categories to the kernel module would not make sense. So with the help of this plugin and a simple DSL (Domain Specific Language), users can execute scripts if an event they define a filter for occurs. For example, they could execute a script that sends a slack message whenever a DRBD resource loses its connections. Another disadvantage we can currently solve is that the kernel module executes such helpers stored on the host file system. This does not fit a containerized world with read-only host file systems, while drbd-reactor can be executed in a container containing all the required scripts. Furthermore, there is no need for any kernel-space to user-space transitions as all components are executed in user-space only.
Without going into too much detail about the UMH filter language, one can define filters, and if it matches, a script gets executed. Information gets passed to the executed script via environment variables. Depending on the match type, the UMH plugin passes along a variety of predefined variables. To give some examples, the matched resource name is passed as $DRBD_RES_NAME, and state changes like the role are passed via OLD and NEW variables accordingly (e.g., DRBD_OLD_ROLE/DRBD_NEW_ROLE).
Example
Let’s look at an example where the UMH plugin can help us solve a concrete problem. DRBD has its own notion of quorum, and one can configure how the DRBD block device should behave when quorum is lost. There are two possibilities for on-no-quorum:
- io-error: If the resource lost quorum return an io-error (i.e., system calls fail with errno set to EIO).
- suspend-io: If the resource lost quorum suspend I/O. The user visible behavior is that I/O “hangs.” IO can be resumed if quorum is gained.
There are advantages and disadvantages to both behaviors. Configuring io-error usually is the faster – and often only – way to propagate a quorum loss to the application that uses the actual DRBD device. Most of the software or services that exist can deal with failed I/O operations one way or the other, even if they just fail and exit. The disadvantage is that such I/O errors occur immediately after quorum is lost, and there is no easy way back. For example, if such a DRBD device is used as storage for VMs, the VM might want to write data, get an I/O error, and the file system in the VM immediately remounts as read-only. And this can happen easier than expected, for example, some hiccup in the non-redundant network used for DRBD traffic. Your replication network is redundant, right?!
A more benign way is setting suspend-io; as in the above-described scenario, the VM would hang, and when quorum is re-established, things continue as nothing happened. The disadvantage is that it might be the case that quorum is never established again, or at least not for a very long time. Then usually, applications just hang forever.
So it somehow would be nice to get the best from both worlds. Start with suspending I/O for some time, and if things go back to normal, then everything is fine; if not, then reconfigure the resource to throw I/O errors. We can do that by placing a helper script to /usr/local/bin that gets triggered by the UMH plugin on quorum loss/gain that then does the right thing. Let’s assume the DRBD resource is called drbd-data. First place this script to /usr/local/bin/on-no-quorum-io-error.sh:
#!/bin/bash
: "${TIMEOUT:=60}"
die() {
>&2 echo "$1"
exit 1
}
set_io_error() {
drbdsetup resource-options --on-no-quorum io-error --on-no-data io-error "${DRBD_RES_NAME}"
}
set_suspend_io() {
drbdsetup resource-options --on-no-quorum suspend-io --on-no-data suspend-io "${DRBD_RES_NAME}"
}
is_in_use() {
role=$(drbdadm role "${DRBD_RES_NAME}")
[[ $role == Primary ]]
}
has_quorum() {
drbdsetup events2 --now "${DRBD_RES_NAME}" | grep -q "quorum:yes"
}
case "${DRBD_NEW_QUORUM}" in
true)
echo "Got quorum"
set_suspend_io
;;
false)
echo "Lost quorum"
echo "Checking if in use"
is_in_use || { set_io_error; exit 0; }
echo "Sleeping ${TIMEOUT}..."
sleep "${TIMEOUT}"
echo "Checking for quorum"
has_quorum && exit 0
echo "Setting io-error"
set_io_error
;;
*) die "Quorum state '${DRBD_NEW_QUORUM}' can not happen" ;;
esacThe script is relatively simple: It checks the current state of the quorum (DRBD_NEW_QUORUM), and whenever that is true, it sets the resource’s quorum option to the benign suspend-io. If quorum is lost, then the script checks if the device is even in use, and if so, it gives the resource some time. Finally, if the resource still does not have quorum, it sets the policy to io-error.
The final step is to configure a UMH plugin that calls the script accordingly:
$ drbd-reactorctl edit --type umh drbd-data
[[umh]]
id = "drbd-data"
[[umh.device]]
resource-name = "drbd-data"
old.quorum = true
new.quorum = false
command = "/usr/local/bin/on-no-quorum-io-error.sh"
name = "quorum lost"
[umh.device.env]
TIMEOUT = "30"
[[umh.device]]
resource-name = "drbd-data"
old.quorum = false
new.quorum = true
command = "/usr/local/bin/on-no-quorum-io-error.sh"
name = "quorum gained"Conclusion
The umh plugin provides a more modern and flexible version of the existing user-mode helpers in the DRBD kernel module. It moves all parts from the kernel to user-space, allowing easier container setups and read-only host file systems (i.e., “container distributions”). In addition, there is a broad spectrum of DRBD state changes that can be matched with a simple DSL. That should allow users to handle basically all events that can happen in DRBD according to their specific needs without introducing new handlers in the kernel module for every scenario.
