Perhaps there is a feeling that you might be making the world a better place by choosing to use open source software rather than other alternatives. Maybe this positive feeling spawns an action. If you get a benefit from using a particular open source project, sometimes you might support that project by making a donation to a virtual “tip jar” that you might find on a project page. Sometimes you might give back to a company that backs an open source project by buying support (ahem) or feature development specific to your use case (cough)1. Other times, you might contribute your time to an open source project to improve it, either by reporting a bug, making a feature request, or even contributing code or documentation. Perhaps while doing your own development work, you identify a problem for which there is no existing solution and so you develop one, and make it open source for others to use freely and benefit from.
In the course of developing its own open source software, LINBIT® uses and contributes to open source projects in all of these ways. While not dismissing the other ways of contributing, developing new solutions and making them open source might be the most interesting to read about and is the focus of this article.
⚠️ WARNING: This article might contain statements that have the appearance of tooting-our-own-horn. While we at LINBIT are certainly proud of our developments, the article’s main intent is to bring attention to some of the lesser known incidental work that LINBIT developers are doing. This is work that benefits the open source community at large and that others might find useful in their own work.
Creating innovations out of necessity
Innovation is often based on need. Frequently, there isn’t a suitable existing solution to a problem so an innovator creates one. LINBIT has a long history of innovating in the fields of software-defined storage, high availability, and disaster recovery.
Indeed, LINBIT CEO Philipp Reisner created DRBD®, LINBIT’s flagship software for high availability (HA), over 20 years ago because there was no software that existed that replicated block storage over a network in the Linux landscape.
In that spirit, LINBIT’s continued development of DRBD, along with its development of software such as LINSTOR®, DRBD Reactor, WinDRBD, LINBIT VSAN, and others, has lead to new open source solutions. A few of these lesser known innovations are highlighted below.
An AgentX library for Rust
While developing an SNMP notifications feature for DRBD Reactor, LINBIT developer Roland Kammerer wrote a new AgentX library for the Rust language and made it open source. DRBD Reactor is a modular utility, written in Rust, that handles all kinds of DRBD events. Because of its modular nature, it has many use cases, including as an alternative-to-Pacemaker cluster manager by virtue of its promoter plugin, a Prometheus monitoring endpoint for DRBD metrics, and now as an AgentX events processor which brings SNMP functionality to DRBD events.
Before Kammerer’s coding, there was no AgentX library for Rust. Now, developers working in other Rust-coded projects that might want to implement a feature that needs to use the AgentX protocol have a ready-to-use library as a building block, thanks to Kammerer’s efforts. By virtue of the library having an open source license, developers can use the library freely and even modify it as needed to suit their needs.
Fixing an issue in a Rust library
Also on the Rust language front, LINBIT developer Joel Colledge used the Berkeley Packet Filter (BPF) to investigate a DRBD bug, making use of the libbpf-rs library. Colledge encountered and reported an issue in this library related to its handling of Linux block layer structures. A libbpf-rs project maintainer merged Colledge’s fix into the library’s code in October 2024.
A Linux tool for generating Microsoft Windows security catalog files
When creating the WinDRBD 1.2 development branch, LINBIT developer Johannes Khoshnazar-Thoma reengineered the build process for WinDRBD, the block storage replication software for HA on Microsoft Windows systems. Now you can build WinDRBD from its source code, entirely in Linux and by using open source tools exclusively.
Before this, building WinDRBD required a hybrid Linux and Windows environment. It also required that you use the Microsoft inf2cat.exe proprietary utility for generating security signatures for the installation process. To have more flexibility in the build process, Khoshnazar-Thoma wrote the generate-cat-file tool for Linux and made it open source. This tool allows you to create Microsoft digitally-signed (.cat) files which are required to add 64 bit Windows drivers, such as the WinDRBD driver, to the Windows driver store when you install them.
Khoshnazar-Thoma says, “To date as of my knowledge there is no open source implementation that does not depend on the wintrust.dll library (which would require either a Windows build machine or Wine which unfortunately does not implement this particular feature). So I wrote a small C program from scratch that creates the .cat file.”
Creating an open source NETIO.SYS-compatible driver
Another open source innovation that Khoshnazar-Thoma developed is a networking driver for the ReactOS operating system, an open source operating system for running Microsoft Windows applications on. Khoshnazar-Thoma wrote an open source driver that is compatible with Microsoft’s closed source NETIO.SYS driver that provides the means for other drivers, such as WinDRBD, to interface with the Microsoft Windows networking stack. By using this driver, one can do things such as:
- Create Windows networking sockets
- Bind a socket to an address
- Send UDP packets
- Make TCP connections
- Send and receive TCP packets
- Close sockets
Khoshnazar-Thoma reports that the driver is working well for its purposes for use with WinDRBD and is working with ReactOS project maintainers to merge this more modern and convenient networking driver into the upstream project. If merged, it will be one of the first Windows NT 6.x compatible drivers implemented within ReactOS. You can read more about Khoshnazar-Thoma’s work with ReactOS as a target development platform in another LINBIT blog, “Development Work on the WinDRBD 1.2 Branch for Highly Available Windows Services”.
Tools for working With virtual machines and automated testing environments
Two more examples of needs-based innovation at LINBIT are the open source Virter and vmshed tools for working with libvirt-based virtual machines (VMs). These tools are especially useful when you are using VMs for development and automated testing.
You can use Virter to quickly and simply start, manage, and clone cloud-init based VMs. While there were other VM management tools that existed before Virter, none of them quite fulfilled the development and automated testing demands that LINBIT software development needed, such as:
- Ease of maintainability (Libvirt is a single binary without dependencies such as Ruby gems)
- Fast and efficient image provisioning based on chains of backing images which allows you to easily create cloned VMs from a single VM image
- Usage of standard cloud images that are “supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.”2
You can use the Virter partner tool, vmshed, to schedule tests in VMs. By using vmshed, you can configure and automate preparing VM images, starting clusters of VMs, and running tests within VMs. This is particularly helpful for use in a CI environment. LINBIT projects use the GitLab CI. At LINBIT, Virter and vmshed combined are used to start around 10000 virtual machines every day as part of the LINBIT automated testing regimen.
Similar to the other examples here, Virter and vmshed are also open source and you can freely use them in your own projects and environments. You can find more details about these tools at their GitHub pages: Virter, and vmshed. In particular, the comparison section of the Virter project page can help you determine if Virter is a good fit for you when compared to other tools such as Vagrant, Multipass, virt-install, and others.
Helm charts for storage snapshots in Kubernetes
While doing development work for Piraeus, the LINBIT-backed open source upstream project for creating and managing persistent storage in Kubernetes, LINBIT developer Moritz Wanzenböck created some Helm charts that might have broader uses outside of just LINBIT developed solutions.
For example, the snapshot-controller and snapshot-validation-webhook are both generic offerings and can be used for a Kubernetes distribution that does not come bundled with either add-on. As their names suggest, these charts can add snapshot functionality for storage in any Kubernetes deployment, regardless of whether that storage is managed by Piraeus, LINBIT SDS, or another storage solution.
Various C++ data structures and algorithms
In developing the DRBDmon utility that is part of the drbd-utils suite, LINBIT developer Robert Altnoeder found the need to create various C++ data structures and algorithms to be building blocks that the larger project needed. DRBDmon is a management and monitoring utility for DRBD with an interactive text user interface. It can make working with DRBD resources, especially a large number of them, easier than working with the drbdadm CLI tool alone.
Altnoeder says that this and other similar projects “are side-effects of solving problems that I stumbled upon, where parts of the solution could be generalized and split out.”
Some of Altnoeder’s other open source projects include:
- UtfRecoder, a Java and Unicode utility for converting between UTF-8 byte and UTF-16 char types that can be used for input transformation by cryptographic functions
- SetId, a tool that allows you to run commands under an arbitrary user id, group id, and supplemental group list
linux-taintinfo, a utility that provides human-readable descriptions of the Linux kernel’s taint flags
Conclusion
While developing LINBIT software, LINBIT developers often face unique problems and create innovative solutions in response. Sometimes these solutions are split up and become their own sub-projects because the sub-projects might be useful outside the scope of the parent project.
Following LINBIT’s value-based commitment to open source and the community built around open source development, these innovations are often made open source themselves. By highlighting some of these innovations in this article, we at LINBIT hope you are either inspired in your own open source work or become interested in trying these innovations out for yourself, or both.
Resources
Some of the open source projects mentioned in this article are highlighted in developer presentations within the September 2023 LINBIT Community Meeting:
- The SNMP Protocol Subagent for DRBD Reactor that spawned the AgentX Library in Rust
- The WinDRBD 1.2 branch build process that lead to the creation of a Linux tool for generating Microsoft Windows security signatures
The following blog articles are detailed overviews of the Virter and vmshed tools:
Changelog
2024-12-06:
- Added summaries of
libbpf-rsRust library andNETIO.SYScompatible driver OSS contributions.
2023-10-04:
- Originally wrote post.
- We at LINBIT certainly appreciate if you are a customer and support the development of LINBIT software in either of these ways. If you are not already a LINBIT customer, you can contact our sales team to set up a free evaluation account.↩︎
- https://cloudinit.readthedocs.io/en/latest/↩︎
