Last month I discussed how AI enabled me to bring my dream of a simpler way to deploy LINBIT software to a reality. Now after much testing and deliberation, I am pleased to announce our LINBIT deployment tool, or “Nightfury” for those on the inside who remember the project’s first working title. Although the vision of the deployment tool is mine, the behind-the-scenes work credit goes to the LINBIT team: namely Matt and Ryan who developed the Ansible playbooks that it calls on to deploy various scenarios.
The goal of the deployment tool is to allow you define what you want without having to know how to get there. It substantially lowers the bar for entry and reduces the time required to get to testing and deploying our software. Users will now be able to easily and rapidly deploy highly available SDS clusters with only having basic knowledge of the target infrastructure. No longer will you need to spend hours learning and reading documentation, although our documentation is some of the best out there and it is good reading in your spare time. The deployment tool as of today supports deploying basic LINSTOR clusters, either with no services, NFS, NVMEoF, and iSCSI as standalone services, or a mix of services that you can choose.
Today the deployment tool supports the following target operating systems: Debian 13+, Ubuntu 24.04+, RHEL 9 and 10 (and derivatives). The deployment tool will run on Apple Silicon, Linux, or Windows. While you can run the it from a Windows system, Windows isn’t supported as a target yet, but a little birdy told me it was bound to happen sooner or later. If there is interest please let us know because priorities can change.
Testing this before release has been the biggest time consumer of it all. Trying to hit every supported OS, deployment mix, and verifying with five different post-install failover scenarios, takes time and thought. Internally we’ve had people using SSH jump hosts, local virtual machines, real hardware, cloud providers, and others, all with their own various caveats. It generally takes me about 12 hours to run the full testing matrix on the hardware I have.
I think the most interesting part of the testing though is the post-install testing. That is, the “Will this hold up in the real world?” question. To answer that question I gathered the various test scenarios our engineers and developers said should happen on all clusters. The tests are as follows:
1. Planned eviction: Running drbd-reactorctl evict –force on the active Primary. This test ensures the resource migrates gracefully and a peer successfully takes over the VIP within the eviction delay.
2. Process kill: Running kill -9 drbd-reactor and systemctl stop drbd-services@<res>.target on the Primary. This test looks to confirm that DRBD demotes properly, a secondary node promotes, and the original node re-joins the cluster cleanly after a restart.
3. Force reboot: Starting a poweroff -f followed by a deliberate 60-second delay to mimic actual hardware POST. The timing is vital here: VMs often reboot faster than DRBD can detect a dead peer, and without this delay, the original node might reclaim Primary before failover even triggers, which can mask real-world bugs.
4. Network partition: Using iptables DROP on every peer link. This verifies that quorum-based fencing correctly isolates the partitioned node and that the surviving majority elects a new Primary. Recovery is then tested via a simple iptables -F.
5. Combined / cascading: Testing a scenario where the controller and an NFS gateway Primary fail simultaneously. The test then kills a second NFS Primary to deliberately break quorum and force the VIP down. The test passes when the nodes return and the cluster auto-recovers the VIP without any manual intervention.
Across every one of these scenarios, the tests verify that the VIP actually responds from a new node. Simply seeing a green systemd unit isn’t enough to call the test a pass.
And there you have it, a small glimpse into what I’ve been doing on the side around my other job duties. I’d love to have you test the software and provide feedback. It being the initial release I’m sure there is something I’ve missed (but I hope not!) You can download the deployment tool here. A disclaimer though: You will need a LINBIT evaluation account to get going. This is free and just gives the LINBIT team a way to stay in touch for feedback or support.
Also, I should mention that the LINBIT Ansible collections are the magic behind how the deployment tool does what it does. You don’t need to know about this to use the it, after all, that is the point of it. But if you’re interested, the LINBIT Ansible collection is a powerful set of tools for deploying and administering LINBIT software.
In the next newsletter I think I’ll discuss how the deployment tool sets up the system for being managed by the GUI. That was a fun path to go down and gives me something else to work on after this.
