Companion Tech Guide Updated using Community Feedback on 3/21/24
LINBIT®️ has been building and supporting Highly Available (HA) iSCSI clusters using DRBD®️ and Pacemaker for over a decade. We recently updated our HA iSCSI how-to guides for RHEL 8 and RHEL 9 to include instructions for both the Linux-IO (LIO) and SCSI Target Subsystem for Linux (SCST) iSCSI implementations for Linux, giving anyone looking to get started more options when designing a cluster to meet their needs and particular use case.
If you are looking to build such a cluster, you might be wondering which iSCSI implementation you should choose, or what the differences between the two implementations are. I will summarize what I believe are the most notable characteristics of each to perhaps help in the decision making process.
LIO and SCST in a Nutshell
LIO is a SCSI target framework integrated into the Linux kernel, providing a standardized interface for exporting storage devices over various SCSI transports, including iSCSI. It allows users to configure and manage iSCSI targets, initiators, and other storage-related parameters within the Linux operating system.
SCST is an SCSI target framework designed for Linux, developed outside the mainline kernel and supporting multiple SCSI protocols, including iSCSI. SCST provides a flexible platform for creating customized storage solutions and is known for its versatility and support for custom plugins.
Similarities Between LIO and SCST
Both LIO and SCST are kernel space implementations, which means they have direct access to the system’s hardware. This means device access is efficient and ensures low latency. This results in better performance than user space implementations which require context switches and system calls that can add latency to the system.
Importantly for clustering clients, both LIO and SCST support persistent reservations. Persistent reservations allow for the smooth hand off of LUNs during initiator failover and restrict concurrent access from multiple initiators which could introduce corruption.
Differences Between LIO and SCST
LIO is the implementation that was chosen to be included in the Linux kernel. This makes it easier to install than any other iSCSI implementation available for Linux. LIO also has a command line user interface, targetcli, that makes managing the target subsystem accessible to new users. These aspects make LIO more user friendly than SCST.
SCST is often found “under the hood” of Linux based storage appliances, likely because of its flexibility and ability to support custom plugins. SCST is a more general-purpose SCSI target framework, making it a good choice for purpose-built storage solutions. Configuration of SCST is done through configuration files. LINBIT’s VSAN product uses SCST as its iSCSI implementation since version 0.17.0, where versions prior used LIO.
The most notable difference in LINBIT’s experience having used both LIO and SCST in LINBIT VSAN is SCST’s ability to recover from certain failure scenarios that could occur under heavy load specifically when the iSCSI initiator is a VMware ESXi host. ESXi uses the vStorage APIs for Array Integration (VAAI) Atomic Test and Set (ATS) command for its datastore heartbeats. The VAAI ATS command has a shorter timeout than a regular SCSI command, so under load, and usually when attaching to targets over busy networks, the VAAI ATS commands will be the first to fail due to exceeding a timeout. This will cause ESXi to detach from the datastore. LIO does not allow for a second data session to be established. SCST does allow this, and so can work around the issue when it does crop up. Outside of this very specific scenario, we at LINBIT have had much success building and supporting HA iSCSI clusters using LIO and we’re still digging into this issue.
Conclusion
In conclusion, options are good and our iSCSI guide now gives you more of them! If you’re considering building an HA iSCSI cluster specifically for use with VMware’s ESXi, I would recommend SCST. Otherwise, I would say the LIO stack is easier to install and maintain because of its inclusion in the mainline kernel. That said, you should absolutely evaluate both implementations’ technical aspects from the perspective of your own specific use case.
LINBIT prides itself in its openness and aims to allow our users to choose what is right for them. That is why LINBIT builds and supports open source tools and frameworks for clustering rather than packaging “one size fits all” application stacks.
If you have any feedback on anything written above, or any insights or experiences you want to share with us regarding the iSCSI implementations mentioned (or not mentioned) in this blog, reach out!
