In distributed storage environments, ensuring data consistency and availability across multiple nodes is crucial. LINSTOR®, the software-defined storage (SDS) software created by LINBIT®, is a robust tool for managing data replication and maintaining data integrity both locally for high availability (HA) and across distances for disaster recovery (DR).
LINSTOR is an SDS technology with snapshot shipping integrated into its code. LINSTOR also enables real-time replication by using DRBD®. Managing DRBD is arguably LINSTOR’s core function in most user’s environments. However, real-time replication might not always be the data replication method that you want to use. When you replicate data in real-time, you need your network connection to have very low latency and high bandwidth. This is to ensure that data replication over the network does not become a bottleneck and impact your storage or application performance. When looking to satisfy disaster recovery needs, as in the following scenarios, the opposite is generally true:
- When replicating your data to another data center in disaster recovery scenarios.
- When replicating between two LINSTOR clusters in a particular time interval
- When storing your data on S3-compatible storage solutions for backup scenarios.
These scenarios typically have low bandwidth and high latency networks, making them a good fit for LINSTOR’s snapshot shipping. The next sections examine each case one-by-one.
Disaster Recovery Scenarios
For disaster recovery scenarios, data centers need to be located in remote and geographically dispersed areas to reduce their chances of being affected by localized disasters. However, this makes communication between data centers difficult. In such cases, network traffic is usually served with high latency and low bandwidth, due to higher costs for faster networking infrastructure. Over long distances, real-time replication is either not possible or else turns into an expensive solution.
If the RPO and RTO values meet your expectations, instead of replicating the data in real time, you can ship data between two different LINSTOR clusters at specified time intervals or to another LINSTOR node in the same LINSTOR cluster in a different data center. This second option, stretching a LINSTOR cluster across data centers, is a topic discussed in an appropriately titled LINBIT® blog article, The Stretched Cluster Disaster Recovery Strategy.
Cluster Replication
LINSTOR does data replication between its nodes by using DRBD. However, there might be situations where you want two different LINSTOR clusters to transfer data to each other without DRBD. In such cases, LINSTOR needs to replicate storage resources by different means. Here, you can use snapshot shipping to transfer data from one LINSTOR cluster to another LINSTOR cluster.
Backup Scenarios
Replicating data is not the same as a backing up data. Replication is continuous and always on. It moves in step with changes to your data. Because of this, replication is an essential component of HA. Backing up however is discrete. Making regular data backups is a way of creating copies of your data that you can restore from, should you need to. As such, backup is a critical component in your DR plan.
To take a backup, you need to freeze the data at a specific point in time and transfer that backup to some other remote and secure storage. Commonly, this remote storage will be object storage, due to its API structure and ease of use. Because of this, LINSTOR can transfer backups of its resources directly into AWS S3 or S3-compatible object storage, and can be configured to do so automatically at specified time intervals (hourly, daily, weekly, monthly, and so on).
Because snapshot shipping has become a requirement for these three scenarios, LINSTOR developers have integrated snapshot shipping into the code. With this, you can operate from a single command line or GUI when recovering from disaster scenarios, configuring cluster replication, and managing your backup needs.
Snapshot Shipping in a Few Simple Commands Using LINSTOR
First, you need to create a “remote” definition in LINSTOR. The remote definition can be either an AWS S3 or S3-compatible storage or another LINSTOR cluster.
To create an S3 target:
# linstor remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 access_key secret_key
Or to create a remote LINSTOR cluster:
# linstor remote create LINSTOR myRemote 192.168.0.15
Then all you have to do is start the shipping process with the following command:
# linstor backup create myRemote myResource
For more detailed version of the functions, refer to the LINSTOR User Guide.
If you’re using LINSTOR as your storage provider for Kubernetes, a colleague’s blog titled, Abstracting Persistent Storage Across Environments With LINBIT SDS, explains how snapshots and snapshot shipping work in a Kubernetes environment.
Why Snapshot Shipping is Efficient
Snapshot shipping is a bit of a loosely used term. Actually, it means the complete storage volume is transferred (in all relevant cases, this is a thinly provisioned volume, and only the parts of the volume that have real data are transferred). For efficiency, after the initial shipment of the complete volume, only the differences between the subsequent snapshots are transferred.
Expressed in other words, the full snapshot is shipped once, and only the incremental changes are shipped afterwards. It is a good idea to periodically take and ship full snapshots, so you don’t have to rely on dozens of incremental snapshots when the time comes to restore a resource. These are commonplace concepts in the data backup and recovery world.
A second important aspect is that a flaky network connection should not influence I/O performance on the active volume. A snapshot delta (a snapshot of changes since the last snapshot) gets applied into a new snapshot on the target, so an interrupted transfer leaves the previous snapshot untouched in place.
If the LINSTOR storage pool that your resource belongs to has a ZFS back end, the snapshot feature uses ZFS built-in send and receive utilities. For storage pools that have a thin-provisioned LVM back end, the LINSTOR snapshot feature uses the lesser known LINBIT created and developed open source thin-send-recv utility.
