LINSTOR – Controlling Data Replication with Snapshot Shipping

Snapshot Shipping

Whatever the situation, LINSTOR covers your back.

LINSTOR® is an SDS technology with snapshot shipping integrated into the code, capable of real-time replication using DRBD®. However, real-time replication may not always be the method we want to use. When you replicate data in real-time, you want your connection to have very low latency and high bandwidth as much as possible. In the following cases, we want the opposite:

  • When replicating our data to another data center in disaster recovery scenarios.
  • When replicating between two LINSTOR clusters in a particular time interval
  • When storing our data to s3 compatible storage solutions for backup scenarios.

Let’s take it one by one:

Disaster Recovery Scenarios

In those cases, the data centers used in DR scenarios need to be located in remote areas to reduce their chances of being affected by disaster situations. However, this makes communication difficult. In such cases, traffic is usually served with high latency and low bandwidth. (due to high prices) so we see real-time replication turn into an expensive solution.

If the RPO and RTO values (we will cover this topic in a different blog post) meet your expectations, instead of replicating the data in real-time, you can ship data between two different LINSTOR clusters at certain time intervals or to another LINSTOR node in a single cluster in a different data center.

Cluster Replication

LINSTOR does data replication between its nodes with DRBD. However, there may be situations where you want two different LINSTOR clusters to transfer data to each other without DRBD. In such cases, we cannot expect LINSTOR to share a single DRBD resource among different clusters. Therefore, we can use snapshot shipping to transfer data from one LINSTOR cluster to another LINSTOR cluster.

Backup scenarios

As we all know, replication is not equal to backup. Therefore, although replication is an essential issue for your HA needs, backup is also critical in business continuity plans.

To take a backup, you need to freeze the data at a specific time and transfer the data up to that moment to some storage.

Today, this storage is accepted as object storage due to its API structure and ease of use. Here, LINSTOR can transfer any DRBD resource to object storage (s3 compatible) at certain time intervals (hourly, daily, weekly, monthly, etc.)

Because snapshot shipping has become a requirement for these three scenarios, LINSTOR engineers have integrated snapshot shipping into the code.

In this way, you can operate from a single command line or GUI for disaster scenarios, cluster replication, and backup needs that an SDS needs to solve and that you need on the storage side.

Snapshot shipping in a few simple commands:

First, you need to create a “remote” definition in LINSTOR. The remote definition can be either an s3 compatible storage or another LINSTOR cluster.

To create an S3 target:

# LINSTOR remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password

Or to create a remote LINSTOR cluster:

# LINSTOR remote create LINSTOR myRemote 192.168.0.15

Then all you have to do is start the shipping process with the following command:

# linstor backup create myRemote myResource

For more detailed version of the functions, please check out our LINSTOR SNAPSHOT SHIPPING UserGuide.

Why snapshot shipping is efficient

Snapshot shipping is a bit of a sloppy term. Actually, it means, at first, the complete volume is transferred (in all relevant cases, this is a thinly provisioned volume, and only the parts of the volume that have real data are transferred). After the initial shipment, only differences between snapshots are transferred.

Expressed in other words, with shipping of incremental snapshot differences for all overwrites between two snapshots, only the last write gets transferred once.

A second important aspect is that a flaky network connection can not influence IO performance on the active side. A snapshot-delta gets applied into a new snapshot on the target, so an interrupted transfer leaves the previous snapshot untouched in place.

If LINSTOR’s storage pool is managed with ZFS, this feature leverages on zfs’s send/recv utilities. For LVM-thin, it uses the less known thin-send-recv utility by LINBIT®.

Yusuf Yıldız

Yusuf Yıldız

After nearly 15 years of system and storage management, Yusuf started to work as a solution architect at LINBIT. Yusuf's main focus is on customer success and contributing to product development and testing. As part of the solution architects team, he is one of the backbone and supporter of the sales team.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.