Snapshot Shipping

LINSTOR – Controlling Data Replication with Snapshot Shipping

Whatever the situation, LINSTOR covers your back.

LINSTOR is an SDS technology with snapshot shipping integrated into the code, capable of real-time replication using DRBD. However, real-time replication may not always be the method we want to use. When you replicate data in real-time, you want your connection to have very low latency and high bandwidth as much as possible. In the following cases, we want the opposite:

  • When replicating our data to another data center in disaster recovery scenarios.
  • When replicating between two LINSTOR clusters in a particular time interval
  • When storing our data to s3 compatible storage solutions for backup scenarios.

Let’s take it one by one:

Disaster Recovery Scenarios

In those cases, the data centers used in DR scenarios need to be located in remote areas to reduce their chances of being affected by disaster situations. However, this makes communication difficult. In such cases, traffic is usually served with high latency and low bandwidth. (due to high prices) so we see real-time replication turn into an expensive solution.

If the RPO and RTO values (we will cover this topic in a different blog post) meet your expectations, instead of replicating the data in real-time, you can ship data between two different LINSTOR clusters at certain time intervals or to another LINSTOR node in a single cluster in a different data center.

Cluster Replication

LINSTOR does data replication between its nodes with DRBD. However, there may be situations where you want two different LINSTOR clusters to transfer data to each other without DRBD. In such cases, we cannot expect LINSTOR to share a single DRBD resource among different clusters. Therefore, we can use snapshot shipping to transfer data from one LINSTOR cluster to another LINSTOR cluster.

Backup scenarios

As we all know, replication is not equal to backup. Therefore, although replication is an essential issue for your HA needs, backup is also critical in business continuity plans.

To take a backup, you need to freeze the data at a specific time and transfer the data up to that moment to some storage.

Today, this storage is accepted as object storage due to its API structure and ease of use. Here, LINSTOR can transfer any DRBD resource to object storage (s3 compatible) at certain time intervals (hourly, daily, weekly, monthly, etc.)

Because snapshot shipping has become a requirement for these three scenarios, LINSTOR engineers have integrated snapshot shipping into the code.

In this way, you can operate from a single command line or GUI for disaster scenarios, cluster replication, and backup needs that an SDS needs to solve and that you need on the storage side.

Snapshot shipping in a few simple commands:

First, you need to create a “remote” definition in LINSTOR. The remote definition can be either an s3 compatible storage or another LINSTOR cluster.

To create an S3 target:

# LINSTOR remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password

Or to create a remote LINSTOR cluster:

# LINSTOR remote create LINSTOR myRemote 192.168.0.15

Then all you have to do is start the shipping process with the following command:

# linstor backup create myRemote myResource

For more detailed version of the functions, please check out our LINSTOR SNAPSHOT SHIPPING UserGuide.

Why snapshot shipping is efficient

Snapshot shipping is a bit of a sloppy term. Actually, it means, at first, the complete volume is transferred (in all relevant cases, this is a thinly provisioned volume, and only the parts of the volume that have real data are transferred). After the initial shipment, only differences between snapshots are transferred.

Expressed in other words, with shipping of incremental snapshot differences for all overwrites between two snapshots, only the last write gets transferred once.

A second important aspect is that a flaky network connection can not influence IO performance on the active side. A snapshot-delta gets applied into a new snapshot on the target, so an interrupted transfer leaves the previous snapshot untouched in place.

If LINSTOR’s storage pool is managed with ZFS, this feature leverages on zfs’s send/recv utilities. For LVM-thin, it uses the less known thin-send-recv utility by LINBIT.

Like? Share it with the world.

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on whatsapp
WhatsApp
Share on vk
VK
Share on reddit
Reddit
Share on email
Email