Tutorial: Stacked Block Storage in LINBIT SDS (aka LINSTOR)

In this tutorial, I will show you how to use LINBIT SDS to orchestrate DRBD, bcache and ZFS in a stacked block storage layer.

 

We have three satellite nodes in this demo cluster, all installed the latest CentOS 8.4, each node has 1 x NVMe drive and 3 x HDD drives.

 

Satellite01 192.168.122.41
Satellite02 192.168.122.42
Satellite03 192.168.122.43

Install ZFS

Follow the official OpenZFS document to install ZFS first.

 

dnf install “https://zfsonlinux.org/epel/zfs-release.el8_4.noarch.rpm”

 

DKMS and kmod both work fine, in this tutorial, we choose kmod to avoid development dependencies installation, also a little bit faster during ZFS installation.

 

yum-config-manager —disable zfs
yum-config-manager —enable zfs-kmod
dnf install kmod-zfs zfs

Install bcache

CentOS 8.x and EPEL repositories don’t have bcache RPM packages at this moment, you have to compile by yourself, or simply install from LINBIT repository for whom within the validated LINBIT credentials.

dnf install kmod-bcache bcache-tools

Install DRBD

Same as bcache, if you have the validated LINBIT credentials.

dnf install drbd kmod-drbd

Optional configuration

By default, ZFS kernel module will be loaded automatically until at least one zpool exists, or you could manually load ZFS kernel module on boot.

echo “zfs” > /etc/modules-load.d/zfs.conf

cache

echo “bcache” > /etc/modules-load.d/bcache.conf

DRBD

echo “drbd” > /etc/modules-load.d/drbd.conf
echo “drbd_transport_tcp” >> /etc/modules-load.d/drbd.conf

Double check the modules

lsmod | grep -i bcache
lsmod | grep -i zfs
lsmod | grep -i drbd

If you see some outputs for each three commands above on three satellite nodes, then that means you have everything working, congratulations!

Now we are going to set up ZFS, nothing to change in DRBD and bcache for now.

Create zpool(s) on all satellite nodes

zpool create DataPool /dev/sda /dev/sdb /dev/sdc
zpool create CachePool /dev/nvme0n1

Let’s look into “LINBIT SDS”

# add nodes
linstor node create Satellite01 192.168.122.41
linstor node create Satellite02 192.168.122.42
linstor node create Satellite03 192.168.122.43

# create storage pool for data
linstor storage-pool create zfs Satellite01 LinstorData DataPool
linstor storage-pool create zfs Satellite02 LinstorData DataPool
linstor storage-pool create zfs Satellite03 LinstorData DataPool

# create storage pool for cache
linstor storage-pool create zfs Satellite01 LinstorCache CachePool
linstor storage-pool create zfs Satellite02 LinstorCache CachePool
linstor storage-pool create zfs Satellite03 LinstorCache CachePool

# create resource group
linstor resource-group create RG01 —storage-pool LinstorData —place-count 3 —layer-list drbd,bcache,storage

# set bcache for resource group
linstor resource-group set-property RG01 BCache/PoolName LinstorCache

## configure bcache in LINBIT SDS (optional)
# linstor resource-group set-property RG01 BCache/Writeback true
# linstor resource-group set-property RG01 BCache/Discard true

linstor volume-group create RG01
linstor resource-group spawn RG01 RD01 2G

 

All done! RD01 is a stacked block device which is backed by DRBD, bcache and ZFS from top to bottom.

 

What does that stacked block device mean?

 

When writing to RD01, DRBD will replicate data to other nodes, then bcache involves to improve write performance, persistent data on ZFS pools in the end.

 

What about deduplication? Any thoughts 🙂

 

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Share on whatsapp
Share on vk
Share on email

Share this post

Joel Zhou

Joel Zhou

Joel Zhou is a Solutions Architect at LINBIT, and has more than 15 years of experience in open source software, especially focusing on enterprise infrastructure (cloud computing, software-defined storage, etc.). He's been to the basecamp of Everest, dived in the ocean in the Maldives, and is a coffee lover and Rock & Roll fan.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.