Disaster recovery (DR) is often perceived as a purely technological solution implemented within storage or virtualization environments, thanks to aggressive marketing by software vendors. However, DR involves more than software alone. It’s a comprehensive strategy that integrates diverse technologies, requiring businesses to recognize this broader perspective to mitigate potential risks and ensure effective recovery.
Challenging the Idea of a Single Ideal Solution
In an era of infrastructure advancements, we tend to seek miraculous solutions to our challenges. However, relying solely on a single technological solution, such as cloud services or backup systems or software can be insufficient in disaster recovery (DR). While these technologies are crucial components, they are merely part of a comprehensive DR strategy.
Disaster recovery planning involves developing a comprehensive strategy to ensure business continuity in case of a disaster. This plan should address not only technical aspects but also the roles and responsibilities of team members, and the processes and procedures to follow during an emergency. By having a well-defined plan in place, organizations can minimize downtime, maintain operations, and protect critical data and resources.
Dig the well before you’re thirsty. —Chinese Proverb
In 1959, Volvo engineer Nils Bohlin invented the three-point seat belt, which is widely used today. Volvo generously shared the invention with competitors, creating an incentive for them to incorporate seat belts into their vehicles. (Thank you Mr Bohlin!)
Strategies for Disaster Recovery Planning
People usually don’t think of creating a DR plan until they’re having data center level problems or some catastrophic events happen in the infrastructure. At this stage, they might start thinking about how to replicate and run their virtualization infrastructure or existing bare metal infrastructure in a remote data center.
A DR plan has several strategies:
- Data replication
- Infrastructure replication (virtualized or physical infrastructure)
- Application replication (HA clustering)
- Network replication (IP failover or similar solutions)
- Failover and failback plan
It is not possible to fit each of these stages into this article. Maybe we can make a video about this in the future. The part I especially want to emphasize and discuss in this article is data replication and the disaster planning around that.
At this point, over several years, I’ve taken on many proofs of concept at LINBIT® for a variety of businesses and organizations. These proofs of concept have included setting up various configurations for DR operations on a variety of infrastructures. The most common issue that I’ve encountered is figuring out how to replicate data at customer sites.
When replicating data, you don’t have many options. Either you’ll perform “real-time” data write replication (for example, by using DRBD®) or you’ll synchronize disks between two locations at specified intervals using a feature such as backup shipping. In the first case, the connection between the two locations needs to have low latency and high bandwidth. In the second case, because usually only data delta snapshots are synchronized at intervals, slightly worse network connections between sites can work.
As mentioned in the title of this article, DR is not a technology but a plan. To talk about the data replication aspect of this statement, replicating data by using one technology or another alone won’t save you from trouble in a disaster situation. Before deploying a data replication technology, you need to ask yourself how you will bring that data back up when there is a disaster event.
If you’re only replicating your data by using DRBD on bare metal infrastructure, setting up your applications or services for DR won’t be too troublesome. Open source software such as Pacemaker or DRBD Reactor can be sufficient for your needs in this case. However, making your CloudStack, Proxmox VE, or other virtualization infrastructures DR-ready can be a bit more complicated.
Disaster Recovery Site Architecture Strategies
In situations such as these, I recommend using different types of site architecture as a part of your DR plan. These site architectures are:
- A stretched cluster: By using a stretched cluster setup that spans at least two main data centers and third minimal arbitrator site, in case of a disaster, one data center can handle the workload of both data centers adequately.
- An Active-Passive data center: Using an active data center and setting up a passive cluster in a secondary data center, ensuring manual or automatic failover in case of a failure.
With the help of the LINBIT-developed open source software-defined storage (SDS) solution, LINSTOR®, both scenarios are achievable.
While I mentioned in the automatic failover in the architecture summary above, I can say that after designing many many many storage architectures across unique environments, I still have yet to encounter a single customer interested in automatically failing over to a DR environment. It’s not because it’s too hard to implement; it’s because failing over to another data center is a huge technical challenge to automate when the time comes to fail back to the original data center.
Essential Questions to Ask When Creating a Disaster Recovery Plan
When creating your DR plan, it’s essential to ask the following questions within your organization:
- What are the critical applications and assets of your business? (Do you really need a DR solution for the whole infrastructure or only some parts of it?)
- Does your storage solution and infrastructure support DR scenarios? (If not, how do you mitigate that?)
- What are the acceptable levels of downtime and data loss for each critical application? (Answering this question helps you to select the correct replication technology. To sound fancier, you can call them RPO and RTO)
- How often do you test and update your DR plan? (Believe me! This question is more important than whatever technology and DR plan you choose to use. If you never test your DR plan, that means your DR plan is not working at all. Testing and updating a DR plan is preparing for a disaster event. Without this, you are not prepared.)
Conclusion
Similar to insurance policies, DR plans might seem like a needless investment until a crisis occurs. However, having a solid DR plan in place can be crucial when unexpected events strike. Implementing a DR plan can make the difference between a smooth business recovery or facing a significant disruption. While it is ideal for a DR plan to remain unused, having one provides peace of mind and will ensure seamless business operations in the face of unforeseen events.
