Scaling Jenkins for high availability (HA) and disaster recovery (DR) was not a primary driving force in the open source automation platform’s development history. “Developers who extended Jenkins favored speed of creating functionality over performance or scalability of the entire system,” observe the writers of an Amazon Web Services (AWS) DevOps blog article, “Jenkins high availability and disaster recovery on AWS”. The same blog article overviews a way to scale Jenkins for HA by using Amazon Elastic Kubernetes Service (Amazon EKS). EKS takes care of scaling the compute side of this continuous integration and continuous deployment (CI/CD) platform. However, scaling the data storage aspect of Jenkins for HA is not as straightforward.
The Challenge of Scaling Jenkins Storage for HA
The AWS DevOps blog article goes into detail about the issues that make scaling Jenkins storage for HA a challenge. To summarize, JENKINS_HOME, by default on a local file system, might be fine for a single Jenkins server, but when trying to scale a Jenkins deployment, even placing JENKINS_HOME on shared storage such an NFS export or a SAN likely means accepting a performance hit and does not provide true HA or DR in the event of a storage-level failure.
AWS offers two relevant storage types that might be used to back Jenkins storage: Amazon Elastic Block Store (Amazon EBS) and Amazon Elastic File System (Amazon EFS). Again, the AWS DevOps blog provides details about each of these and the pros and cons of using each type for Jenkins storage.
Complementing AWS Storage by Using LINBIT SDS in Kubernetes
Because the compute side of scaling Jenkins for HA relies on using Amazon EKS, you can use LINBIT SDS (LINSTOR®, DRBD®, and related utilities) for Kubernetes to help solve the issues around scaling Jenkins storage. LINBIT SDS can run in Kubernetes, and Kubernetes variants such as EKS, and is easily deployed by using the LINSTOR Operator and kustomize or Helm. LINBIT SDS in Kubernetes gives you a cloud-native way to provide highly available persistent volume claims (PVCs) to your applications and services running in Kubernetes.
In this use case, EKS worker nodes running Jenkins agents or the Jenkins controller will have their PVCs backed by LINSTOR defined HA storage. DRBD’s high performance data replication mirrors data across storage nodes and facilitates the HA. The most interesting feature LINBIT SDS brings to EKS, and what truly makes this a worthy HA solution, is granting applications (in this case Jenkins) the ability to migrate or failover between availability zones (AZs). While this is possible using other storage solutions, like EFS, LINBIT SDS enables this with minimal impact to performance. LINBIT SDS can also be used in your DR plan. By using LINSTOR’s volume snapshot shipping feature you can send volume backups to Amazon S3 storage in different AWS regions.
Jenkins HA in AWS on EKS with LINSTOR
The above diagram shows a singleton Jenkins pod running in an EC2 instance within a single AWS AZ. It writes its persistent data to a LINSTOR volume, which is synchronously replicated to its LINSTOR peer volumes, each in their own different AZ. LINSTOR’s control and data planes communicate over private or public subnets attached to each AZ and routed by AWS.
LINSTOR’s volumes are backed by EBS volumes in each instance’s AZ, which are always identical to each other thanks to LINSTOR’s synchronous replication. In case of an AZ outage, LINSTOR’s HA controller for StatefulSets will kick in and move the Jenkins workload to another AZ within a few minutes.
The EKS managed load balancer will transparently reroute users to the active AZ whenever the Jenkins pod is migrated.
LINSTOR supports snapshot shipping of volumes to Amazon’s S3 to satisfy disaster recovery requirements. The Amazon S3 bucket you ship your backups to should be in a different AWS region than your EKS cluster.
How To Deploy Jenkins to Scale for HA and DR
The AWS DevOps blog article has some setup details but at this point they are dated. In the time passed since the AWS DevOps blog article was published, LINBIT SDS has an Operator v2 that you can deploy by using kustomize and which has many improvements over the v1 edition. For complete, step-by-step instructions on how to deploy this solution, LINBIT® Solutions Architect Matt Kereczman and AWS team members James Bland (APN Global Tech Lead) and Welly Siauw (Sr. Partner Solution Architect) collaborated and wrote a comprehensive how-to guide.
You can download the “Jenkins High Availability Clustering Using LINBIT SDS in EKS” how-to guide from the LINBIT website.
If you have any questions about this architecture or any aspect of the how-to guide, reach out to the LINBIT team. You can also seek help within the LINBIT Community Forums, a place LINBIT team members also contribute to.
