Skip to content

Jenkins backup and restore on Kubernetes

Jenkins backup and restore on Kubernetes involves managing the persistence and recovery of Jenkins data, primarily stored in /var/jenkins_home, using Kubernetes-native resources like Persistent Volume Claims (PVCs), CronJobs, and cloud storage integrations^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Data Persistence

By default, the Jenkins Helm chart uses a dynamically managed Persistent Volume Claim (PVC) to store data at /var/jenkins_home^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]. This ensures that data survives Pod restarts and deployments. It is possible to mount additional volumes using persistence.volumes and persistence.mounts parameters^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Users can either rely on Dynamic Provisioning or specify an existing PersistentVolumeClaim by setting persistence.existingClaim^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]. Additionally, the storage class can be defined via persistence.storageClass; setting this to a dash (-) disables dynamic provisioning^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Backup Configuration

The Jenkins Helm chart includes a feature to add a backup CronJob along with required RBAC resources^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]. This facilitates automated backups to destinations like [[Google Cloud Storage]] (GCS) or [[AWS S3]].

Backup to Google Cloud Storage

To implement a backup workflow to GCS, the following steps are typically required^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]:

  1. Prerequisites: Create a GCP account, a GCS bucket, and a GCP Service Account.
  2. Permissions: Bind the roles/storage.admin role to the Service Account to grant read/write access to the bucket^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].
  3. Key Management: Create a Service Account Key and store it in a Kubernetes Secret within the Jenkins namespace^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].
  4. Helm Configuration: Deploy the chart with backup enabled, referencing the secret and destination.

Example configuration: 

backup:
  enabled: true
  schedule: "0 2 * * *"
  existingSecret:
    jenkinsgcp:
      gcpcredentials: sa-credentials.json
  destination: "gcs://BUCKET_NAME/jenkins-k8s-backup"

Backup to AWS S3

The backup process supports AWS S3 similarly. If the AWS_REGION environmental variable is not provided, the chart defaults to eu-central-1^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]. To use a bucket in a specific region, the region must be explicitly defined in the backup.env configuration^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Restore Procedure

Restoring Jenkins from a backup involves copying data from cloud storage back to the Jenkins PVC. This can be achieved using the underlying skbn tool (via kube-tasks) to transfer files from cloud storage to the Kubernetes pod^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Restore from Google Cloud Storage

To restore from a GCS backup (e.g., after losing an installation), the process involves reinstalling Jenkins and running a restore Job^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]:

  1. Reinstall Jenkins: Deploy a clean instance of the Jenkins chart using the same configuration values^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].
  2. RBAC Setup: Create a Kubernetes ServiceAccount, ClusterRole, and ClusterRoleBinding to allow the restore job to access pods and exec commands^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].
  3. Restore Job: Create a Kubernetes Job that uses the maorfr/skbn image. The Job arguments should specify the source (GCS bucket and backup timestamp folder) and the destination (the Jenkins Pod's volume path, typically k8s://jenkins/jenkins-0/jenkins/var/jenkins_home)^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]. The GCP credentials Secret must be mounted to provide authentication^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].
  4. Reload Configuration: Once the Job completes and data is copied, log in to Jenkins and navigate to Manage Jenkins -> Reload Configuration from Disk to apply the restored state^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

A similar process applies for restoring from AWS S3 backups, using the appropriate configuration values in the Restore Job manifest^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md].

Sources

^[400-devops-06-kubernetes-devops-helm-helm-jenkins-readme.md]