Skip to content

Jenkins backup and restore strategy

For Jenkins deployments running on Kubernetes, particularly those managed via Helm, a robust backup and restore strategy is essential to prevent data loss. This typically involves utilizing cloud storage solutions, such as Google Cloud Storage (GCS) or AWS S3, and leveraging Kubernetes Jobs or CronJobs to handle data transfer.^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md]

Backup Strategy

The recommended approach for backing up Jenkins is to use a Kubernetes CronJob that periodically uploads data from the Jenkins Persistent Volume Claim (PVC) to a cloud storage destination^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Prerequisites

Before enabling backups, ensure persistence is enabled in your Helm values to guarantee that a PVC exists to be backed up^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

persistence:
  enabled: true

Configuration for Cloud Storage

To automate backups, you must provide credentials for the backup service to access your cloud storage.

  • Google Cloud Storage (GCS):

    1. Create a GCS bucket and a Service Account with the roles/storage.admin role^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].
    2. Create a Service Account Key and store it as a Kubernetes Secret^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].
    [kubectl](<./kubectl.md>) -n jenkins create secret generic jenkinsgcp --from-file=sa-credentials.json=/path/to/sa_key.json

  • AWS S3:

    • If the S3 bucket is not in the default region (eu-central-1), you must explicitly provide the region via environment variables^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Enabling the Backup CronJob

You can enable the backup CronJob by modifying your values.yaml file. This configuration sets the schedule, destination, and the secret required for authentication^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

backup:
  enabled: true
  schedule: "0 2 * * *" # Run daily at 2 AM
  existingSecret:
    jenkinsgcp: # Matches the secret name
      gcpcredentials: sa-credentials.json # Key in the secret
  destination: "gcs://BUCKET_NAME/jenkins-k8s-backup"

Once deployed, the CronJob will create a time-stamped folder in the destination bucket during each execution^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md]. You can verify the success of the operation by checking the logs of the backup pod^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Restore Strategy

Restoring Jenkins involves copying data from cloud storage back to the Jenkins PVC. Because the restore process requires writing to the volume of another Pod, specific RBAC resources must be created first^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Step 1: Create RBAC Resources

You must define a ServiceAccount, ClusterRole, and ClusterRoleBinding to authorize the restore job to interact with the Jenkins Pod's filesystem^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

restore-rbac.yaml: 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: skbn
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: skbn
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: skbn
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: skbn
subjects:
- kind: ServiceAccount
  name: skbn
  namespace: jenkins

Apply this configuration with kubectl apply -f restore-rbac.yaml^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Step 2: Create the Restore Job

The actual restore is performed by a Kubernetes Job utilizing the skbn tool, which copies files from the cloud storage source to the Kubernetes destination^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

restore.yaml: 

apiVersion: batch/v1
kind: Job
metadata:
  name: skbn
  namespace: jenkins
spec:
  template:
    spec:
      restartPolicy: OnFailure
      serviceAccountName: skbn
      containers:
      - name: skbn
        image: maorfr/skbn
        command: ["skbn"]
        args:
        - "cp"
        - "--src"
        - "gcs://BUCKET_NAME/jenkins-k8s-backup/BACKUP_NAME"
        - "--dst"
        - "k8s://jenkins/jenkins-0/jenkins/var/jenkins_home"
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/run/secrets/jenkinsgcp/sa-credentials.json
        volumeMounts:
        - mountPath: /var/run/secrets/jenkinsgcp
          name: jenkinsgcp
      volumes:
      - name: jenkinsgcp
        secret:
          secretName: jenkinsgcp

  • Replace BUCKET_NAME and BACKUP_NAME (the time-stamped folder) with your specific values^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].
  • The destination path k8s://jenkins/jenkins-0/jenkins/var/jenkins_home assumes the Jenkins controller Pod is named jenkins-0 (StatefulSet) and data is stored in the default home directory^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Deploy the job using kubectl apply -f restore.yaml and monitor the logs to ensure completion^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Step 3: Finalize the Restore

Once the data has been copied to the disk, you must reload the configuration within the Jenkins UI^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

  1. Log in to Jenkins.
  2. Navigate to Manage Jenkins.
  3. Click Reload Configuration from Disk.
  4. Important: Log in using the credentials from the restored backup, not the credentials of the fresh installation^[400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md].

Sources

  • 400-devops__06-Kubernetes__devops-helm__helm-jenkins__README.md