Backing up and restoring MarbleRun state

In a production environment, you should regularly back up the state of the MarbleRun Coordinator to be able to restore it in case of failure. Backup is easy, but there are differences based on how you deployed MarbleRun.

Prerequisites

Restoring a backup includes recovering the Coordinator, so you need to define recovery keys in the manifest.

Backing up the Coordinator state

The Coordinator supports live backup, so you can back up its state without stopping it.

Kubernetes, distributed Coordinator

info

This feature is an option with MarbleRun Enterprise.

Make a copy of the marblerun-state Secret in the marblerun namespace:

kubectl -n marblerun get secret marblerun-state -o yaml > marblerun-state-backup.yaml

Kubernetes, single Coordinator

The state is stored on a PersistentVolume bound to the PersistentVolumeClaim coordinator-pv-claim in the marblerun namespace. Check the documentation of your Kubernetes distribution on the available options to back up PersistentVolumes.

Alternatively, you can copy the sealed_data file from the PersistentVolume (via the Coordinator Pod) to your machine:

podname=$(kubectl -n marblerun get pods -l app.kubernetes.io/name=coordinator -o jsonpath='{.items[0].metadata.name}')
kubectl -n marblerun cp $podname:/coordinator/data/sealed_data sealed_data_backup

Standalone

Make a copy of the sealed_data file in the marblerun-coordinator-data directory.

Restoring the Coordinator state

Kubernetes, distributed Coordinator

1. Stop all Coordinator instances:

   kubectl -n marblerun scale --replicas=0 deployment/marblerun-coordinator

2. Apply the state from the backup:

   kubectl apply -f marblerun-state-backup.yaml

3. Scale the Coordinator back to the desired number of instances:

   kubectl -n marblerun scale --replicas=3 deployment/marblerun-coordinator

tip

If you want to restore MarbleRun in a fresh cluster, you can apply the state from the backup before installing MarbleRun:

kubectl create ns marblerun
kubectl apply -f marblerun-state-backup.yaml
marblerun install ...

Kubernetes, single Coordinator

• If you have a backup of the PersistentVolume, stop the Coordinator instance, restore the volume, and start the Coordinator again.

• If you have a backup of the sealed_data file, copy it to the PersistentVolume and then restart the Coordinator:

  podname=$(kubectl -n marblerun get pods -l app.kubernetes.io/name=coordinator -o jsonpath='{.items[0].metadata.name}')
  kubectl -n marblerun cp sealed_data_backup $podname:/coordinator/data/sealed_data
  kubectl -n marblerun delete pod $podname

Standalone

Stop the Coordinator, copy back the sealed_data file to the marblerun-coordinator-data directory, and start the Coordinator again.

After restoring the state from the backup, you may need to recover the Coordinator.

Things to consider

Backup events: In addition to regular backups, you may want to back up the state after significant changes, such as manifest updates.

Cluster backup: If you use a Kubernetes cluster backup solution, the MarbleRun state may already be included in that backup. You should check if restoring and recovering the Coordinator works as expected.

Marbles: Marbles may have state and that state may depend on the Coordinator state (e.g., secrets, monotonic counters). If so, you may need to back up Marble state and Coordinator state together.