Backups

Overview

PodWarden's backup system creates incremental, deduplicated, encrypted backups of workload volumes using Restic. Backups are orchestrated by the PodWarden Backup Operator — a Kubernetes operator that runs inside your cluster and manages the full backup lifecycle.

What gets backed up: persistent volume contents (PVC data). The backup engine mounts the same volumes as your workload and streams them to a storage connection you control.

Where backups go: any NFS or S3-compatible storage connection registered in PodWarden. One storage connection can serve as the backup target for many workloads. Each policy gets its own isolated Restic repository path.

Prerequisites

The PodWarden Backup Operator must be installed in your cluster before backup policies can be used. Without it, policies can be created but will remain in Pending state and no backups will run.

Install via the Hub catalog or by applying the operator manifest directly:

kubectl apply -f https://www.podwarden.com/operators/backup-operator/install.yaml

See System Apps for more details on operator installation and detection.

How It Works

PodWarden creates BackupPolicy custom resources in your cluster. The backup operator watches for these resources and:

1. Schedules backup runs according to the cron expression in the policy
2. Creates BackupRun resources for each execution
3. Runs Restic inside the cluster to copy volume data to the storage target
4. Applies retention rules and updates snapshot metadata

You do not need to manually apply or remove any Kubernetes resources — the operator handles scheduling automatically when a policy is created or updated.

Backup Modes

Hot mode

The backup Job mounts the PVC volumes while the workload is running. Add a pre-backup hook to run a command inside the pod before file copy starts:

mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /var/lib/mysql/dump.sql

The hook runs via kubectl exec. File backup starts only after the hook completes.

Cold mode

1. PodWarden scales the workload to 0 replicas
2. Restic backs up the volumes
3. Workload scales back to 1

Typical downtime is 20-60 seconds depending on backup storage speed.

PostgreSQL Backups

In addition to volume backups, PodWarden supports direct PostgreSQL database backups. Instead of mounting PVC files, a PostgreSQL backup runs pg_dump against the database and stores the dump file via Restic.

When to use

• Volume backup with pre-hook — best for most databases. The hook runs pg_dump inside the running pod, then Restic copies the dump file along with other volume data.
• PostgreSQL backup — use when the database is external to K8s (e.g. a managed database or a database on another host), or when you want a dedicated backup pipeline separate from volume data.

Creating a PostgreSQL Backup Policy

1. Go to Backups and click New backup policy
2. Set Backup type to postgres
3. Fill in the database connection details:
   • Host — database hostname or IP
   • Port — default 5432
   • Database — database name
   • User — PostgreSQL user
   • Password — stored as a Kubernetes Secret
4. Set the schedule and retention as usual
5. Select a storage target
6. Click Create policy

How it works

The operator creates a BackupRun with two containers:

1. Init container (postgres:16-alpine) — runs pg_dump --format=custom to a shared temporary volume
2. Main container (restic/restic) — uploads the dump to the Restic repository

The custom format produces a single compressed file suitable for selective restore with pg_restore.

Creating a Backup Policy

1. Go to Backups in the sidebar
2. Click New backup policy
3. Select the deployed workload to back up
4. Choose backup mode: Hot or Cold
5. If hot mode: optionally enter a pre-backup hook command
6. Set the schedule (daily, weekly, or custom cron)
7. Select the storage target (a registered storage connection)
8. Configure retention
9. Click Create policy

The operator picks up the new BackupPolicy and begins scheduling runs automatically. Click Run now to trigger an immediate backup outside the normal schedule.

Schedule

Retention

Retention rules are passed to Restic's forget command after each backup:

Restic applies them cumulatively — a snapshot is kept if it matches any rule.

Snapshots

Click any policy row to see its snapshots. Each shows the Restic snapshot ID, timestamp, size, status, and file counts.

Restic's incremental deduplication means subsequent backups of unchanged data are fast and store only deltas.

Restoring from a Snapshot

1. Click the restore button on a successful snapshot
2. Confirm the restore
3. PodWarden creates a RestoreRun CRD; the operator scales the workload to 0, runs restic restore, then scales back to 1

Restore takes 30-120 seconds plus data transfer time. Data written after the snapshot is overwritten.

The operator performs a compatibility check before restoring — verifying that the target PVC names and sizes are compatible with the snapshot. If the check fails, the RestoreRun is marked Failed with a reason.

Pre-backup Hooks

Shell commands run inside the running pod before Restic copies files. Hot mode only.

# MySQL
mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /var/lib/mysql/dump.sql
# PostgreSQL
pg_dumpall -U postgres > /var/lib/postgresql/data/dump.sql
# Redis
redis-cli BGSAVE && sleep 2

If the hook exits non-zero, the backup is aborted. Hub templates may pre-fill hooks via backup_hooks metadata.

Storage Requirements

Both NFS and S3-compatible storage connections are supported:

• NFS — on-premises or LAN backups, fast, no egress cost
• S3 — off-site backups, supports AWS S3, MinIO, SeaweedFS, Backblaze B2

Each policy creates its own Restic repository at a unique path:

backups/{cluster}/{workload}-{suffix}/

The unique suffix ensures that deleting and recreating a policy for the same workload produces a fresh Restic repository, avoiding password mismatches with leftover encrypted data. Repositories are initialized automatically with a random password stored as a Kubernetes Secret.

Compose Stack Backups

Compose stacks with multiple shared volumes are handled automatically:

• Per-PVC runs — each PVC is backed up by its own BackupRun. Required because local-path volumes are pinned to specific nodes, and different PVCs may live on different nodes.
• Staggered schedules — runs are staggered by 2 minutes. The first run initializes the Restic repository; subsequent runs wait for the repo before starting.
• Single snapshot — per-PVC runs are grouped into one logical snapshot, marked successful only when all runs complete.

No special configuration is needed. PodWarden discovers all PVCs and the operator creates per-PVC runs when the policy is scheduled.

Kubernetes Resources

Each backup policy creates:

• BackupPolicy pw-backup-{workload} — the CRD watched by the operator.
• BackupRun(s) — created by the operator for each scheduled or on-demand execution.
• Secret pw-backup-{suffix} — Restic password, generated once at policy creation.
• Secret for S3 credentials (if applicable) from the storage connection.
• ServiceAccount podwarden-backup with RBAC for Deployment reads and pod exec.

Troubleshooting

Policy stuck in Pending: The backup operator may not be installed. Run kubectl get backuppolicies to confirm the CRD exists. See System Apps for installation instructions.

Stuck in running: BackupRuns missing from K8s after 15 minutes are marked failed. Click refresh to sync. Check operator logs: kubectl -n podwarden-system logs deployment/backup-operator.

Inspecting policies and runs:

kubectl get backuppolicies
kubectl get backupruns
kubectl describe backuppolicy <name>
kubectl describe backuprun <name>

Pre-backup hook fails: Verify the command and that the pod has the required tools.

Restore fails: Check namespace and PVC names. Inspect the RestoreRun: kubectl describe restorerun <name>.

Password mismatch / ciphertext verification failed: The repo was initialized with a different password. Delete the policy, create a new one (fresh repo path and password), and run a backup.