Disaster Recovery

Self-hosted KubeWatch can run its own scheduled backups instead of the manual pg_dump/cron approach in Backup & Restore. Each backup is pushed to one or more destinations you control, then immediately restored into a scratch database to verify it actually works, before that scratch database is dropped.

What it does

On the schedule you configure (every 24 hours by default: an interval on Docker Compose, a standard cron schedule on Kubernetes, since Kubernetes already has native CronJob scheduling):

1. Runs pg_dump against your KubeWatch database.
2. Pushes the dump to every destination listed in BACKUP_DESTINATIONS (one, or several at once for redundancy).
3. Restores the dump into a scratch database on the same Postgres server, never the real kubewatch database, and runs a basic verification query.
4. Drops the scratch database and records the result, per destination.
5. Prunes backups older than BACKUP_RETENTION_DAYS, deleting them from every destination as well as from the backup history.

You can also trigger a backup + restore test immediately from Disaster Recovery in the dashboard, without waiting for the schedule.

Choose your destination(s)

Set a comma-separated list of any of these four, then add that destination's own settings below. List more than one to push every backup to all of them at once.

Docker Compose (~/kubewatch-erp/.env):

BACKUP_DESTINATIONS=s3                       # or: s3,sftp  /  webhook  /  s3,azure  / etc.
BACKUP_INTERVAL_SECONDS=86400                # optional, default 24h

Kubernetes (Helm values, or --set on helm upgrade):

--set backup.enabled=true \
--set backup.destinations=s3 \
--set backup.schedule="0 2 * * *"            # standard cron syntax, default daily at 02:00

The env var names below are the same either way. Docker reads them from .env, and on Kubernetes you set the equivalent backup.<destination>.<field> Helm value, e.g. BACKUP_S3_BUCKET becomes --set backup.s3.bucket=....

S3-compatible object storage (s3)

BACKUP_S3_ENDPOINT=s3.amazonaws.com          # or any S3-compatible endpoint
BACKUP_S3_BUCKET=my-kubewatch-backups
BACKUP_S3_ACCESS_KEY=AKIA...
BACKUP_S3_SECRET_KEY=...
BACKUP_S3_USE_SSL=true                       # set false for a plain-HTTP endpoint (e.g. local MinIO)

Any provider that speaks the S3 API works: AWS S3, MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2, Wasabi, and others.

SFTP (sftp)

BACKUP_SFTP_HOST=backups.example.com
BACKUP_SFTP_PORT=22                          # optional, default 22
BACKUP_SFTP_USER=kubewatch-backup
BACKUP_SFTP_PATH=/home/kubewatch-backup/dumps
BACKUP_SFTP_PRIVATE_KEY=<base64-encoded PEM private key>

Generate a dedicated key just for this, rather than reusing an existing admin key:

ssh-keygen -t ed25519 -f kubewatch_backup_key -N ""
base64 -w0 kubewatch_backup_key   # paste this into BACKUP_SFTP_PRIVATE_KEY

Add kubewatch_backup_key.pub to the target server's ~/.ssh/authorized_keys for that user, scoped to only what it needs (write access to the backup directory).

Generic webhook (webhook)

BACKUP_WEBHOOK_URL=https://your-service.example.com/backups
BACKUP_WEBHOOK_AUTH_HEADER=Bearer your-token-here   # optional

The dump is sent as a raw POST request body (Content-Type: application/octet-stream), with an X-Backup-Filename header carrying the suggested filename, and the Authorization header set to BACKUP_WEBHOOK_AUTH_HEADER if you set one. Wire this into whatever upload pipeline you already run.

Azure Blob Storage (azure)

BACKUP_AZURE_CONTAINER_SAS_URL=https://youracct.blob.core.windows.net/container?sv=...&sig=...

Generate a container-level SAS URL with write permission from the Azure Portal (Storage Account → Containers → your container → Shared access tokens) or the CLI:

az storage container generate-sas \
  --account-name youracct --name container \
  --permissions w --expiry 2027-01-01 -o tsv

Set an expiry that's comfortably far out, or plan to rotate it. Once it expires, uploads will start failing until you generate and set a new one.

Retention

BACKUP_RETENTION_DAYS=30                     # optional, default 30. 0 keeps every backup forever.

Every time a backup runs (scheduled or manual), any backup older than BACKUP_RETENTION_DAYS is deleted: the object from S3, the file from SFTP, the blob from Azure, and the row from the backup history. Set it to 0 to disable pruning entirely and keep every backup indefinitely.

Apply the configuration

Docker Compose:

cd ~/kubewatch-erp
docker compose up -d

Kubernetes: re-run your helm upgrade with the backup.* values set (see Kubernetes (Helm) for the full install command this extends).

Viewing backup history

The Disaster Recovery page in the dashboard shows every backup (when it ran, how it was triggered, size, and status), which destination(s) it landed in and whether each one succeeded, and its restore-test result (passed, failed, and how many organizations were verified in the restored copy).

Why the restore test matters

A backup file that was never actually restored is an assumption, not a guarantee. Corruption, a pg_dump version mismatch, or a destination permissions issue can all produce a file that looks fine but doesn't actually restore. Every backup this feature makes gets proven to restore, automatically, the same day it's taken. It isn't "rehearsed" once a quarter. It happens at every scheduled interval.

Limitations (this release)

• Only the KubeWatch Postgres database is covered. VictoriaMetrics/VictoriaLogs time-series data still needs its own volume-level backup, exactly as described in Backup & Restore.