Backup & Restore

KubeWatch stores relational data (organizations, users, agents, alert rules, and alert history) in PostgreSQL, metrics time-series in VictoriaMetrics, and continuously-collected container/pod log content in VictoriaLogs. Protect all three: back up the PostgreSQL database, the VictoriaMetrics data volume, and the VictoriaLogs data volume.

Back up your .env first

Before anything else, back up ~/kubewatch-erp/.env. It holds two values that are tied to your running state and cannot be regenerated:

• JWT_SECRET signs login sessions. If it is lost, gateway and auth refuse to start and the app is unreachable until you set a secret again. A restored database is useless without a working stack.
• DB_PASSWORD is baked into the PostgreSQL data volume on first startup. A restored database volume only opens with the password it was created with.

Copy the file somewhere safe (a secrets manager or an encrypted store), and keep it with your database backups:

cp ~/kubewatch-erp/.env ~/kubewatch-erp/backups/env-$(date +%Y%m%d).bak

If you ever rebuild the host, restore this .env exactly as it was. Reusing the original JWT_SECRET keeps existing sessions valid. Generate a new one instead (openssl rand -hex 32) and the stack will still start, but everyone gets signed out and has to log in again. Never change DB_PASSWORD on an install with existing data.

Manual database backup

cd ~/kubewatch-erp
docker compose exec postgres pg_dump \
  -U kubewatch \
  --format=custom \
  --compress=9 \
  kubewatch > backup-$(date +%Y%m%d-%H%M%S).dump

The --format=custom flag creates a compressed, custom-format dump that is faster and more flexible to restore than plain SQL.

Scheduled backups with cron

Add a cron job to back up the database daily:

crontab -e

Add this line:

0 2 * * * cd ~/kubewatch-erp && docker compose exec -T postgres pg_dump -U kubewatch --format=custom --compress=9 kubewatch > ~/kubewatch-erp/backups/backup-$(date +\%Y\%m\%d).dump

This runs at 2:00 AM every day. Make sure the ~/kubewatch-erp/backups/ directory exists:

mkdir -p ~/kubewatch-erp/backups

VM snapshots

Before any upgrade, take a full VM snapshot from your cloud provider's console (AWS EC2 AMI, GCP snapshot, Azure managed disk snapshot, etc.). A snapshot captures everything: the database, config, and images, and it can be restored to the exact pre-upgrade state within minutes.

Restoring from a backup

cd ~/kubewatch-erp

# 1. Stop the application services (keep postgres running)
docker compose stop gateway auth ingestion query alert-engine scaling-engine agent-ops \
  workload-advisor remediation-engine notification live-data clusters incidents pipelines \
  webhook-gateway iac credentials iac-worker cloud dataops analytics devops-agent integrations frontend

# 2. Drop and recreate the database
docker compose exec postgres psql -U kubewatch -c "DROP DATABASE IF EXISTS kubewatch;"
docker compose exec postgres psql -U kubewatch -c "CREATE DATABASE kubewatch OWNER kubewatch;"

# 3. Restore the dump
docker compose exec -T postgres pg_restore \
  -U kubewatch \
  -d kubewatch \
  --format=custom \
  < backup-20260615-020000.dump

# 4. Start all services
docker compose up -d

What is backed up

Backup retention recommendation

• Keep daily backups for 7 days
• Keep weekly backups for 4 weeks
• Keep monthly backups for 12 months

Automate rotation with a script that deletes backups older than your retention window, or use your cloud provider's object storage lifecycle policies (e.g., S3, GCS, Azure Blob).