Auto Scaling

Auto Scaling turns KubeWatch from an observability product into an observability plus control product. Alongside watching your workloads, it can scale them. You set a per-workload policy (thresholds, replica bounds, cooldowns) and the Scaling-Engine acts on the same metrics you already see on your dashboards.

The same policy model backs both runtimes, but what happens underneath differs sharply:

Creating a policy

Navigate to Auto Scaling, then New policy:

New policies start in dry-run (see below). Cooldowns default to an asymmetric pattern, quick to scale up and slow to scale down, because reacting late to a load spike is worse than keeping a spare replica around a few extra minutes.

Dry-run first

Every policy can run with dry_run on: the Scaling-Engine evaluates the thresholds against live data and records exactly what it would do, including the rendered HorizontalPodAutoscaler YAML or the docker.scale command and chosen host, but applies nothing. This is the recommended first week of any policy's life. When you are confident, toggle Go live.

Kubernetes: HPA and Karpenter

For a live Kubernetes policy, KubeWatch renders a standard autoscaling/v2 HorizontalPodAutoscaler (and a Karpenter NodePool when a node-pressure target is set) and the in-cluster agent server-side-applies it with the kubewatch-scaling-engine field manager. The cluster's own HPA controller then moves the replica count. KubeWatch never writes the replica count directly, so it never fights the HPA, and scaling keeps working even if KubeWatch is unreachable.

KubeWatch-generated NodePools default to the conservative WhenEmpty consolidation policy, so a cost-saving node consolidation can't terminate a disruption-sensitive workload you didn't opt in. Node-level scaling also requires Karpenter to already be installed. The policy editor doesn't hide or disable the option when Karpenter is missing, it just warns you; enabling it anyway fails to apply with a clear error, and the policy card offers a one-click way to remove the node-pressure target and fall back to a plain HorizontalPodAutoscaler.

Docker: placement and the managed load balancer

Standalone Docker has no HPA, so KubeWatch owns the whole loop. On scale-up, the engine picks a placement, the host with the most CPU/memory headroom, subject to max replicas per host, and then enqueues a docker.scale command. The agent clones an existing replica (same image, env, and mounts) to add containers, or stops the newest one to remove them.

Because Docker has no Service to route traffic, KubeWatch can run a managed load-balancer pool (opt-in, set KUBEWATCH_MANAGED_LB=true on the agent): a lightweight Caddy reverse proxy on the host, one upstream pool per container group, with active health checks so a new replica receives no traffic until it passes. When the pool is active, the policy shows routing managed. Without it, KubeWatch adjusts the container count and you route to the replicas yourself.

Rollback

Every action that changed something gets a Roll back action in the decision log, which is append-only and never pruned. The behaviour differs by runtime, and the UI says so at the point of action:

• Kubernetes: a clean, unqualified restore. KubeWatch re-applies the previous HPA/NodePool spec and the cluster reconciles. No pods are recreated.
• Docker: restores the previous replica count. Rolling back a scale-up cleanly removes the newest containers, but rolling back a scale-down starts fresh containers from the image instead, since the originals are already gone, and the confirmation says so plainly.

Safety

• Cooldowns prevent flapping, the single most important field here. Kubernetes inherits this as HPA stabilizationWindowSeconds, and Docker enforces it directly in the engine.
• Hard bounds cap the blast radius. Setting max = current is a legitimate "scaling paused" configuration.
• Approval gates (approval_required) hold high-stakes actions for a human, delivered through your existing notification channels.
• The decision log is the single source of truth for what happened and why: every evaluation that resulted in an action, including dry-run previews, applies, and rollbacks, in chronological order. It is append-only, so a rollback is a new entry referencing the one it reverts, never a rewrite.

How agents receive instructions

KubeWatch agents are push-only and never accept inbound connections, a property worth preserving for a self-hosted product. The autoscaler keeps it that way: agents receive commands over an agent-initiated long-poll, which is just one more outbound HTTPS request, indistinguishable at the firewall from the existing metrics push. The Kubernetes path doesn't even use this channel, since it applies native objects directly.

Required Kubernetes RBAC

The in-cluster agent needs a small set of additional permissions to apply autoscaling objects, and nothing more. Notably it has no verb on pods or deployments directly, and no delete:

- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["karpenter.sh"]
  resources: ["nodepools"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]

The KubeWatch agent Helm chart includes these rules.