Agent architecture

The only thing you run is the KubeWatch agent. It sits inside your infrastructure, collects data locally, and pushes it to KubeWatch over an outbound HTTPS connection. There are no inbound ports to open, and the agent is stateless and lightweight (< 50 MB RAM).

How the agent collects data

The agent auto-detects its environment and enables the right collectors. It can run on a Docker host, in a Kubernetes cluster, or on edge hardware, and several collectors (GPU, databases, message queues, tracing) run in any of those modes since they connect directly to the thing they monitor rather than through Docker or Kubernetes.

Docker

On a Docker host the agent mounts the Docker socket read-only (/var/run/docker.sock) and collects:

• Every running and stopped container: image, state, status, created time, restart count, published ports, labels.
• Per-container live stats, from Docker's own stats API: CPU %, memory usage/limit/%, network RX/TX bytes, and block I/O read/write bytes.
• Docker networks: driver, scope, subnet, gateway, and which containers are attached to each one (with their IPv4/MAC addresses).
• Container events, and container logs, both live-tailed on demand when you open them in the dashboard and continuously collected in the background so they're searchable afterward. See Log collection below.

Kubernetes

In a cluster the agent talks to the Kubernetes API with a read-only service account. That's one DaemonSet pod per node, with a single elected pod reporting cluster-wide data so nothing gets duplicated. It collects:

• Pods: phase, node, IP, container list with readiness/restarts, owning Deployment/StatefulSet/DaemonSet (recovered from OwnerReferences), and live CPU/memory usage plus network RX/TX (requires metrics-server for the usage figures).
• Nodes: status, roles, kubelet/OS/kernel version, capacity and allocatable resources, live CPU/memory/ephemeral-storage usage, cloud provider/instance type/region/zone (parsed from providerID and topology labels), conditions, and addresses.
• Services and Ingresses: type, cluster/external/load-balancer IPs, ports, endpoint counts, and every Ingress rule (host, path, backend service).
• Deployments, StatefulSets, DaemonSets: replica counts, rollout strategy, container images, labels.
• ConfigMaps and PersistentVolumeClaims: names, key names (ConfigMap values are never read), status, capacity, storage class.
• CustomResourceDefinitions: schema-level info (group, kind, versions, established) for every CRD, always on.
• Cluster events and scheduling failures.
• Pod logs, both live-tailed on demand and continuously collected in the background (see Log collection below).

Three collections are opt-in, each behind its own RBAC flag on the Helm chart, since they're more sensitive by nature. See Kubernetes Agent for how to enable them:

• Secrets (rbac.allowViewSecrets): names and key names only. Values are never read, and kubernetes.io/service-account-token secrets are excluded entirely. The same flag also enables certificate expiry tracking, which parses every Secret's PEM data for X.509 certificates (subject, issuer, validity window, DNS names) without ever collecting the raw certificate or private key bytes.
• RBAC (rbac.allowViewRBAC): Roles/ClusterRoles and their bindings, so you can audit who can do what.
• Custom resource instances (rbac.customResourceReadGroups): full object content (minus managedFields) for CRDs in an explicit apiGroup allow-list, capped at 500 instances per kind.

Log collection

Container and pod logs are collected two ways, independently of each other:

• Live tail, on demand: while you have a container's or pod's Logs tab open in Live mode, the agent streams them to you in real time. This has always worked this way and still does.
• Continuous bulk collection: the agent also tails every running container's and every pod's logs in the background, all the time, whether or not anyone has a Logs tab open, and ships them to KubeWatch on a short interval (every 10 seconds by default). Every line keeps its own real timestamp (from Docker's and Kubernetes' own log timestamping), not the time it happened to arrive. This is what powers the Logs tab's History mode: full-text, field, or raw LogsQL search (source, namespace, pod, container) over your log history, not just whatever's scrolled by since you opened the page.

Host-level metrics (every node)

Independent of Docker or Kubernetes, every agent also reads its own host's /proc filesystem (and statfs's its root filesystem) for metrics that only exist locally to that machine:

• Network RX/TX throughput
• Disk read/write IOPS and byte throughput per physical device
• Context switches per second
• File descriptor usage
• System load average (1/5/15-minute)
• Physical memory and swap usage
• CPU core count and utilization
• Filesystem usage
• Uptime

In Kubernetes this runs on every DaemonSet pod regardless of which one is the elected leader, since this data can't come from anywhere else. In Docker mode it runs on the single agent container. This powers the Overview page's Host Metrics dashboard, which works the same way across Kubernetes nodes and Docker hosts.

GPUs and AI inference

Where present, the agent collects this in any mode (Docker, Kubernetes, or edge) since it talks directly to the hardware or the inference process, not through a container/orchestrator API:

• GPUs, via nvidia-smi: utilization %, memory used/total, temperature, and power draw, per GPU. Silently skipped (not an error) on hosts with no GPU.
• Inference servers, by scraping their own Prometheus metrics endpoint: vLLM, Triton, TGI, KServe, Ollama, DCGM, llama.cpp, and SGLang are recognized out of the box. Configure targets via KUBEWATCH_INFERENCE_ENDPOINTS.

Databases and Kafka

The agent connects directly to these, no exporter sidecar required:

• PostgreSQL (KUBEWATCH_POSTGRES_INSTANCES): version, uptime, replication state, key settings (max_connections, shared_buffers, etc.), session counts, per-database stats (commits, rollbacks, cache hit ratio, rows touched, deadlocks, temp file usage), background-writer stats, and lock counts by mode.
• Kafka (KUBEWATCH_KAFKA_CLUSTERS): per-topic partition offsets and message counts, and per-consumer-group lag, both totalled and broken down per partition. The agent never consumes messages. It only reads broker metadata.

OpenTelemetry Collectors, Cluster Autoscaler, and Ingress-NGINX

• OpenTelemetry Collector (KUBEWATCH_OTEL_COLLECTOR_TARGETS): scrapes a customer-operated Collector's own self-telemetry, receiver/processor/exporter throughput and failure counts, so you can see whether your own tracing pipeline is healthy. This is separate from and unrelated to sending your application's traces to KubeWatch's own OTLP ingest.
• Cluster Autoscaler: auto-discovered in-cluster (no configuration needed), scraped directly over the pod network for node counts by state, unschedulable pods, and scale-up/down history.
• Ingress-NGINX: also auto-discovered, summed across every controller replica, for connection states, config-reload health, and per-Ingress request counts, latency histograms, and payload sizes.

Latency probes

The agent times TCP/HTTP round trips to: any target you configure (KUBEWATCH_LATENCY_TARGETS), every discovered node's kubelet port, and every Docker container's first published host port, plus its own last push round-trip time, so latency to your own workloads is visible without a separate synthetic-monitoring tool.

Edge devices

Running with KUBEWATCH_MODE=edge switches the agent into a reduced-footprint mode built for cameras, sensors, and industrial controllers: no container/pod/node collection at all, just whatever GPU/inference collectors apply plus device-specific battery percentage and local buffer depth. See Edge AI Fleet Observability.

Command Channel and interactive features

Alongside the metrics push, the agent holds an agent-initiated long-poll open (GET /agents/{id}/commands) so KubeWatch can ask it to do something, without ever needing an inbound connection to the agent. Every action here is opt-in, plan-gated, and (for anything mutating) requires an explicit admin approval step in the dashboard before the agent ever sees it:

• Docker: scale a container group up/down, roll back to a previous replica count, restart a single container in place.
• Kubernetes: server-side-apply a workload edit or an arbitrary manifest, roll back a prior apply, manage a Helm release (render a preview, then apply only after a second admin approves the exact rendered output), or trigger a rolling restart (used by Autonomous Remediation).
• Edge devices: an OTA model-update command that invokes a script you provide on the device (KUBEWATCH_MODEL_UPDATE_HOOK). KubeWatch has no generic way to know how your inference stack is deployed on your own hardware, so this hands off to your own logic rather than guessing.
• Interactive pod exec and port-forward: the agent dials an outbound WebSocket to bridge a terminal session or a raw TCP tunnel to a pod, both Enterprise-gated, read-write-approved capabilities.
• Live log streaming: continues to work exactly as described above, on demand, while you have a container or pod's logs open in the dashboard. This is separate from, and doesn't require, the continuous background log collection described in Log collection.

What happens to the data

1. The agent registers on startup and receives a token, persisted locally so restarts resume the same registration instead of creating a new one.
2. It collects a metrics snapshot on an interval (default every 15 seconds, configurable via KUBEWATCH_INTERVAL), and separately batches continuously-collected log lines for a flush on their own interval (default every 10 seconds, configurable via KUBEWATCH_BULK_LOG_FLUSH_INTERVAL), since log volume and metrics volume have very different shapes.
3. It pushes each to KubeWatch over HTTPS, and keeps the command long-poll open for anything the server needs it to do.
4. If a metrics push fails (network blip, server restart), the snapshot is buffered locally (up to 1000 snapshots or 32 MiB of buffered data, whichever limit is hit first, oldest dropped first) and replayed in order once connectivity returns, each one timestamped by when it was actually collected rather than when it was finally delivered, so a gap never gets smeared into a single misleading spike.
5. Your metrics, logs, alerts, and history appear in the dashboard in real time.

Deployment modes

Hosted SaaS: the agent connects to KubeWatch Cloud automatically. You deploy only the agent, and KubeWatch runs and maintains everything else. Your data is isolated per organization.

Self-hosted: you run the full KubeWatch platform on your own infrastructure and point agents at your own instance. No data leaves your network.

Security

• Outbound only: agents open no inbound ports. They initiate every connection themselves, including the command channel.
• Read-only by default: the Docker socket is mounted read-only and the Kubernetes service account's RBAC is read-only out of the box. Secrets, RBAC objects, and custom resource instances are each opt-in behind their own flag, never on by default.
• Never collected at all: Secret and ConfigMap values are never read, only key names, under any circumstance. Certificates are parsed for their metadata only, never their key material.
• Encrypted in transit: all traffic to KubeWatch is over TLS.
• Logs: container and pod log content is collected continuously in the background (see Log collection) so it's searchable after the fact, in addition to the on-demand live-tail view. This is a deliberate product capability, not an oversight. If your organization needs logs to never leave a container's or pod's own host, self-host KubeWatch so that storage stays inside your own network instead of KubeWatch Cloud.
• Write capabilities are opt-in and gated: on Kubernetes, Enterprise plans can opt into specific write capabilities (interactive terminal, port forward, workload edits, manifest apply, Helm release management, autonomous remediation) one at a time, each behind its own RBAC flag, plan check, and admin approval step. See Kubernetes Agent for the full RBAC reference.