Alert Not Firing
Debug why an alert rule isn't triggering when you expect it to.
If you've created an alert rule but it never fires even when the metric exceeds the threshold, work through these checks.
Check 1: Is the rule enabled?
Go to Alerts in the dashboard and find your rule. There's a toggle on each rule, make sure it's switched on (indigo). Newly created rules default to enabled, but it's easy to accidentally disable one.
Check 2: Are you using a metric that actually exists?
Some conditions that sound plausible aren't collected by KubeWatch yet: disk usage percentage, a pod crash-loop
detector, container OOM-kill detection, and container image-pull-error detection don't exist as alert
conditions today. A rule built around any of these will never fire, silently, because nothing ever writes that
data. Docker container CPU and memory usage, by contrast, are collected (via Docker's own stats API, the same
data docker stats shows) exactly like Kubernetes pod CPU/memory, so a rule using one of the real metric keys
below works the same way for both runtimes.
Check 3: Is the metric name spelled correctly?
Metric names are case-sensitive and must exactly match one of the options in the rule's Metric dropdown.
Correct metric names:
| Metric | Key |
|---|---|
| Docker container CPU usage percentage | container_cpu_percent |
| Docker container memory usage percentage | container_memory_percent |
| Docker container memory usage, in MB | container_memory_usage_mb |
| Kubernetes pod CPU usage percentage | pod_cpu_percent |
| Kubernetes pod memory usage percentage | pod_memory_percent |
| Kubernetes pod restart count | pod_restart_count |
| Kubernetes node readiness | node_ready (1 = Ready, 0 = NotReady, so a "node not ready" rule needs operator < and threshold 1, not >) |
| Minutes since an agent last pushed | agent_silent_minutes |
Common mistakes: cpu_percent or cpuPercent (missing the container_/pod_ prefix, or camelCase), cpu
(too short), memory_usage (wrong name), node_not_ready (the key is node_ready, inverted).
Check 4: Is the current metric value actually exceeding the threshold?
Check the current value in the dashboard:
- Go to Containers → click the container you're monitoring
- Open the Metrics tab
- Read the current CPU % or memory value
If the current value is below your threshold, the alert is working correctly, it's just not fired because the condition isn't met.
If you're trying to test an alert, you can temporarily lower the threshold to a value you know will be exceeded.
Check 5: Is the condition true at evaluation time?
KubeWatch evaluates each rule every cycle against the latest sampled value. An alert fires the moment the condition is true and resolves the moment it isn't, there's no "sustained duration" or "for 5m" window. So if a metric only crosses the threshold briefly between samples, it may not get caught. While testing, lower the threshold or have the source report more often so it's easier to trigger.
Check 6: Is a notification channel configured?
An alert can be in firing state even if you never received a notification, if no notification channel is configured or the channel is broken.
- Go to Alerts → find your rule → check the Channels field
- Go to Integrations → verify the channel is connected (teal status indicator)
- Send a test notification from Integrations → [channel] → Test
Check 7: Check alert history
Go to Alerts → History. Filter by your rule name. If the alert fired but you didn't receive a notification, the issue is with the notification channel, not the rule itself.
If the alert doesn't appear in history at all, the condition was never met for the configured duration.
Check 8: Is the agent reporting metrics for this container?
If the container you're targeting stopped reporting (agent disconnected, container stopped), the alert won't evaluate. Check:
- Containers page, is the container listed and does it have recent metrics?
- Agents page, is the agent Connected with a recent "last seen" timestamp?