If You Can't Monitor It, You Don't Own It

Server access logs tell me who visited this site. Nothing told me whether they could.

Two weeks ago the disk on this box filled to 100% and nginx started returning 502s for the site itself. I knew because I kept re-curling it between fixes. That's not monitoring — that's a habit. If I'd gone to lunch, nobody would have known until I came back.

The homelab page has had "monitoring dashboard" sitting on the backlog for months. This is that — minus the dashboard vanity. The part that actually matters is the part that wakes you up.

The trap: a monitor that shares a failure domain

The obvious move was one command right next to the site itself:

docker run -d --restart unless-stopped -p 127.0.0.1:3001:3001 \
  -v uptime-kuma:/app/data louislam/uptime-kuma:1

Same box, same Docker, one compose pull away. And completely worthless for the failures I actually fear. OOM killer, swap death, a bad kernel update, an IAM rule that locks me out of the instance — every one of those takes the monitor down with the thing it's watching. A green screenshot isn't monitoring. Monitoring is a witness that survives the crime.

Failures come in two classes, and they need different vantage points:

• Service dead, box alive — the 502 case. nginx is running, the disk is full, certbot is failing quietly, and the box hums along looking fine to anything on it.
• Box dead — everything stops, including any monitor unlucky enough to be on it.

One box can't reliably witness either of its own failures. So the rule became: the thing that watches must not live on the thing being watched.

The layout

The separate node is the same one from the n8n post — the whole reason n8n doesn't sit beside the site itself. Monitoring inherits that reasoning: different failure domain or it doesn't count.

The health gate: the only thing that runs on the box

The heartbeat isn't a ping. A ping says "the kernel still answers." I want "the things that rot quietly are still fine" — so it's a 20-line bash script that checks nginx, the cert expiry, the disk and the last successful deploy, then pushes one overall status to Kuma's push API:

#!/usr/bin/env bash
set -u
BASE="https://status.network-ai.co.za/api/push/${KUMA_TOKEN}"
fail=()

systemctl is-active --quiet nginx || fail+=("nginx-down")

# a 502 on the site itself is the failure that actually matters
curl -fsS -o /dev/null --max-time 10 https://network-ai.co.za/ \
  || fail+=("site-not-200")

cert=/etc/letsencrypt/live/network-ai.co.za/cert.pem
expiry=$(openssl x509 -enddate -noout -in "$cert" | cut -d= -f2)
days=$(( ( $(date -d "$expiry" +%s) - $(date +%s) ) / 86400 ))
(( days < 14 )) && fail+=("cert-expires-in-${days}d")

(( $(df -P / | awk 'NR==2{print $5}' | tr -d %) > 90 )) && fail+=("disk-over-90")

if (( ${#fail[@] == 0 )); then
  curl -fsSG --max-time 10 "$BASE" --data-urlencode "status=up" --data-urlencode "msg=ok"
else
  msg=$(IFS='|'; echo "${fail[*]}")
  curl -fsSG --max-time 10 "$BASE" --data-urlencode "status=down" --data-urlencode "msg=$msg"
fi

And the dead-man's-switch part is free: if the script dies, if cron dies, if the box dies — the push stops arriving, and Kuma marks the monitor DOWN anyway. The failure mode fails safe. That's the whole trick. A heartbeat you have to poll for is a chore; a heartbeat that expires on its own is a witness.

One cron line:

* * * * * /usr/local/bin/health-gate.sh >/dev/null 2>&1

The --max-time 10 matters more than it looks. A hung curl without a timeout stacks a new run every minute until the box is out of processes — exactly the kind of self-inflicted outage this script exists to catch.

Kuma on the other node

Uptime Kuma itself lives on the separate node, behind nginx with TLS, bound to localhost like everything else:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    container_name: uptime-kuma
    volumes:
      - uptime-kuma:/app/data
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
volumes:
  uptime-kuma:

Three monitors, two failure domains:

• Push monitor — the health gate's heartbeat, 60-second grace.
• HTTPS: network-ai.co.za — catches nginx death, expired TLS, DNS pointed somewhere stupid, and the 502 class from outside, which is the exact blind spot from the day the disk filled.
• HTTPS: the n8n webhook endpoint — because an alerting pipeline nobody checks is the most optimistic kind of monitoring.

Kuma's built-in status page goes on a subdomain — status.network-ai.co.za — and deliberately lives off the broken box. When the site is down, the status page is the one thing still up telling people so. It replaces the "is it just me?" messages I used to get on WhatsApp, which is the MSP version of this same problem.

Turning DOWN into an actual page

Kuma's webhook feeds n8n — the same instance from the automation post. But a raw DOWN-to-Telegram pipeline trains you to ignore alerts the first time something flaps during a docker compose pull. So the workflow waits, re-checks, and only pages if the failure was real:

{
  "trigger": "webhook: kuma.monitor.down",
  "steps": [
    { "action": "wait", "seconds": 60 },
    { "action": "http_request", "url": "kuma /api/monitor/{id}", "expect": "status=down" },
    { "action": "if_still_down", "then": ["notify_telegram", "notify_email"] },
    { "action": "otherwise", "then": ["log_as_flap"] }
  ]
}

The DOWN message from the push monitor carries the health gate's reason string — cert-expires-in-12d or disk-over-90 — so the first message is already the diagnosis, not just "something is wrong, go look."

And per the rule from the n8n post: the whole thing ships with an Error Workflow. If n8n can't reach Telegram, it posts the failure to email instead. A silently broken alerter is worse than no alerter, because now you also trust something you can't trust.

Watching the watcher

There's one more ring out, and it's free. If the separate node dies, Kuma dies with it — and the witness to my EC2 just vanished without telling anyone. So one free-tier external monitor points at the status page's heartbeat URL and pages Telegram directly, bypassing n8n entirely.

The principle: every layer alerts through a channel that doesn't depend on the layer below it. Kuma routes through n8n because Kuma and n8n failing together is the case the outer ring exists for. Two hops to Telegram for routine alerts, zero hops for the catastrophe.

Day one

The first DOWN fired before the first real incident: a docker compose pull restarted Kuma for 40 seconds, the push timed out, and DOWN went into the queue. The 60-second re-check saw it back UP before Telegram ever fired — logged as a flap, nobody woken up. The grace step earned its keep before the feature ever had to be useful.

The motivating case would have been different. On the day the disk filled, the external check on network-ai.co.za would have caught the 502s in the first minute, with the reason attached, on my phone — while I was still re-curling it by hand.

The cert tripwire is the quiet one. certbot.timer runs every morning and renews at 30 days out — I checked. But a timer that runs isn't a renewal that succeeds, and cert expiry is the failure that looks exactly like an outage to a user: a browser scary page, no server logs to grep, nothing in error.log. Fourteen days before expiry, the health gate starts reporting DOWN with cert-expires-in-13d whether or not certbot knows it's broken.

What it cost

• Money: $0. Uptime Kuma is free and self-hosted on a node I already pay for; the outer ring is a free tier doing exactly one job.
• RAM on the tight box: ~0. The 908MB EC2 runs one cron line; the dashboard's ~100MB deliberately lives somewhere else — the box only has 908MB to give in the first place.
• Maintenance: docker compose pull on the node, and rotating the push token if it ever leaks. Push URLs are bearer tokens — treat them like the credentials vaulted in the n8n post, not like a URL you paste into a README.

Lessons

1. A monitor that dies with the server is a screenshot. Put the witness in a different failure domain — a different box at minimum, a different provider if you can. One docker run on the box you're watching is motion, not monitoring.
2. Heartbeat beats ping. Bake the checks that rot quietly — containers, cert days, disk — into what you push. One token, whole-system truth, and a failed push fails safe to DOWN.
3. Grace before page. Wait, re-check, then wake a human. Flapping trains people to ignore alerts, and ignored alerts are slower than any outage.
4. The status page must survive the incident it describes. Host it off the broken box, then point one free external monitor at it. Every layer needs somebody watching from further out.
5. Cert expiry is a monitored service. A running timer is not a successful renewal. Put a day-count tripwire in the gate so expiry shows up as DOWN two weeks early instead of as a browser warning on launch day.
6. Alerts route through n8n with an Error Workflow. Same lesson as the automation post: the alerter itself can rot, and the one thing that must never fail silently is the thing that tells you things are failing.

The site hasn't had an outage since this went in, which is the wrong way round — monitoring earns its keep on the bad day, not the good ones. But the blind spot is closed: something that doesn't live on this box now notices, within a minute, when the box or its services stop being what they're supposed to be.

You don't own the stack because you can SSH into it. You own it when something that isn't on it notices it's broken before your users do — and knows exactly who to tell.