← All projects Monitoring · reliability

Knowing when the triage agent stops

systemd · Elastic · healthchecks.io · Python · Bash

What this demonstrates: Monitoring the monitoring. A least-privilege check catches a stopped agent, a failed run and an expiring key, and an outside dead man's switch covers the case where nothing reports. I tested each alert path live except one, and the page says which.

Homelab project: a fake company, real systems, my own test traffic.

In short: my AI triage agent checks for new alerts every 5 minutes. A separate read-only check confirms it's still ticking, that its last run succeeded, and that none of the three keys it has expiry dates for is about to expire. The check reports to an outside service, which emails me on a failure and also when the check goes quiet. I tested the failure paths live.

Problem

A triage agent that has stopped looks exactly like a quiet day: no posts in Slack. There are three ways it can fail:

It doesn't run. The machine is off, or the timer is dead. Nothing on the machine can report that.
It runs but fails. A run crashes or can't reach a service it needs.
It runs fine until an alert arrives. Listing new alerts needs only the SIEM. An expired model API key would only break a run when there's an alert to triage, so a simple heartbeat stays green right up to the moment it matters.

Design

The highlighted boxes are the controls. The check is separate from the agent and only reads. The outside service catches the case where nothing reports at all.

A separate check, not a ping from the agent. It confirms each tick's record actually reached the SIEM. It gives the agent no new outbound destination and no new secret, and it changes nothing in the agent.
It fails on the first of four states: stale (no successful tick recorded in 15 minutes), unit_failed (the agent's last run didn't succeed), key_expiring (a key the agent needs expires within 7 days) and key_expired.
It asks for the agent's service by name. The machine's overall service state already reads "degraded" because of two desktop services that can't run on a headless server. A check on the overall state would always fail, and soon nobody would read it.
Key dates are written into the check. They aren't secrets, so the check never needs to open a key file. The catch is that I have to update a date when I renew a key.
One short line leaves the machine: a status word, counts, minutes and a key's name. The script that sends it refuses to send anything that doesn't fit that format.
The outside service is the dead man's switch. It expects a report every 5 minutes, with 10 minutes' grace. It emails me on a failure, and also when reports stop.

Before and after

The same moment, seen two ways. Here the model API key expires in 3 days. All values are made up, in the check's real output format.

A heartbeat alonesynthetic example

tick: 0 open alerts in 24h, 0 not yet run, running 0
heartbeat: ok                     # green until the first alert hits the expired key

The timer checksynthetic example, real format, two lines and the unit name left out

Window:  last 15 minutes
Records: 3
Unit:    agent service: success
Keys:    anthropic_api_key, 3 days left
SUMMARY fail status=key_expiring records=3 last_min=2 unit=success key=anthropic_api_key key_days=3

# the only thing sent to healthchecks.io:
FAIL status=key_expiring records=3 last_min=2 unit=success key=anthropic_api_key key_days=3 tries=1

The warning arrives a week ahead, while every tick still succeeds.

What I got wrong

An exposed ping URL. Anyone with a check's ping URL can report it healthy. The first URL I created for this check was exposed, so I deleted that check and made a new one rather than keep using it. The new URL lives in a file only my user can read, is passed to the sender without being printed, and is on the list of files my AI coding assistant is never allowed to read.
An old mistake, guarded against. On an earlier check I'd put the wrong check's ping URL in a settings file, so its missed runs could never alert (that write-up). This check's sender refuses to run with any of my other checks' URLs.
A known cost I kept. If the machine is off for more than 15 minutes, the first check after it comes back reports stale, then recovers at the next tick. That's true, so I didn't hide it.

How I tested it

31 unit tests: 18 for the check and 13 for the sender. They cover each state, the format check and refusing other checks' URLs.
Run by hand first: ok, and the outside service showed the check as up.
A failed run: with a stand-in failure the check reported unit_failed. The failure email came at 7:58 PM and the recovery email at 8:02 PM.
The agent stopped: I stopped the agent's timer. The next check reported stale, 16 minutes after the last tick, and the down email came. Then I restarted the timer.
What I didn't test: the check itself going quiet past the grace period. That path is the outside service's own, and I'd already tested it live on two other checks. The design gives the email about 15 minutes after the last report.

What I learned

A heartbeat proves something ran. It doesn't prove the next real job will work. Watching the slow failures too, like a key that's about to expire, turns a surprise outage into a reminder a week ahead.