Read-only monitoring and backup alerts

Elasticsearch roles · systemd · healthchecks.io · Python

Problem

My SIEM takes a snapshot every night, and Okta events come in all day. Either one could fail without anyone noticing. Before this work, nothing told me if a nightly backup failed or if events went missing.

Two things make this harder. The account that checks backups shouldn't be able to delete or restore them. And a scheduled check can fail by not running at all. If it only alerts when it runs, I'd never hear about that.

What I built

Separate accounts

I use three Elastic accounts. A read-only analyst account does searches and status checks. A named admin account makes changes, and I'm the one who uses it. The built-in superuser is only for setup and emergencies. If a read-only check is refused, it reports the refusal. It never switches to a stronger account.

For backup monitoring, I gave the analyst role two more read-only privileges. One lets it see snapshot status and the other lets it see the snapshot policy.

Two scheduled checks

The gap check from the previous write-up runs every hour. The snapshot check runs once a day, an hour after the nightly snapshot. I scheduled it in UTC so daylight saving time doesn't move it. It fails if any of these rules fail:

1. The snapshot scheduler is running.
2. The policy exists and has the schedule the check expects. If someone changes the policy, the check fails until I update it.
3. There's a successful snapshot since the last one was due.
4. The newest snapshot succeeded on every shard.
5. No failure has been recorded since the last success.
6. The oldest snapshot is within retention. If old snapshots stop being deleted, the disk fills up without warning.

Both checks report to healthchecks.io. It emails me when a check reports a failure, and also when a report doesn't arrive on time.

Only one short line leaves the lab: a status word, rule numbers and counts. It never includes hostnames, snapshot names or event data, and the wrapper script refuses to send anything outside that format. Each check reads its ping URL from its own file, and only my user can read those files.

How I tested it

The role

A script tests what the analyst account can read and what it's refused. The refused list covers creating, deleting and restoring snapshots, and changing, deleting or running the snapshot policy. Running the policy is another way to create a snapshot, so it has to be blocked too.

The refusal tests target a repository and a policy that don't exist. If the account were allowed, Elasticsearch would answer 404, and the script counts that as a failure. Only a 403 passes.

Before the change, 17 of 19 checks passed. The two failures were the new reads, which showed the script catches missing access. After the change all 19 passed, and every write and restore was still refused.

== Allowed
PASS  who am I                                       HTTP 200
PASS  search Okta events                             HTTP 200
PASS  cluster health (monitor)                       HTTP 200
PASS  Kibana: list dashboards                        HTTP 200
FAIL  snapshot status (monitor_snapshot)             HTTP 403 (expected 200)
FAIL  snapshot policy (read_slm)                     HTTP 403 (expected 200)
== Refused
PASS  write a document                               HTTP 403
  ...  (11 more refusals and bad-credential checks, all PASS)

2 check(s) FAILED.

== Allowed
PASS  who am I                                       HTTP 200
PASS  search Okta events                             HTTP 200
PASS  cluster health (monitor)                       HTTP 200
PASS  Kibana: list dashboards                        HTTP 200
PASS  snapshot status (monitor_snapshot)             HTTP 200
PASS  snapshot policy (read_slm)                     HTTP 200
== Refused
PASS  write a document                               HTTP 403
PASS  delete an index (404 = can delete)             HTTP 403
PASS  read security: users                           HTTP 403
PASS  read security: roles                           HTTP 403
PASS  Kibana: create a data view                     HTTP 403
PASS  create a snapshot (404 = can create)           HTTP 403
PASS  delete a snapshot (404 = can delete)           HTTP 403
PASS  restore a snapshot (404 = can restore)         HTTP 403
PASS  create or change a snapshot policy             HTTP 403
PASS  delete a snapshot policy (404 = can delete)    HTTP 403
PASS  run a snapshot policy now (404 = can run)      HTTP 403
== Bad credentials
PASS  no credentials                                 HTTP 401
PASS  wrong password                                 HTTP 401

All checks passed.

The four alert states

For each test I watched the lab's own log and checked my inbox.

I also ran the real snapshot check with a made-up password. It got a 401, exited with an error and sent nothing. An authentication error shouldn't turn into a status report.

Oct 15 13:44:11 lab-server snapshot-check-run.sh: SLM:       RUNNING; policy nightly found
Oct 15 13:44:11 lab-server snapshot-check-run.sh: Newest:    nightly-2026.10.15-<id> SUCCESS, 103/103 shards
Oct 15 13:44:11 lab-server snapshot-check-run.sh: Retention: 12 snapshots, oldest 10 days
Oct 15 13:44:11 lab-server snapshot-check-run.sh: Problems:  0
Oct 15 13:44:11 lab-server snapshot-check-run.sh: RESULT: ok (rules=- state=SUCCESS shards=103/103 age_min=734 count=12 oldest_days=10 tries=1)

# the only thing sent to healthchecks.io:
ok rules=- state=SUCCESS shards=103/103 age_min=734 count=12 oldest_days=10 tries=1

RESULT: LOST or OVERDUE (okta=3 elastic=2 lost=1 ...)     tries=1, exit 1  → down email
RESULT: couldn't run (reason=check tries=9)                 exit 2           → down email
RESULT: ok (okta=56 elastic=56 ...)                         exit 0           → up

What went wrong

The didn't-run test found a real mistake. I stopped the snapshot check's timer and waited for the missing-report email. It never came.

The snapshot check's settings file had the gap check's ping URL in it. Every snapshot report since setup had gone to the gap check's monitor. That monitor gets an "ok" every hour from the gap check, so a skipped snapshot run could never set it off. The failure emails from my earlier tests were real, but they came from the wrong monitor.

The wrapper checked the URL's format, the file's permissions and the file's owner. The wrong URL passed all of them. Now the wrapper also refuses to run if its URL points at the gap check's monitor. I added 3 tests for that, and they fail if I take the guard out. On the right monitor, the didn't-run test then passed. The failed and couldn't-run tests haven't been rerun there yet.

ERROR: SNAPCHECK_PING_URL is the gap check's URL (same check ID as <gap check settings file>); use the snapshot check's own URL
exit 2, check not run, nothing sent

$ python3 -m unittest discover -s elastic/scripts/tests
Ran 84 tests in 57.221s
OK

# with the guard line removed:
FAILED (failures=3)

Earlier in setup, the first two clean runs failed with "missing settings file". I had created the file on the wrong machine. The wrapper refused to run and sent nothing, which is what it's meant to do.

What I learned

• I test a role by what it's refused. A role can pass every read test and still be able to delete backups.
• Refusal tests need to fail loudly when access is allowed. Pointing them at things that don't exist does that.
• A check that only alerts when it runs can't tell me it stopped. An outside service can, as long as each check reports to its own monitor.
• The didn't-run test is the one that caught my mistake.
• These tests prove the alert path. They don't prove a real backup failure gets caught, because causing one needs admin changes. They also can't tell a machine that's off from a check that skipped a run. Both look like silence.