The nightly backup check had been down for thirteen days. It had paged exactly once, on day one, at the moment it flipped from up to down, and then it had said nothing. The page landed in an ops chat thread among a dozen other messages, was read, was not acted on that night, and the next twelve nights of missing backups arrived in silence.
A dead-man check is the right instrument for a scheduled job: the job pings when it finishes, and if the ping does not arrive, the monitor alerts. Nothing to scrape, nothing to poll, and no agent. We run several dozen of them on our own estate covering cron jobs, hypervisor backups, workflow loops, and off-site syncs. Over two months they found three structural blind spots in the pattern itself, and one incident that was none of the three. This post is what each one looks like and what we built to close it.
Key Takeaways
- A dead-man check only knows about a job that has pinged it at least once; a job that never starts — because its host lost cluster quorum, because a timer never armed, because the scheduler itself was down — never pings, and some monitors report that as up
- A monitor that alerts on state change pages once per flip, so a check that stays down for two weeks produces one message on day one, and a page that is read and not acted on is indistinguishable from a page that never arrived
- Scrape the monitor's own metrics into Prometheus and re-page daily after 24 hours down; the monitor is right about state, it is just quiet about duration
- A self-hosted monitor sits inside the network it monitors, so a WAN outage takes out the thing that would have reported the WAN outage; the only honest coverage is a hosted check pinged from inside, with the alert delivered over a path that does not touch your uplink
- Pin the out-of-band ping from two hosts on two different hypervisor nodes, never both on one, and validate the configuration line after every edit — a single pasted stray character once made every ping fail silently
- "The job ran" is not "the job succeeded": a Proxmox container left locked after a snapshot-delete makes every later backup of that container fail with
CT is lockedwhile the job completes for the rest, and a job-with-errors signal is not the same as a job-never-ran signal - Whatever a hook reports on partial failure is the thing to verify first, because it is the one case nobody tests on the day they wire the hook up
How Dead-Man Checks Work and Why We Chose Them
Each scheduled job gets a unique ping URL. When the job finishes, it requests that URL. The monitor expects a ping every period, allows a grace window past that, and alerts if the window closes without one. A job can also ping a /start endpoint on entry and a /fail endpoint on error, which lets the monitor distinguish "ran slowly" from "ran and failed" from "did not run".
We self-host the monitor as a container on one of the hypervisors, on the LAN only, with checks tagged by class — page, digest, or watchdog — and alerts fanned out through a workflow router to an ops chat group. Cron jobs on the inference host, per-node hypervisor backup hooks, workflow loops, and database backups all ping it. A job that silently stops running is exactly the thing it was built to catch, and for the common case it does.
The three blind spots are all in the uncommon cases.
Blind Spot One: A Job That Never Starts Never Pings
The first gap is the sharpest, and it is structural.
Our hypervisor backups run as a cluster-scheduled job across several nodes. Each node has a hook that pings the monitor as the job progresses. On one night the cluster lost quorum — a node had been leaving and rejoining for a stretch, and then the cluster could no longer form a majority. A Proxmox cluster without quorum does not run scheduled backup jobs. So the job never started on that node. The hook never fired. The monitor received nothing.
That should be the case a dead-man check is for. It was not, for two reasons.
First, the check for that node's backup job was already in a down state from an unrelated partial failure, which meant there was no state change for the quorum loss to produce. The monitor had already said its one thing.
Second — and this is the more general trap — a monitor that stores check state as "new" for a check that has never pinged may report that check as healthy. Ours exposes a metrics endpoint, and a check in the new state reports hc_check_up = 1. A check that was created but whose job never ran even once looks, to anything consuming those metrics, like a check that is fine. The nag we built for the second blind spot cannot see an unarmed check at all.
The rule we took from it: a dead-man check covers a job that has run before and stopped. It does not cover a job that has never run, and it does not cover the scheduler that would run it. The scheduler itself — cluster quorum, the cron daemon, the systemd timer unit — needs its own liveness signal, or you are trusting the thing you most need to distrust. A cluster quorum loss is a hypervisor-level event, and it needs a hypervisor-level alert, not a job-level one.
Blind Spot Two: A Check That Stays Down Pages Once
The second gap is the thirteen-day one, and it is a design decision rather than a bug.
A dead-man monitor alerts on transition. Up to down: page. Down to up: recovery message. Down to down, day after day: nothing, because nothing changed. This is reasonable behavior for a system designed to avoid alert fatigue, and it is why a backup check sat down for thirteen days and produced one message.
The fix is to add duration to a system that only knows state. We stood up a Prometheus scrape of the monitor's metrics endpoint and wrote an alert rule family that fires when a check has been down for 24 hours and keeps firing. Alertmanager gets a route matched on a repage label, placed before the normal page route, that repeats the notification every 24 hours for as long as the condition holds. Check class — page, digest, watchdog — is read from tags on the check, which a small script keeps in sync with each check's channel attachments. Re-run it whenever a check's channels change, or the class drifts.
A canary check proved the loop end to end: a deliberately silent check, a page on day one from the monitor, and a second page from the re-page route a day later.
Two gotchas from building it, each of which cost a real hour:
- The metrics endpoint wanted the project's dashed UUID in its URL, not the compact 32-character code stored in the database; the compact form returned
404. - The alert router we fan out through has a ten-minute deduplication window, so a re-page that lands inside it is swallowed; the daily cadence clears it, a shorter test interval does not.
And the one from blind spot one applies here too: a never-pinged check reports up, so the re-page rule is blind to it. Duration alerting assumes the check was armed.
Blind Spot Three: The Monitor Is Inside the Thing It Monitors
The third gap is the one every self-hosted monitoring setup shares and most do not write down.
Our monitor is a container on a hypervisor on the LAN. When the WAN link goes down, the monitor is fine — it is on the LAN — and every check that pings it from the LAN is fine. The notification path, though, goes out through the WAN to reach a phone. So a WAN outage is an outage the monitor sees nothing wrong with and could not tell anyone about if it did.
The only honest coverage is a check that lives outside the failure domain. We use a hosted dead-man service for exactly one check, pinged every two minutes by a small systemd timer on two hosts that sit on two different hypervisor nodes. The grace is three minutes, so silence from both hosts for roughly five minutes produces a direct message, delivered by the hosted service over the mobile network, to a phone that is not on the LAN. The gate test was mundane and conclusive: stop both timers, wait five minutes, receive the message, start the timers, receive the recovery.
Two rules came out of it. Never put both legs on one node. Two timers on the same hypervisor are one timer with extra steps; a host reboot would page as a WAN outage. And validate the configuration line after every edit. The ping URL lives in a root-only environment file on each host. One edit pasted a stray > onto the end of the line. Every subsequent ping requested a URL that did not exist, the hosted service answered 404 to each, the timer exited cleanly, and nothing anywhere said so. The check went silent and, because the hosted side had never seen a ping succeed after the edit, there was no down-transition on our side to notice. A one-line assertion now runs after any change:
grep -cE '^PING_URL=https://<monitor host>/[0-9a-f-]{36}$' /etc/<pager>.env # must print 1
A page from this lane while the LAN is otherwise healthy means both hosts lost internet egress at once. That points at the gateway or the carrier's equipment, not at anything inside.
The Fourth Case: The Job Ran and Still Failed
None of the three blind spots covers the incident that actually cost the most backup days.
The nightly cluster backup job was ending with TASK ERROR: job errors night after night. Inside each night's log, a handful of containers read:
ERROR: Backup of VM <id> failed - CT is locked (snapshot-delete)
The mechanism is specific to snapshot-mode backups on Proxmox. The backup tool creates a vzdump snapshot of the container, backs it up, and removes the snapshot. On these containers the removal had hung or failed, leaving the container with a lock of type snapshot-delete and a stale snapshot from that night. Every later backup of that container failed at the lock check before doing anything. The job as a whole still ran — it backed up every other container — so the job-level signal was "ran, with errors", and the per-container failure was buried in a log nobody opened.
We found five containers in that state on one node, the oldest locked for eleven days. Four more had failed the same morning and been cleared by hand during the day without anyone connecting it to the pattern. At least nine days of backups were missing for the monitoring host itself and four other guests.
Finding them is a loop over the node's containers, because pct is node-local:
for id in $(sudo pct list | awk 'NR>1{print $1}'); do
l=$(sudo pct config $id | grep '^lock')
[ -n "$l" ] && echo "$id $l"
done
sudo pvesh get /nodes/$(hostname)/tasks --typefilter vzdump --limit 3
Read the job log in text form and grep for Starting Backup of VM|ERROR. The JSON form with a limit returned nothing for us, and a bare container ID matches line numbers.
Clearing one is two commands, and the order of the precondition matters more than the commands:
# only after ps / the task list confirms no vzdump is running
sudo pct unlock <id> && sudo pct delsnapshot <id> vzdump
Unlocking a container while a backup of it is genuinely in progress is how you turn a stale lock into a corrupt snapshot. Confirm nothing is running first, every time. The proof is the next scheduled run.
The part that connects back to the monitor: the per-node hook reports the job. A job that ends with errors and a job that never ran are different events, and what the hook sends on a partial failure is the first thing to verify when you wire one up, not an open question to discover nine days later. Ours sends a failure ping on a backup abort; whether it does so on a job that completes with per-guest errors was unconfirmed at the time.
Honest Limits
These four cases are ours, found on one estate over roughly two months. They are not a complete inventory of ways dead-man monitoring goes quiet. A job that pings success after doing the wrong thing — backing up an empty directory, syncing to a stale target — passes every check here, and only a content-level verification catches it.
The re-page design assumes the monitor exposes metrics that Prometheus can scrape and that you already run Prometheus and Alertmanager. If you do not, a cron job that queries the monitor's API for checks down longer than a day and sends a message is cruder and sufficient.
The hosted WAN check costs a dependency on a third party for the one alert that matters most when everything else is dark. We accept that trade knowingly. It covers egress, not inbound reachability; a public service that is down while the LAN still has internet needs a separate external URL probe.
The snapshot-lock case is Proxmox-specific and snapshot-mode-specific. Stop-mode and suspend-mode backups do not leave this lock. Whether the hook fires on partial failure is something we have flagged, not something we have fixed, and we are not going to claim otherwise.
Where This Fits
Every one of these gaps is the same shape: a control that reports what it was built to report and is silent about the case nobody asked it to cover. A liveness ping says the job finished. It does not say the job started, does not say how long it has been failing, does not say whether its own messenger can reach you, and does not say whether "finished" included every guest. We have written about this shape before — a health endpoint that stays green while the model it fronts falls back to the CPU, and a secret scanner whose zero findings carried no information about history it never scanned.
Designing the monitoring layer so that each of these cases produces a distinct signal — scheduler liveness, duration re-paging, an out-of-band lane, and per-guest backup verification — is managed infrastructure work, and running it day to day is what IT support looks like when it is done on purpose.
If your scheduled jobs ping a monitor and nobody has tested what happens when the job never starts, book a call and we will go through the blind spots against your estate.