systemctl list-timers showed our inventory sync timer exactly where it belonged. active. A LAST column from the top of the hour, PASSED a few minutes ago, NEXT on schedule. One layer up, Prometheus reported 138 of 140 targets up, a ratio that had not moved in a week and that nobody had any reason to question. Both facts were true. Neither meant anything, because the service that timer fired had exited status=2 on every single run for six days, and the target file it existed to regenerate had been frozen since the afternoon a batch of guests migrated between nodes on our Proxmox cluster.
The cause was one of the quietest bugs we have hit. The sync script read its inventory with a cluster-wide command and wrote its output with a node-local one. The two halves agreed with each other until a guest moved, and then they silently stopped. This is what the freeze looked like from each layer, the two instruments that reported health throughout, and the rule we now grep for after every migration.
Key Takeaways
systemctl is-active <timer>reports the state of the timer unit, not the result of the service it fires — a timer staysactiveand shows a freshLAST/PASSEDwhile its service fails on every firing- To know a scheduled unit is working, check the service:
systemctl status <svc>,journalctl -u <svc>, or the modification time of the artifact it is supposed to produce - In Proxmox,
pctis node-local andpveshis cluster-wide;pct execandpct pushonly address guests on the node you run them from, and fail after a migration with a missing-configuration error that reads like a deleted file - A script that reads via
pveshand writes viapctlooks correct until a guest migrates, and then its halves disagree — an asymmetric fix is a latent outage - A healthy-looking ratio computed over stale input is not evidence; 138/140 targets up was accurate and hid 18 wrong
nodelabels, one container that had never been scraped, and one phantom down target - Verify a monitoring system's input freshness — file mtime, diff against live truth — before trusting anything it computes from that input
- After any guest migration, grep automation for bare
pctcalls against a container ID that may have moved, and extend the "did it actually run" suspicion from cron to systemd timers
What a Timer's "Active" Actually Reports
The first wrong assumption is the one we had been making for as long as we had been using systemd timers: that an active timer with a recent LAST means the job is working.
It does not. A timer unit and the service it activates are two separate units with two separate states. systemctl is-active inventory-sync.timer answers a question about the timer: is it loaded, is it scheduled, will it fire again. LAST and PASSED in systemctl list-timers record when the timer last triggered, which it did, faithfully, every hour. The timer never missed a firing. The timer was doing its job perfectly. Its job was to start a service, and it started one that failed.
$ systemctl is-active inventory-sync.timer
active
$ systemctl list-timers inventory-sync.timer
NEXT LEFT LAST PASSED UNIT ACTIVATES
<today> 15:00:00 41min left <today> 14:00:00 18min ago inventory-sync.timer inventory-sync.service
Nothing in that output is wrong. Nothing in it carries information about whether the service exited 0. That is the design, and it is a defensible design — a timer's state should not depend on the outcome of the work it schedules. The error is in reading a scheduling fact as a success fact.
We already distrusted cron for exactly this reason. A cron entry that exists and a cron job that ran and succeeded are different claims, and we keep a mechanical check for cron jobs that compares artifact timestamps against a freshness window rather than reading logs. We had not extended that suspicion to systemd timers. It felt as though systemd, with its richer state model, had solved the problem. It had only moved it one unit over.
Six Days of status=2
The service unit told the truth the entire time, to anyone who asked it.
$ systemctl status inventory-sync.service
● inventory-sync.service - Regenerate Prometheus targets from the cluster inventory
Loaded: loaded
Active: failed (Result: exit-code)
Process: ExecStart=/usr/local/bin/inventory-sync (code=exited, status=2)
And the journal held the same two lines, once per hour, for six days:
inventory-sync[…]: Configuration file 'nodes/<node>/lxc/<id>.conf' does not exist
systemd[1]: inventory-sync.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Read that first line carefully, because it is the whole root cause and it is phrased in a way that invites the wrong theory. "Configuration file does not exist" sounds like a missing file — something deleted, a path typo, a permissions change, a Proxmox update that moved a directory. The first stretch of the investigation went to exactly that: checking whether the guest had been removed, whether the config path had changed, whether something had wiped the cluster filesystem's view of it.
The guest existed. Its configuration file existed. It existed under a different node's directory from the one the script was running on, and the command the script used to reach it cannot see across nodes.
The Read Half and the Write Half Disagreed
Proxmox gives you two families of command-line tools, and they have different scopes.
pct — the container tool — is node-local. pct exec, pct push, pct status, and their siblings look up the guest's configuration under the local node's directory in the cluster filesystem, and if the guest's config lives under another node's directory, they fail with exactly the error above. They are not wrong. A guest's config is owned by the node it runs on, and pct is a tool for that node's guests.
pvesh — the API shell — is cluster-wide. pvesh get /nodes/<node>/lxc/<id>/status/current works from any member of the cluster because you address the guest by node explicitly, and pvesh get /cluster/resources returns every guest everywhere with its current node attached.
# node-local: only guests whose config lives on THIS node
pct push <id> ./targets.json /etc/prometheus/targets.json
# cluster-wide: addressed by node, works from any member
pvesh get /cluster/resources --type vm --output-format json
pvesh get /nodes/<node>/lxc/<id>/status/current
Our sync script used both. It built its inventory from pvesh — correctly, cluster-wide, every guest with its node. It then delivered the generated target file into the monitoring guest with pct push — node-locally. For as long as the monitoring guest sat on the node the timer ran from, the two halves agreed and the script worked. The migration moved it. From that moment the read half kept producing a perfect, current inventory every hour, and the write half failed to deliver it, every hour, with exit status 2.
That shape is worth naming because it is not specific to Proxmox. A script that reads with one scope and writes with another looks correct until the environment exercises the difference. A fix that touches one half — making the read cluster-aware while leaving the write node-local, or the reverse — is a latent outage, scheduled for whenever the topology next changes. Topology changes are routine. Migration is how you patch a node.
138 of 140 Was True and Meant Nothing
The second instrument that stayed green was Prometheus itself, and this one is more insidious than the timer because its number was accurate.
Prometheus reported 138 of 140 targets up throughout the six days. That is a real computation over real scrapes. It was also computed over a target file that had not changed since the migration, and a stale input turns an accurate ratio into a confident description of a world that no longer exists. When we diffed the frozen file against the live inventory, the healthy-looking 138/140 had been hiding three distinct problems:
- 18 targets carried the wrong
nodelabel. Guests that had migrated were still attributed to the node they had left. Every per-node dashboard, every per-node alert threshold, every "which host is this on" lookup had been wrong for six days without a single target going down. - One container had never been scraped. It was not in the frozen file. The inventory would have picked it up on the next successful run, and there was no next successful run. It did not appear in the 140 at all, so it could not appear as down.
- One of the two down targets was a phantom. It corresponded to nothing live. It had been counting against the ratio as a real failure nobody could find, which is its own small time sink and a reason the ratio reads as "normal noise" rather than as a signal.
None of those three shows up as a drop in the ratio. Two of them are invisible to the ratio by construction — a target that is not in the file cannot be down, and a target with the wrong label is up. The one visible symptom, a phantom down target, reads as a flaky host rather than as a stale file.
A healthy ratio computed over stale input is not evidence of health. It is evidence that the computation ran. When a monitoring system says most things are fine, the question to ask before believing it is how old its idea of "things" is.
Check the Service, the Artifact, and the Input
Three checks separate the states above, and all three are cheap. We now run all of them for any timer-driven job whose output something else depends on.
Check the service, not the timer. systemctl status <svc> or journalctl -u <svc> --since -1d. The service unit is the one that knows whether the work succeeded, and it says failed on the first screen. If you want this on a schedule rather than on demand, give the service an OnFailure= dependency that fires a notification unit, so a non-zero exit does something louder than writing a journal line nobody reads.
Check the artifact's modification time. A sync job exists to produce a file. If the file is older than the job's interval, the job is not working, regardless of what any unit reports. For Prometheus file-based discovery there is a metric for exactly this — prometheus_sd_file_mtime_seconds — and an alert that fires when time() minus that value exceeds two intervals is the single rule that would have caught our freeze on its second hour. It is the first rule to write.
Check the input against live truth. Periodically diff what the monitoring system thinks exists against what the platform says exists. For us that is the target file against pvesh get /cluster/resources. The diff is a few lines of shell, and it would have surfaced the 18 wrong labels and the never-scraped container directly, rather than leaving them to be found during a post-mortem.
The pattern underneath all three is the same one we keep rediscovering across this estate: assert the property you depend on, not a proxy for it. We depend on a fresh target file. The timer's state, the service's existence, and a ratio over whatever file happens to be there are all proxies. Only the file's age and its agreement with the cluster are the thing itself.
The Asymmetry Rule After Any Migration
Fixing the script was straightforward once the cause was clear: the write half has to address the guest the same way the read half does, by the current node the cluster-wide inventory already provides, rather than by assuming the local one. The more durable fix is procedural.
After any Proxmox guest migration — planned or HA-triggered — we now grep our automation for bare pct calls against any container ID that may have moved:
grep -rnE 'pct (exec|push|pull|enter|status) +[0-9]+' scripts/ systemd/ cron.d/
Each hit is either addressed to a guest that is pinned to that node by design, in which case it gets a comment saying so, or it is a latent failure that will fire on the next migration. There is no third category. The grep takes seconds and the review takes minutes. The alternative was six days.
The same rule applies to any orchestration layer that distinguishes node-local from cluster-wide tooling, which is most of them. Container runtimes on a single host and the same runtime under an orchestrator, storage commands that work only on the node holding the volume, backup clients that assume the guest is where it was last week — every one of these has a command family that silently assumes locality. Migration is the event that tests the assumption.
Honest Limits
This post describes a single sync script on a small Proxmox cluster, and the specific commands are Proxmox commands. The locality asymmetry generalizes; the exact tool names do not. If you are not on Proxmox, the useful exercise is to identify which of your commands are node-scoped and which are cluster-scoped, and to check that any script touching a movable workload uses one scope consistently.
The prometheus_sd_file_mtime_seconds alert covers file-based discovery only. If your targets come from a service-discovery integration that queries the platform directly, there is no file to go stale, and the equivalent failure is the discovery query erroring while Prometheus retains the last good result. The check in that case is the discovery's own failure counters, which are also exposed, and which also nobody alerts on by default.
The three hidden problems — the wrong labels, the never-scraped container, the phantom — are what a six-day freeze produced for us. A longer freeze, or one spanning more churn, produces a different list. The method of finding them, diffing input against live truth, is the transferable part.
And none of this solves the general problem of a timer whose service fails. The artifact-age alert and the failure notification are per-job additions, and every timer-driven job on the estate needs them individually. Any timer added next month without those two things has exactly the same gap.
Where This Fits
This is the same failure shape we wrote about when our local inference server fell back to the CPU and every health check stayed green: a check that confirms a thing is running passing straight through a state where the thing is not working. Liveness is not health. A timer that fires is liveness. A service that exits 0 and a file that is younger than its interval are health, and they are the only two facts here worth alerting on.
Keeping cluster automation honest across migrations, and keeping a monitoring stack's picture of the estate current, is managed infrastructure work. The part where every scheduled job gets a mechanical "did it actually run" gate rather than a log line somebody might read is DevOps engineering.
If your scheduled jobs are verified by the state of their timers, book a call and we will walk through which of them have been failing quietly.