The credential was restored, the offsite backup agent authenticated, and the first thing it did with its new access was delete two backups. Not stale copies of something held elsewhere — the two oldest snapshots the system had, which existed on that one destination and nowhere else. The change carried a written acceptance gate: the three offsite backups present before the fix must still be present after it. The fix worked. The gate failed. Both were true at the same moment.
The tool was Home Assistant, backing up to local storage, a NAS, and an offsite Cloudflare R2 bucket with a retention of three copies. The mechanism is not specific to Home Assistant, which is the reason to write it down: any backup system with one retention rule applied across several destinations will do the same thing the moment a destination that fell behind catches up. This is what happened, the theory that was wrong, and the order of operations that would have avoided it.
Key Takeaways
- A backup destination that has been failing holds the oldest copies the system knows about, so the moment it reconnects, its contents are the first candidates for deletion under any count-based retention rule
- Retention evaluated against the whole backlog rather than per destination treats a lagging destination's unique history as surplus, and nothing in its log distinguishes that from a routine prune
- Triggering a manual backup instead of the scheduled one is not a mitigation: retention also runs when the agent reconciles after reconnecting, so the prune fires on authentication, not on schedule
- The safe order is raise or suspend retention, restore the agent, confirm the new upload landed, then restore retention — and copy anything that exists only on the stale destination out first if it matters
- An acceptance gate that names the objects which must survive turns "the backup job succeeded" into "the backups we cared about are still there", which is a different claim, and is worth writing even when it fails
- Before deleting anything described as regenerable or redundant, ask what currently runs from it or depends on it; "reconstructible" has never meant "unused"
- Deletions with a delayed fuse — a service running from memory until its next restart, a snapshot nothing reads until a restore — are the expensive ones, because the failure arrives hours or weeks after the cause
Retention Is Evaluated Against the Backlog, Not the Destination
The offsite leg had been failing for a week. Its credential for the bucket had gone bad on the fourteenth, and every scheduled run since had succeeded locally, succeeded to the NAS, and failed offsite. The failure was logged; it did not reach anyone until the week was over, which is its own lesson and not this one.
Here is the inventory on the day we noticed, by destination:
local 08-19 08-20 08-21
NAS 08-19 08-20 08-21
offsite 08-12 08-13 08-14 # credential failing since 08-14
The local and NAS legs had rolled forward normally under copies: 3, each holding the three newest backups and nothing older, exactly as configured. The offsite leg, frozen on the day its credential broke, still held the three from before the break — the only surviving copies of 08-12, 08-13, and 08-14 anywhere.
The retention configuration that produced this is unremarkable:
retention:
copies: 3
Three copies, no age limit. What that configuration does not say is three copies of what, counted where. We had been reading it as "each destination keeps its three newest". The engine reads it as "the system keeps its three newest, and each destination is brought into line with that set". The two readings agree as long as every destination succeeds on every run. They diverge the instant one falls behind, in the worst direction: the lagging destination's unique history is precisely what the engine identifies as surplus.
The moment the credential was corrected and the offsite agent authenticated, the engine reconciled that agent against the current backlog of six. 08-12 and 08-13 were outside the window of three. They were deleted from the bucket. The log line for that deletion was indistinguishable from every routine prune line before it. There is no "this object exists nowhere else" check in a count-based prune, because the prune is not asking that question.
Write the mechanism down in one sentence: a destination that has been failing holds the oldest copies the system knows about, so restoring it makes them the first thing retention deletes.
The Fix Succeeded and the Gate Failed at the Same Time
The change had a written acceptance gate, which is the practice we apply to our own infrastructure work. It read, in effect:
Gate: the three offsite backups present before the change
(08-12, 08-13, 08-14) must still be present after
the credential is restored.
credential restored ............ PASS
agent authenticated ............ PASS
new upload landed offsite ...... PASS
08-12 still present ............ FAIL
08-13 still present ............ FAIL
Every line the fix was responsible for passed. The two lines the gate was written to protect failed. Had the gate been "the offsite backup job succeeds", the change would have been closed as done and the loss would have surfaced only when somebody needed one of those snapshots.
That is the value of writing the gate in terms of the objects you care about rather than the operation you are performing. It did not prevent the loss — a gate is a check, not a guard — but it made the loss a known fact that day instead of a surprise during a restore months later. The uncomfortable part is that the gate was right and we still lost the data, because we evaluated it after the change instead of designing the change so it could not fail.
The Manual-Backup Theory Was Wrong
We did anticipate the risk, in part. The concern going in was that the next scheduled backup would trigger a prune, and the obvious mitigation was to avoid the scheduled run: restore the credential, immediately trigger a manual backup under our own control, confirm it landed, and inspect the destination before the schedule could fire.
That theory assumed retention runs on a schedule. It does not only run on a schedule. Retention also runs when an agent reconciles, which is the first thing a freshly authenticated agent does. The prune fired on authentication, before any backup was triggered at all. By the time the manual backup began uploading, the two tarballs it was meant to protect were already gone.
The wrong turn has a familiar shape: we modelled the dangerous operation as the one visible on the schedule, and the system had a second trigger for it that we had not modelled. Reconciliation is a comparison of what a destination holds against what the catalogue says it should hold, and a comparison that finds surplus deletes the surplus, whenever the agent can talk to its destination.
Choosing a manual backup over the scheduled one is not a mitigation for this class of problem. The only mitigations that work operate on retention itself, or on the objects at risk, before the agent reconnects.
The Order of Operations That Would Have Saved Them
The procedure we now apply before re-pointing or re-credentialling any backup agent that has fallen behind:
1. List what the stale destination holds. Identify every
object that exists there and nowhere else.
2. If any of those objects matter, copy them out to a
holding location that no retention rule governs.
3. Raise retention well above the total backlog
(copies: 3 -> copies: 20), or suspend automatic
cleanup entirely if the tool allows it.
4. Restore the credential or endpoint. Let the agent
authenticate and reconcile.
5. Confirm the new upload landed by listing the
destination, not by reading the job status.
6. Restore retention to its intended value.
7. List the destination again. Compare against step 1.
Steps one and two cost nothing and would have been sufficient on their own. Listing the bucket took seconds; copying two tarballs out would have taken minutes. The copy-out is the step with no failure mode: retention rules can be misread, suspended cleanup can be re-enabled by a restart, a raised copy count can be lowered by a config sync, but a duplicate in a location the retention engine does not know about cannot be pruned by it.
Step five is deliberately "list the destination" rather than "check the job succeeded". The job status is the tool's opinion of what it did; the listing is what is there. It is the same distinction as a liveness check staying green through a complete outage: assert the property you depend on, not the operation that was supposed to produce it. Step seven closes the loop against the gate with evidence produced mechanically rather than asserted from memory.
This Is Not a Home Assistant Problem
The tool in our case was Home Assistant and the offsite destination was Cloudflare R2, and neither is the point. The pattern needs three ingredients, and most backup systems have all of them: one retention rule applied as system-wide policy; several destinations; and independent failure, where any one destination can fail while the others continue — the entire reason to have several.
Given those three, a destination that fails for longer than the retention window will hold unique history the others have already pruned, and reconnecting it will delete that history, unless the tool evaluates retention per destination or refuses to delete the last copy of anything. Many do neither, and the documentation rarely says which. Get three answers by experiment rather than by reading:
- When a destination reconnects after a gap, does retention run immediately, or only on the next scheduled job?
- Is the retention count evaluated per destination, or against the catalogue as a whole?
- Will the tool delete an object from a destination when it is not present on any other?
A throwaway configuration with a short retention, one destination broken deliberately, and the broken one reconnected while you watch its listing answers all three in an afternoon — cheaper than learning them on the destination holding your only copy of something.
Before Deleting Anything "Regenerable", Ask What Runs From It
The backup incident is one instance of a rule we had already written down after an earlier, unrelated purge. The two fail the same way.
Earlier in the year we reclaimed disk space on a workstation by deleting build artifacts and dependency directories across a projects tree — the node_modules and virtual-environment directories every developer treats as disposable. The pre-flight check was careful by the usual standard: every one of 113 paths was confirmed untracked by version control and regenerable from a lockfile.
Nine of those directories were the runtime dependency trees of small services that were running at the time. They kept running all day, because each process had already loaded its dependencies into memory, and would have failed on the next restart with no obvious connection to a purge hours earlier. A virtual environment deleted from under a live process was silently rebuilt on disk while the process reported a missing module that was, by then, present again. The disk looked fine. The process was broken.
"Gitignored and regenerable" means reconstructible. It has never meant unused. The check we ran answered a question about source control; the question that mattered was whether anything currently ran from the directory. Three practices followed, and each applies to backup destinations as much as to build directories.
Guard deletes by what is live, not what is tracked. Our purge script now checks, in order: directories any configured service runs from, directories a live process references, directories with open file handles, and only then whether the path is untracked. Each of the first three would have stopped the original incident. The fourth, alone, approved it.
Never gate a delete from inside the same shell invocation that performs it. We lost five environment files to a cleanup whose listing step had found them and whose removal ran regardless, because the two were one command. Run the check, read its output, then delete as a separate step. The pause is the control.
Before destroying or repurposing anything — a machine, a tenant, a container, a bucket — sweep for references first, including those that live only in a running process's environment. A grep across repositories misses the route set as an environment variable on a live container and never committed; we found one pointing at a tenant hours before it was to be repurposed. Snapshot first, and state the evidence — what you listed, what referenced it, what runs from it — before the delete, not after.
The backup incident broke the same rule from the other direction. We treated the offsite tarballs as safe because the system was configured to keep three copies, and "configured to keep" is as much the tool's opinion as "untracked by git". The evidence that would have protected them was a listing showing they existed nowhere else, and we did not produce it.
Honest Limits
This post describes one tool's retention behaviour as we observed it on one occasion. We did not reconstruct why the third stale backup survived the reconciliation that deleted the other two; the procedure above assumes every unique object on a stale destination is at risk, which is the only safe assumption.
The seven-step procedure assumes the tool lets you raise or suspend retention without a restart, and that a restart does not reset it. Some tools reload retention from a synced configuration on start, which would quietly undo step three. Check that first, or rely on the copy-out in step two, which has no such dependency.
Per-destination retention, where a tool offers it, removes this failure but lets a destination hold copies older than your stated window, which matters under a data-retention obligation. Know which mode you are running.
Finally, none of this covers whether the surviving backups can actually be restored. A listing proves presence. It does not prove the tarball is complete or that a restore from it produces a working system. That is a separate verification with its own failure modes.
Where This Fits
Backups to several destinations are the right design, and this failure is not an argument against it. It is an argument for treating retention as a system-wide policy with a documented scope, for gates that name the objects that must survive, and for never reconnecting a lagging destination without first asking what it holds that nothing else does.
If you are deciding where those destinations should live in the first place, we wrote separately about the self-hosted versus cloud trade-off; the retention problem exists in both. Designing and operating backup pipelines that survive a destination falling behind — retention scoping, reconnection procedures, listing-based verification — is managed infrastructure work within our managed services practice.
If your backups go to more than one place and you have never tested what happens when one of them falls behind and catches up, book a call and we will walk the reconnection procedure with you before it runs on its own.