The restore finished. rsync exited 0. A verification loop walked all 97 git repositories in the restored projects directory and reported every one of them healthy, and we said so, out loud, to the person waiting on the machine. Fifty-two of them were broken. Each had a .git directory containing objects and refs and nothing else — no HEAD, no config, no index — which is to say a directory git recognizes well enough to error on and not well enough to do anything with.
The source had vanished partway through the copy. We were restoring a home directory on one of our own workstations from a mounted APFS local snapshot, on a volume that was 98 percent full, and macOS reclaimed that snapshot under disk pressure while rsync was still reading from it. rsync did not see an error. It saw its remaining source cease to exist, finished what it had, and exited clean. This is the story of how a copy tool, a verification loop, and a confident report all said "done" about a restore that was half finished, and the rules that came out of it.
Key Takeaways
rsynccan exit 0 when its source disappears mid-run — the exit code of a bulk copy proves the tool finished its run, not that the destination matches a source that no longer exists- macOS APFS local snapshots are the first thing the system reclaims under disk pressure, and they are frequently the very source you are restoring from; a restore onto a nearly full volume is doubly suspect
- A path existing proves nothing about its contents —
[ -e repo/.git ]was true for all 97 repositories, including the 52 with noHEAD,config, orindex - Never let a failing command's empty output be read as a clean result; a
git status --porcelainthat errors produces the same empty stdout as a clean tree, and a check that does not test the exit status is inverted - Verify a restore by exercising each artifact —
git rev-parse HEAD,python -c "import x",sqlite3 db .tables,openssl x509 -noout -in cert— never by checking that it is present - An alphabetical boundary between fine and broken (a–h intact, i–z broken) is the signature of a sequential process that stopped early; look for an ordered pattern before assuming random damage
- Copy the irreplaceable, non-reproducible items first — credentials, secrets, machine-local configuration — and bulk data second; that ordering is what made this recovery survivable
The Exit Code That Was Technically Correct
The first wrong theory is the one everybody holds about rsync until it fails them: that exit 0 means the destination now matches the source.
It means something narrower. rsync's exit status is a report on rsync's own run — whether it encountered errors it classifies as errors. It has a code for files that vanish between listing and transfer, and a code for partial transfers, and in a normal run on a healthy source those codes are what you get when things go wrong. Our run was not normal. The snapshot was unmounted out from under the process, and rsync, which had already built its file list and was partway through transferring, came back with 0.
We did not fully reconstruct which internal path produced a clean exit rather than a partial-transfer code, and we are not going to pretend we did. The operational fact is the one that matters, and it is the one to write on the wall: a copy tool's exit status is a statement about the tool's run, not about the equivalence of two trees. If the thing you need to know is "is the destination complete", the exit code is not that fact, and no amount of reading rsync's manual converts it into that fact. The same is true of cp -R, tar over SSH, a sync client, or a backup agent: each reports on its own execution, and completeness has to be checked separately, against the destination, after the tool is done.
The Snapshot That Was Reclaimed While We Read From It
The second wrong assumption was that a mounted snapshot is a stable source for the duration of a copy.
macOS keeps local APFS snapshots of the boot volume, and they are the fastest restore source available when a home directory has been damaged — mount the snapshot, rsync the directories you need back into place. tmutil listlocalsnapshots / shows what exists. The catch is that these snapshots are also the first thing the operating system reclaims when the volume runs low on space. They are explicitly designed to be disposable.
Our volume was at 98 percent. The restore was writing into the same volume the snapshot belonged to, so every file rsync copied consumed free space that was keeping the snapshot alive. The copy was, in a precise sense, destroying its own source as it ran. At some point macOS crossed its threshold and purged the snapshot. There was no dialog, nothing surfaced to the shell, nothing in rsync's output. The source simply stopped.
The rule we took from this: a restore under disk pressure is doubly suspect. The pressure is probably why you are restoring in the first place, the local snapshot is probably your source, and the restore makes the pressure worse. If the volume is near full, clear space first or copy to a different volume, and treat the snapshot as something that may vanish at any second rather than as a stable disk.
The Verification Loop That Inverted Itself
The part that turned an incomplete restore into a false report was our own check. It looked reasonable. It was wrong in two independent ways, and each one alone would have produced the same false "all healthy".
for r in ~/projects/*/; do
if [ -e "$r.git" ]; then
dirty=$(git -C "$r" status --porcelain | wc -l)
echo "$r dirty=$dirty"
fi
done
Inversion one: presence is not integrity. [ -e "$r.git" ] asks whether something exists at that path. For all 97 repositories it did. For 52 of them what existed was a directory with objects/ and refs/ and nothing that would let git locate a branch, a working tree, or its own configuration. The test passed every one of them through to the next line.
Inversion two: an error's empty output scored as clean. git status --porcelain on a repository with no HEAD does not print a list of changes. It prints an error to stderr and exits non-zero. Its stdout is empty. wc -l counts zero lines. The loop printed dirty=0, which is exactly what a clean, intact repository prints. The pipeline discarded the one signal that distinguished them — git's exit status — because a pipe returns the status of its last command, and wc was perfectly happy.
So 52 broken repositories and 45 intact ones all reported dirty=0, and we read 97 lines of dirty=0 as 97 clean repositories.
The corrected loop exercises the repository and checks the exit status explicitly:
for r in ~/projects/*/; do
if git -C "$r" rev-parse --verify 'HEAD^{commit}' >/dev/null 2>&1 && [ -f "$r.git/config" ]; then
echo "OK $r"
else
echo "BAD $r"
fi
done
git rev-parse --verify 'HEAD^{commit}' cannot succeed unless HEAD resolves to a commit object the store actually contains — the ^{commit} suffix is what forces git to look the object up rather than merely read the hash out of the ref, and a bare git rev-parse HEAD will happily print a hash whose object is gone. [ -f .git/config ] catches the case where HEAD survived and the configuration did not. The condition is the command's success, not the shape of its output. Run against the restore, this version listed 52 lines of BAD immediately.
The general rule is sharper than "write better checks": never let a failing command's empty output be read as a clean result. Any verification that scores "no output" as "no problems" is inverted — the check's failure mode is indistinguishable from success. If you are counting lines, set pipefail and check the status. If you are grepping for errors, confirm the command actually ran.
Verify by Function, Not by Presence
The corrected check is one instance of a rule that applies to every kind of artifact a restore brings back.
A path existing tells you a copy tool created an entry at that location. It tells you nothing about whether the bytes behind it are complete, consistent, or usable. The only check that establishes usability is the one that uses the artifact the way its consumer will:
- A git repository:
git -C <repo> rev-parse HEAD, and[ -f <repo>/.git/config ] - A Python package or environment:
python -c "import <module>" - A SQLite database:
sqlite3 <db> .tables, or a query against a table you know exists - A certificate:
openssl x509 -noout -in <cert> - An archive: list it, or extract one known member
- A virtual machine image or container layer: boot it, or at least have the hypervisor parse the header
Each is cheap, each is specific to the artifact, and each fails loudly on a truncated copy where a stat call succeeds. A restore is not verified until each item has been exercised. Presence is the hypothesis. Function is the evidence.
The Alphabetical Boundary Was the Whole Story
Once the corrected loop produced its list, the pattern in it was immediate and it told us more than any log did.
The 45 intact repositories were, alphabetically, a through h. The 52 broken ones were i through z. There was one boundary, it was sharp, and it sat exactly where a sorted sequential copy would have stopped if its source disappeared at that moment. rsync processes a directory tree in sorted order; the snapshot vanished while it was somewhere in the i's; everything before that point had been fully transferred and everything after had not.
That observation ruled out an entire class of theories before we spent time on them. Random damage — a failing disk, filesystem corruption, bit rot in the snapshot — produces scattered failures, not a clean split in a sorted list. An alphabetical or otherwise ordered boundary between fine and broken means a sequential process stopped early. Look for that pattern first, before assuming the damage is random, because it points directly at the process and its timing rather than at the hardware.
It also told us what to expect inside the broken repositories, and it is worth being honest that we did not reconstruct exactly why each partial .git had the particular shape it did — objects and refs present, the small top-level files missing. We noted it and moved on to rebuilding them. The boundary was the actionable fact.
Copy the Irreplaceable First
The reason this incident was a bad afternoon rather than a bad month is an ordering decision made before the bulk copy started.
The restore went in two phases. First, the small, non-reproducible items: credentials, secrets, SSH material, machine-local configuration, anything that existed nowhere else and could not be regenerated. Second, the bulk: the projects directory, the 97 repositories, the large reproducible tree. The snapshot was still intact for the first phase. It vanished during the second.
A git repository is, in the ordinary case, recoverable. It has a remote. Rebuilding 52 repositories from their remotes is tedious, but the cost is bounded by whatever in each one was never pushed. A credential that existed only in that home directory and was in the last third of the alphabet is not recoverable from anywhere. Had the two phases been ordered the other way, or merged into one large rsync of the whole home directory, the purge would have landed somewhere in the irreplaceable set, and we would be writing a very different post.
Copy the irreplaceable, non-reproducible items first. Bulk second. The ordering costs nothing when the restore goes well and is the entire difference when the source disappears partway through — which happens more often than exit codes admit.
Correct the False Report Immediately
There is a last rule, and it is about what to do once you discover that the verification you already reported was wrong.
We had said all 97 repositories were healthy. The correct response, on finding 52 were not, was to say so at once, with the true scope — the exact count, the exact boundary, the list — rather than quietly fixing it and hoping the earlier statement would not be relied on. A false "all verified" is worse than no verification at all, because it ends the investigation. Whoever heard it stops checking. They start work in a repository that will fail on its first commit, or they report upward that the restore is done. Every one of those actions rests on a check that measured the wrong thing, and the only way to stop them is to retract the result the moment it is known to be wrong, with the true scope attached.
Honest Limits
This describes a restore from a macOS local snapshot onto the same volume, using rsync, verified with a shell loop. The failure mode — source vanishing, tool exiting clean, presence check passing — generalizes well beyond that; the specific mechanics of APFS snapshot reclamation do not. On another platform the vanishing source might be a network mount or a sync client's cache, and the warning signs differ.
We did not establish which code path in rsync returned 0 rather than a partial-transfer status, and we are not claiming it always will. We are claiming that it did, that it cannot be ruled out in advance, and that the exit code is therefore not the fact to rely on.
The exercise-the-artifact list above is the one we needed. It does not cover artifacts that are only verifiable by their consumer under real load — a database that opens but has a truncated last table — and those need a deeper check than a header parse.
A local snapshot is not a backup. It lives on the volume it protects and it is reclaimed by the same pressure that damages that volume. This incident is an argument for an off-volume backup with its own verification, which is a different post, and one this incident has moved up the queue.
Where This Fits
The shape here is the one we keep writing about because we keep hitting it: a passing check that measured the wrong thing. A secret scanner reporting zero findings proves no known pattern matched, not that history is clean. A verification loop reporting dirty=0 proves a pipeline produced no output, not that a repository is intact. In both cases the control ran, the control was green, and the green carried no information about the property anyone actually cared about.
Restores, backup verification, and the ordering and gating around them are managed infrastructure work. The workstation side — getting a damaged machine back to a trustworthy state without losing the parts that cannot be regenerated — is IT support.
If your restore procedure ends at an exit code, book a call and we will build the verification that exercises what came back.