The backup server's datastore reached 100% on its local ZFS mirror and stayed there. Prune marked snapshots for removal as scheduled; garbage collection, which is the step that actually frees chunks, aborted at its own precondition every night. A rescue that borrowed from ZFS's slop reserve bought a day of headroom and the next backup run refilled it. A full copy-on-write datastore cannot free space, because freeing space requires writing metadata, and there is nowhere to write it.
The way out was a second datastore on a 4 TB iSCSI LUN served by our NAS over a single 10 GbE link. That part took an afternoon. The part that took longer was a formatting command that would not finish, an iSCSI login that would not authorize, and a backup notification that was configured on a different server from the one we were looking at. This is the whole move, including the three traps and the one thing we are still paying for.
Key Takeaways
- A PBS datastore at 0 bytes free cannot heal itself: garbage collection fails its
atimesafety check before it reclaims anything, so prune marks and nothing is freed — retire the store rather than trying to prune it back zpool listshowing a few GB free on a full pool is the slop reserve, not usable space; trustzfs listreportingAVAIL 0B- Use XFS, not ext4, for a PBS chunk store: it is millions of small files spread across 65,536 directories, and ext4's inode count is fixed at format time
mkfs.xfson a LUN that reports no space reclamation must use-K— the default discard pass ran past eight minutes with no end; with-Kit finished in seconds.mkfs.ext4 -E nodiscardis the same rule- Synology puts CHAP and mutual CHAP on the iSCSI target, even when the LUN's own permission reads "Allow all"; login fails with an authorization error until both are set on the initiator's node record
- Prove a new datastore with a real restore to a scratch container that you then destroy, not with a backup that merely reports success
- vzdump notifications are configured in Proxmox VE's cluster notification config, not in PBS — and a webhook matcher on
type=vzdumpmust also match severityerror,warning, or it pages on every successful nightly backup
A Full Copy-on-Write Datastore Cannot Prune Itself Back
The first instinct with a full backup store is to shorten retention and let garbage collection catch up. That does not work on a full ZFS pool, and the reason is mechanical rather than a tuning problem.
PBS garbage collection begins with a safety check: it bumps the access time on a chunk and verifies the bump landed, to confirm the filesystem will honour the atime updates that its mark phase depends on. On ZFS at zero bytes free, that single metadata write cannot be allocated. GC aborts with atime safety check failed: update atime failed for chunk/file, and since GC is the only step that reclaims chunks, prune's work is purely nominal.
We tried the documented rescue, raising spa_slop_shift from 5 to 6 to release some of the pool's slop reserve. It freed about 90 GB. The next nightly backup refilled it and died mid-run. The pool's own tools also mislead here: zpool list reported roughly 14 GB free on a 464 GB pool, which is the reserve, not space you can use; zfs list reported AVAIL 0B, which is the truth.
Retire a full store. Build the replacement, repoint the backup job, leave the old pool mounted read-only for restores, and let it age out. The rest of this post is the replacement.
XFS, Not ext4, for a Chunk Store
A PBS datastore is content-addressed. Every chunk of every backup is a separate file, named by its hash, in one of 65,536 .chunks subdirectories. On our store that is millions of small files, and it grows with every unique block that enters the estate.
ext4 allocates its inode table at format time. The default ratio is generous for ordinary filesystems and is a genuine exhaustion risk for this one: run out of inodes and the store is "full" with free blocks to spare, which is the same deadlock as above wearing a different hat. XFS allocates inodes dynamically and has no equivalent ceiling. For this workload the choice is not close.
The mount is by filesystem UUID with network-aware options, so that a reorder of block devices after a reboot cannot point the datastore at the wrong disk:
UUID=<xfs-uuid> /mnt/pbs-datastore xfs _netdev,nofail,x-systemd.device-timeout=180,x-systemd.requires=open-iscsi.service 0 0
The _netdev and nofail are standard for an iSCSI-backed mount on an unattended box. The explicit dependency on the iSCSI service unit and the long device timeout are there because a short timeout races the iSCSI login after a reboot where the NAS comes up slower than the server — and with nofail, the server boots without its datastore and says nothing. We gave that failure mode its own Prometheus rule: fire when the mountpoint's filesystem metric is absent for ten minutes.
Format It: Prove It Empty, Then Skip the Discard
We reused an existing 4 TB thick-provisioned LUN rather than creating a new one. Reuse carries one obligation: prove the LUN holds nothing before writing a filesystem over it. A used filesystem always leaves a superblock at offset zero, but a partition table or a previous volume manager can leave signatures elsewhere, so the check is three-fold — blkid reports nothing, there is no partition table, and zero-probes at several offsets come back empty. Two details keep the probe honest. od prints zero bytes as space-separated zero words, so the pattern has to allow whitespace throughout, not only before the first digit. And a probe only counts if the read itself succeeded and returned the full sample: a LUN you cannot read — wrong permissions, a dropped iSCSI session, a path that is not the device you think it is — produces exactly the same silence as an empty one, so the loop checks dd's exit status and the sample length and stops the procedure on either failure instead of reading silence as clearance:
#!/bin/bash
lun=/dev/<lun>; probe=$(mktemp)
blkid "$lun" # expect no output
fdisk -l "$lun" | grep -i 'disklabel' # expect no label line
for off in 0 1 64 1024 4096; do
if ! dd if="$lun" of="$probe" bs=1M skip="$off" count=1 iflag=fullblock status=none \
|| [ "$(stat -c %s "$probe")" -ne 1048576 ]; then
echo "read failed or short at ${off} MiB — not inspected, do not format"; exit 1
fi
od -An -v "$probe" | grep -qvE '^[ 0]*$' && echo "non-zero data at ${off} MiB"
done
rm -f "$probe"
Then the format, and the trap. Modern mkfs.xfs issues a discard across the whole device before writing the filesystem, so that a thin LUN or an SSD can release everything the previous occupant held. On this LUN the discard ran for more than eight minutes with no indication of ever ending. The NAS reports Space reclamation: Not Supported for it, and the kernel's view of the target agrees — emulate_tpu=0 in the target configuration — so every discard request was being accepted and doing nothing, four terabytes' worth.
The fix is to tell mkfs not to bother:
mkfs.xfs -K -L pbs-store /dev/<lun> # -K: skip the discard pass
It finished in seconds. The same rule applies to ext4 as mkfs.ext4 -E nodiscard. The general form: check whether the device can reclaim space before you ask it to, and if it cannot, do not wait on a no-op.
CHAP Lives on the Target, Not the LUN
The iSCSI login failed with error 24 - authorization failure. The Synology UI showed the LUN's permission as Allow all, which reads as "no authentication required" and is the wrong place to be looking.
On DSM, CHAP is a property of the iSCSI target, and a LUN's permission governs which initiators may map it once they have reached it. Our target had both CHAP and mutual CHAP enabled — visible on the NAS side as authentication=1 in the target's configfs entry — and the LUN permission said nothing about it. The login succeeded only when both the outbound credential and the inbound one were set on the initiator's node record for that target:
iscsiadm -m node -T <target> -o update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T <target> -o update -n node.session.auth.username -v <chap-user>
iscsiadm -m node -T <target> -o update -n node.session.auth.password -v <chap-secret>
iscsiadm -m node -T <target> -o update -n node.session.auth.username_in -v <mutual-user>
iscsiadm -m node -T <target> -o update -n node.session.auth.password_in -v <mutual-secret>
iscsiadm -m node -T <target> --login
Mutual CHAP is worth keeping — it is what lets the initiator verify it has reached the right target — but it means two credential pairs and an error message that names neither.
We run this over a single dedicated 10 GbE path, one NIC on the backup server to one port on the NAS, and deliberately do not configure multipath. One path is one thing to monitor and one thing to debug, and the datastore's ceiling is the NAS's disks, not the link.
Prove It With a Restore, Not a Backup
A new datastore is not proven when the first backup reports success. It is proven when something comes back out of it and runs.
The backup half: a real vzdump of a production container in snapshot mode, 3.8 GiB written to the new store in 163 seconds. The restore half, which is the one that matters: that backup restored to a scratch container with a fresh ID, 5.7 GiB in 78 seconds at roughly 75 MB/s. The scratch container's root filesystem mounted and inspected — the expected Ubuntu release, the expected application directory, 156,309 files — and then the scratch container destroyed, with zero leftover volumes on the hypervisor and the original container untouched throughout.
Those numbers set expectations as much as they prove function. This is slower than the local ZFS mirror it replaced, and it is supposed to be. A PBS chunk store is small-random-IO heavy, and the NAS volume behind the LUN is an SSD-cached HDD array. The move bought capacity and a second physical location for the chunks. It did not buy speed, and a plan that assumes otherwise will be disappointed on the first full verify.
One retention trap surfaced the week after the repoint. Retention lives in two places — the PBS prune job and the prune-backups setting on the PVE backup job, which PVE applies after every run — and the PVE one wins nightly. Our new store had no prune job yet and the PVE job carried a short keep-last, so for several days the estate had three days of retention. Change both layers, and verify by counting snapshots per group rather than reading the policy.
Thin Over Thick, and the Copy That Still Pins 4 TB
The LUN we reused was thick-provisioned: 4 TB committed on the NAS volume from the moment it was created, used or not. On September 19 we moved the datastore again, to a thin-provisioned LUN of the same nominal size, carrying the backup history across so the oldest snapshot is still the one from July.
The thick LUN is still mapped to the target. It is unmounted, in no fstab, and in no PBS configuration, and it still holds its full 4 TB on a NAS volume that is already under pressure. It is a stale copy of the datastore as it stood before the move. It will be deleted, but not until two things are true: a read-only mount confirms that every backup group on it also exists in the live store, and the thin LUN has been through enough nightly runs and a full verify to be trusted on its own. Until then the old copy is the only thing that makes the migration reversible, and 4 TB of pinned space is the price of that.
The rule we apply before deleting anything that looks regenerable: ask what still runs from it, and what would be restored from it, and get a positive answer to both before the delete.
The Backup Alert Lived in PVE, Not PBS
The last trap had nothing to do with storage. After the move, failed backups were not reaching our ops chat channel. The natural place to look is PBS's notification configuration, and PBS's notification configuration held only the built-in mail-to-root target, which was working. Nothing there was wrong, because nothing there was involved.
vzdump runs on the Proxmox VE node, so the PVE cluster's notification stack owns its events. The failing target was in PVE's cluster notification config: a webhook POSTing to a push topic that had become ACL-locked and was returning 403 to everything. When a vzdump notification fails, look in PVE, not PBS.
The replacement was a PVE webhook to a chat bot, and the matcher is where the second half of the trap lives. The original matcher was exact:type=vzdump and nothing else — it matched every vzdump event, success included, and had only ever been quiet because the target it pointed at was broken. Repointing it unchanged would have paged on every successful nightly backup across the whole estate. The fixed matcher adds match-severity error,warning, so failures page and successes still go to mail through the default matcher. We verified it both ways: a deliberately failing vzdump reached chat and mail, a successful one reached mail only.
Honest Limits
This is one backup server, standalone rather than a cluster member, with one NAS over one link. We have not tested multipath, and the single-path choice would be wrong for anyone whose restore window cannot tolerate a NIC failure. The restore throughput is a function of our NAS's disks and cache, and a different array gives a different number.
The discard trap is specific to LUNs that report no space-reclamation support. On a thin LUN that does support it, the default discard pass is correct and -K would leave stale allocations in place on the NAS side; check what the target reports before choosing.
Reusing a LUN is a shortcut we took with a full set of emptiness checks and would not recommend without them. A fresh LUN removes the question entirely.
And the thick copy is an admitted loose end. A reversible migration costs space until it is verified, but it is pinning capacity on a volume that is already tight, and "we will verify and delete it" is a commitment with a date on it, not a state we intend to live in.
Where This Fits
Backups are the one piece of infrastructure whose failure is invisible until the day it is the only thing that matters. Everything in this post was found by testing — a restore to a scratch container, a deliberately failing vzdump, a zero-probe on a LUN — rather than by waiting for the nightly job to report green, because the nightly job had been reporting green through a full datastore, a broken notification target, and three days of retention.
If you are deciding whether to own this layer at all, we wrote about the self-hosted versus cloud decision separately. Running backup servers, the storage under them, and the alert rules that make their failures loud is managed infrastructure work; the evidence side — proving a restore rather than trusting a backup — is the same discipline that audit logging for compliance frameworks demands of every other control.
If your backup job reports success every night and you have not restored from it to a scratch system this quarter, book a call and we will do the restore with you.