We cleared 1.55 TB from the NAS in one sitting: every share's recycle bin, a set of old hypervisor backup archives, and a 469 GB disk image nobody had used in a year. Then we ran df. The used figure was identical to the byte, before and after. Not close — identical.
The deletions had worked. The files were gone from every share. What had not changed was the set of blocks the volume considered in use, because every one of those blocks was still referenced by a snapshot, and the newest snapshots on the volume cannot be deleted by anyone, root included, until a window set in one dialog box of the DSM UI has elapsed. This is what pinned the space, how we found the window, the quota command that nearly broke a backup target in the same session, and the monitoring rule that had been quietly reporting the whole thing as a second, unrelated problem.
Key Takeaways
- On a copy-on-write volume with snapshots, deleting a file frees nothing until no snapshot references its blocks —
dfmeasures blocks in use, not files you can see - Synology Snapshot Replication's immutable snapshots are enforced by the kernel: delete returns
err=[0xFC00],lsattr -dshows theiflag, and evenbtrfs property set ro falseas root is refused - Immutability windows vary per share and are not where you expect — we observed about 2 days on one share, about 7 on others, and none on one — and they are set only in the DSM UI, not in any flat file or package database
- Before promising freed space on a snapshotted NAS, list the snapshot directory for the share and plan against the date the last referencing snapshot can be deleted — immutability expiring makes a snapshot deletable, not deleted — or shorten the window first and wait for it
synosharequota settakes megabytes and silently ignores aGsuffix:1500Gset a 1.5 GB quota on a 1.9 TB Time Machine share — always read the value back after writing it- A node_exporter that reports the data volume under mountpoint
/turns one full volume into two alerts; a "root disk low" page on a NAS is a duplicate until proven otherwise - A control working exactly as configured can still produce a misleading signal — immutability that refuses root and an exporter that mislabels a mount are both doing their jobs
Where 1.55 TB of Deleted Data Went
The NAS volume runs Btrfs with Snapshot Replication taking a daily snapshot of nearly every share. A snapshot is a point-in-time reference to the blocks that made up the share at that moment. Deleting a file afterwards removes the directory entry from the live share; the blocks stay allocated for as long as any snapshot still points at them.
That much is standard copy-on-write behaviour and most people who run snapshots know it in the abstract. What makes it bite is the arithmetic. If yesterday's snapshot references every block of a 469 GB disk image, deleting the image today frees zero bytes until yesterday's snapshot is gone. If the share takes a snapshot every day and keeps a week of them, the freed space arrives in seven daily instalments, each one the size of whatever that day's snapshot was the last to reference. Our cleanup was 1.55 TB of blocks that were all still held by the newest snapshot on each share. The number df reported was correct. The expectation was wrong.
Two theories are worth ruling out before this one, because they are cheaper and sometimes right: that the deletions landed in a recycle bin that has not been emptied, and that the volume is reclaiming lazily and needs a few minutes. The recycle bins were the first thing we emptied, and Btrfs does not reclaim snapshot-referenced blocks on a delay — it does not reclaim them at all while the reference exists.
Root Cannot Delete an Immutable Snapshot
Older snapshots delete cleanly from the shell:
sudo /usr/syno/sbin/synosharesnapshot delete <share> GMT-<older-timestamp>
Then the newest ones refused:
synosharesnapshot delete <share> GMT-<newest-timestamp>
err=[0xFC00]
The error code is not documented anywhere we could find. What it means is visible with lsattr:
lsattr -d /volume1/@sharesnap/<share>/GMT-<newest-timestamp>
----i---------------- /volume1/@sharesnap/<share>/GMT-<newest-timestamp>
That i is the kernel's immutable attribute. It is not a DSM permission layer and it is not a Btrfs read-only property that a privileged user can flip. We tried the Btrfs route anyway, because it is the obvious next step for anyone who has unlocked a read-only subvolume before:
sudo btrfs property set -ts /volume1/@sharesnap/<share>/GMT-<newest-timestamp> ro false
Operation not permitted
Refused, as root. This is the feature working. Immutable snapshots exist so that a compromised administrator account, or ransomware running with its privileges, cannot destroy the recovery points. The same property that protects the data from an attacker with root protects it from an operator with root who would like 1.55 TB back today.
The Window Is Set Only in the UI
The immutability window is per share, and it is not uniform. On the share that holds hypervisor backup archives we observed about two days. On the Time Machine share and a general workspace share, about seven. On one share, none at all. Nothing about the share names or their contents predicts which is which; the windows were set at different times by different hands and never reconciled.
We looked for where that setting lives, because a value you can read is a value you can plan around. It is not in any flat configuration file. It is not in the Snapshot Replication package's SQLite databases. As far as we can tell it exists only in the DSM UI, under Snapshot Replication, Snapshots, Settings, Immutable snapshots, per share. The synosharesnapshot list command shows snapshot names only, and its attribute filters returned results we could not reconcile with lsattr, so we stopped trusting them.
Two practical notes for anyone working at the shell on DSM. The management binaries live under /usr/syno/sbin, which is not on sudo's path, so call them by full path. And lsattr -d on the snapshot directory is the one reliable way to tell an immutable snapshot from a merely read-only one.
Plan Freed Space Against the Expiry Date
The operational rule that came out of this is short: before promising freed space on a snapshotted volume, list the share's snapshot directory, find every snapshot that still references the deleted blocks, and the date the last of them can be deleted is the earliest date the space can arrive.
Two things have to be true before a block comes back, and immutability expiring is only the first. When a snapshot's immutability window ends it becomes deletable; it does not get deleted. It stays on the volume, still referencing every block it captured, until the share's retention rule ages it out or an operator removes it by hand. If retention is longer than the immutability window — ours was, on the shares that mattered — the expiry date is a floor, not a promise. And since every snapshot taken while the data existed references the same blocks, one lingering copy is enough to pin all of them. The space returns when the last referencing snapshot is gone, whichever mechanism removes it.
ls -1 /volume1/@sharesnap/<share>/ | tail -3
for s in $(ls -1 /volume1/@sharesnap/<share>/ | tail -7); do
printf '%s ' "$s"; lsattr -d "/volume1/@sharesnap/<share>/$s" | cut -c1-20
done
If the date is not acceptable, the only lever is to shorten or remove the immutability window in the UI first, and then wait for the already-immutable snapshots to age out anyway — the change applies to new snapshots, not to ones already flagged. Once a snapshot is mutable, delete it with synosharesnapshot delete rather than waiting on retention, and check df after each one: the number moves only when the last reference goes. Either way the cleanup is a scheduled event, not an instant one, and the person asking for the space needs to hear that before the deletes are run rather than after.
What we did not do, and would not recommend, is shorten every window to get the space back faster. The seven-day windows on the shares that matter most are the right length. The fix for a full volume is not a weaker ransomware control; it is a capacity plan that accounts for the snapshot tail.
The Quota CLI That Ignores Its Units
In the same session, tightening a share quota so the Time Machine share could not grow into the space we were trying to recover, we ran:
sudo /usr/syno/sbin/synosharequota set time_machine 1500G
The intent was 1.5 TB. The command accepted it without complaint. Reading it back with synosharequota get showed a limit of 1500 megabytes.
synosharequota set takes its size in megabytes and silently discards a unit suffix. We had set a 1.5 GB quota on a 1.9 TB share whose backups were already far past it. Had we not read it back, the next Time Machine run would have failed against the quota, and so would every one after it, on a backup target we had just been working to protect. The correct call is the bare number of megabytes:
sudo /usr/syno/sbin/synosharequota set time_machine 1536000 # 1.5 TiB
The rule generalizes past this one command and this one vendor: a CLI that accepts a value it does not understand will not tell you. Read every setting back after writing it, and compare against what you meant, not against what you typed.
The Root-Disk Alert That Was the Volume Alert
The monitoring side of this story had been running for weeks before the cleanup and is the reason we were looking at the NAS in the first place.
Our Prometheus estate had two alerts firing for the NAS. One was the expected volume alert: the data volume was above its threshold. The other was a root-disk alert for the same host, reporting that / was nearly full. A full root partition on a NAS is a real emergency — it is where DSM lives — so this one got attention first, and the attention found nothing. On the NAS itself, df -h / showed a 7.9 GB system partition at about 21% used.
The node_exporter on the NAS was reporting the data volume's 34.5 TB under mountpoint /, and the identical figure again under /tmp. Whatever filesystem the exporter's process sees as its root, it is the data volume, not the system partition. A comment in our alert rules file had even recorded, half-correctly, that the NAS exporter "only exposes /, not the volume". It exposes the volume labelled as /. So the root-disk alert and the volume alert were the same disk, and every page for the root disk was a duplicate of a page we had already received.
The fix we applied was in the rule rather than on the NAS, because the NAS exporter is a package we would rather not customize:
- alert: DiskSpaceLow
# threshold and duration unchanged; only the instance exclusion is new
expr: |
node_filesystem_avail_bytes{mountpoint="/", instance!="<nas-instance>"}
/ node_filesystem_size_bytes{mountpoint="/", instance!="<nas-instance>"}
< <threshold>
The alternative is correcting the exporter's --path.rootfs on the NAS so that / means what it says. Either is fine. What is not fine is leaving a rule in place that fires a second, scarier page for the same condition, because the second page is the one that gets investigated and the investigation finds a healthy disk and concludes the alert is noise. From there it is a short step to muting it, and then the real root-disk alert is muted too.
Honest Limits
This is one NAS, one volume, one snapshot schedule. The specific windows we observed are ours, and the immutability setting may be surfaced differently in a later DSM release; we checked the flat files and package databases available to us at the time and found nothing, which is not the same as proving the value is stored nowhere readable.
We did not test whether shortening a share's immutability window shortens the remaining time on snapshots already flagged. Our reading of the behaviour is that it does not, and we planned on that basis, but we did not run the experiment on a share we cared about.
The quota trap is specific to one command's parsing. Other DSM commands may handle suffixes correctly, and we have not audited them; the read-back habit covers the gap regardless.
And the exporter fix is an exclusion, which is a blunt instrument. It means the NAS's real system partition is now unmonitored by that rule. We accept that for now because the volume alert already covers the disk that matters, but it is a gap we have written down rather than a gap we have closed.
Where This Fits
Both halves of this incident are the same shape: a control doing exactly what it was configured to do, producing a signal that read as something else. Immutable snapshots refusing root is the ransomware protection working. An exporter reporting the data volume as / is the exporter reporting what its process can see. Neither was broken. Both cost real time, and both were fixed by understanding the mechanism rather than by overriding it. We wrote about the same failure shape at the DNS layer in a blocklist that silently broke vendor APIs, where a correctly configured filter produced a blank page that looked like an account problem.
Running storage that keeps immutable recovery points, planning capacity around the snapshot tail those points create, and keeping the alert rules above them from paging twice for one disk is managed infrastructure work; the expectation-setting side — telling the person who asked for the space when it will actually arrive — is the part of managed IT support that no dashboard does for you.
If your NAS takes snapshots and you have never checked how long a deleted terabyte takes to come back, book a call and we will walk the snapshot tail with you.