The Restore That Failed

On 2026-07-14 I ran my quarterly restore drill and one of the three destinations failed it. I have written some version of "a backup you have not restored from is a rumor" in at least four entries on this site. It would be convenient to leave this one out. It is also the most useful thing I have measured all quarter, so here it is with the numbers intact.

The drill

Quarterly, since 2023. I pick a destination at random, restore a defined set to a scratch disk, and verify against checksums recorded at backup time. The set is the password vault, the notes archive, the scanned paper folder, and the home configuration repository. Roughly 34 GB.

Three destinations, on the principle that two is one and one is none:

1. The NAS in the utility room.
2. An encrypted object storage bucket with a provider I do not otherwise depend on.
3. An offline drive that lives in a drawer, plugged in weekly, so that a compromise of the running systems cannot reach it.

Destinations 1 and 2 restored cleanly. Median restore time 18 minutes, all checksums matched, no findings.

Destination 3 restored cleanly too. That is what made it interesting.

The finding

The offline drive restored perfectly, quickly, with every checksum matching. It was also 41 days stale.

The most recent snapshot on it was dated 2026-06-03. I had been plugging the drive in less often since moving the desk in early June, and the weekly job that fed it had been running, exiting successfully, and reporting green the entire time.

Because when the drive was absent, the job skipped the destination and exited 0.

Forty-one days of green checkmarks describing work that did not happen. My monitoring was watching the process. Nothing was watching the artifact.

Destination	Newest snapshot at drill	Drift	Restore verified
NAS	2026-07-14	0 days	Yes
Object storage	2026-07-14	0 days	Yes
Offline drive	2026-06-03	41 days	Yes, of stale data

Why this is the worst kind of failure

Consider what a real incident on 2026-07-13 would have looked like. Ransomware or a bad rm, propagating to the NAS because it is online, and to object storage because the credentials to write there live on a machine that was compromised. Both are defensible with versioning and I have it. But the drawer drive is the destination that exists precisely for the case where I am not fine, and it would have handed me a June file set while my monitoring insisted everything was current.

The loss would not have been catastrophic: 41 days of documents, partly recoverable from the sync mesh. The dangerous part is not the size of the gap. It is that I would have made recovery decisions from a dashboard that was confidently wrong, and discovered the truth at the moment I had the least capacity to deal with it.

A monitoring system that reports success when the work did not happen is worse than no monitoring, because no monitoring at least leaves you appropriately nervous.

The fix

Two changes, both small, and I am mildly annoyed that neither occurred to me in three years of drills.

Alert on artifact age, not job exit code. A daily check now reads the newest snapshot timestamp from each destination and pages me if any exceeds 8 days. It does not care whether a job ran, succeeded, or exists. It asks the only question that matters: how old is the newest thing I could actually restore?

Make a skipped destination a failure. The job now exits non-zero when a configured destination is unreachable. Absence is a result, and treating it as a no-op was the original sin.

Both implemented 2026-07-14. The age check fired correctly in testing when I unplugged the drive deliberately, and drift is now 0 across all three.

The general lesson, stated plainly

I have been auditing dependencies on this site for a month, and every one of those audits measured a process. This drill found the gap that process monitoring is structurally incapable of finding, and I think the principle generalizes past backups:

Measure the artifact, not the procedure. Not "did the sync job run" but "how stale is the remote copy." Not "is the generator scheduled" but "when was the last entry actually published." Not "do I have a second bank" but "did I move money through it this quarter." The procedure is a claim about the artifact. Verify the claim.

That last example is not hypothetical, and it is what I am auditing next.

Confidence 0.94 that a meaningful number of people reading this have a green dashboard describing a job that has quietly been skipping its work. Go and look at the timestamp on the newest file you could restore right now. Not the log. The file.

Next drill: 2026-10-13. I will publish the result whether or not it is flattering, since that is now established practice.