Across the technical assessments we run at small and mid-sized companies, one finding comes up more than any other: the backups run, the reports are green, and nobody has ever restored anything.
This is not negligence. It is an entirely understandable conflation of two claims that sound alike: “the backup succeeded” and “the data is recoverable”. The first is verified automatically every night. The second is only verified by trying.
Why a green report proves nothing
A backup tool verifies that it wrote a file without hitting an error. That is all it verifies, and reasonably so: it cannot know whether the contents make sense.
Six ways a restore fails with a perfectly green report.
1. The backup is crash-consistent, not application-consistent. Copying a virtual machine while a database is writing produces a valid file and a database in a half-written state. It will refuse to start, or worse, start with partially written data. This is the most common failure and the most insidious, because nothing signals it.
2. The scope has drifted. The backup covers the machines declared three years ago. The one added in February is not in it. Nobody lied; nobody checked.
3. What matters is outside the scope. Machines are backed up, but not network configuration, certificates, firewall rules, or the contents of container volumes. You can restore the servers without being able to reconstitute the system.
4. The media is unreadable. Tapes have not been read in two years, or remote storage has silent corruption on older blocks. The file exists and will not open.
5. There is nowhere to restore to. The backup is perfect and the only machine capable of hosting it is the one that just died. On a hardware failure, the lead time on a replacement becomes your recovery time.
6. The backup is encrypted and the key lived in the system you lost. This scenario happens. It ends badly.
Ransomware changes the nature of the problem
A hardware failure is blind. Ransomware actively hunts backups, because backups are what stop you paying.
The pattern is well documented: access is obtained, it stays quiet for weeks, it identifies the backup infrastructure, it encrypts or deletes the copies, and only then does it encrypt production. The backup report stays green throughout the first part.
What protects you is not geographic distance, it is separation of permissions. If the account that writes backups can also delete them, and that account is reachable from the compromised environment, the remote copy is worthless. You need at least one copy the compromised environment cannot technically reach: offline media, or storage configured to forbid deletion before a retention date expires.
That is a configuration choice, not a purchase. It is probably the highest-return paragraph on this page.
The test, and it fits in an hour
You do not need to simulate a full disaster. A regular partial test catches most failures.
- Pick a machine that matters. Not the most critical one on your first attempt, but not an empty one either. Something genuinely in use.
- Restore it onto an isolated network. Not over the original. In a separate network with no route to production, so it disturbs nothing.
- Boot it and log in. This is the step the backup report cannot do on your behalf.
- Check a dated record. Open something created the day before the backup ran. That is what proves the capture was consistent, not merely present.
- Time it. The measured duration is an operational fact. It determines what you tell customers on the day this happens for real.
- Write down what you did. Two pages is enough. Without them, the person doing this under pressure, possibly not you, rediscovers everything from scratch.
Quarterly for anything critical. That test is the whole of disaster recovery that most companies will ever need, and it is the part they skip. And always after an infrastructure change, because that is where scopes get lost. A hypervisor migration in particular invalidates the backup chain: the tool knew the old platform, not the new one. A successful migration whose backups no longer work is an incident that has not happened yet, which is why we treat requalifying them as a milestone of the migration itself.
Two questions to ask your provider
If someone else holds your infrastructure, two questions are enough, and neither requires technical knowledge.
“When did you last restore a machine for real, and which one?”
A date and a name, or nothing. “We test regularly” is not an answer.
“How long would a full restore take, and where would we restore to?”
If there is no number, the recovery time has never been measured. That is not necessarily alarming, but you should know it before the incident rather than during.
Those are the two questions on which we most often get silence. Not through bad faith: testing a restore earns nobody anything, is invisible, and appears on no dashboard. It is exactly the kind of task that never happens unless someone is explicitly accountable for it.
Further reading
- Leaving VMware: the real alternatives : why requalifying backups is a milestone, not a formality
- Legacy modernization : the engagement format for a bounded migration
- Technical due diligence : the assessment that starts with this question
- Outsourced IT department : when nobody internally owns this
- Modernize or renegotiate: price the exit first : why this is the foundation to fix first
- Build or buy: the calculation almost nobody runs : the decision that comes before the platform