Every hospital has a backup policy. Far fewer have a tested recovery capability, and the gap between the two only becomes visible on the worst day of the year.
The distinction matters because backup and recovery fail in different ways. Backups fail quietly — a job that has been reporting success while silently skipping a volume for eight months. Recovery fails loudly, in front of clinical leadership, at 3 a.m.
Start With RPO and RTO — Per System
RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time. RTO (Recovery Time Objective) — how long you can afford to be down.
A single hospital-wide pair of numbers is always wrong, because clinical systems have genuinely different tolerances.
| System | Typical RPO driver | Typical RTO driver |
|---|---|---|
| PACS archive | Images are re-acquirable only at patient cost — RPO must be near zero | Radiology stops without it; hours matter |
| PACS database | Loss means orphaned images; RPO near zero | Same as archive — they fail together |
| RIS | Scheduling and reporting data; short RPO | Downtime forms can bridge hours, not days |
| LIS | Result data with legal weight; near-zero RPO | Very short — lab turnaround is clinically time-critical |
| Interface engine | Queued messages in flight | Short: a stalled engine backs up every connected system |
| Reporting / analytics | Hours to a day is often acceptable | Days may be acceptable |
Set these numbers with clinical leadership, in writing, and make the cost visible. An RPO of zero and an RTO of minutes is achievable and expensive. The conversation is more productive when framed as a trade-off with a price rather than a technical preference.
The most common failure here is an aspirational RPO that the actual backup schedule does not deliver. If the policy says fifteen minutes and the database backs up nightly, the policy is fiction. Reconcile them.
The 3-2-1-1-0 Rule
The classic 3-2-1 rule has grown two digits, and both additions came from ransomware.
- 3 copies of the data
- 2 different media types
- 1 copy off-site
- 1 copy immutable or offline — this is the addition that matters most. An attacker with domain credentials can delete every backup they can reach. A copy they cannot modify is the difference between recovery and negotiation.
- 0 errors on verification — a backup that has never been restore-tested is an assumption, not a copy.
For clinical systems, the immutable copy is not optional. Healthcare is a repeatedly targeted sector, and modern intrusions specifically enumerate and destroy backup infrastructure before triggering encryption.
Different Data, Different Strategy
Clinical Databases
Databases need transaction-consistent backups, not file copies of a running data directory. Full plus incremental plus transaction-log backups, with log shipping frequency set to meet the RPO.
The critical, frequently-missed step: verify that the database backup and the image archive backup are consistent with one another. A PACS database restored to Tuesday 02:00 paired with an archive restored to Tuesday 06:00 produces database rows pointing at objects that exist, plus objects with no database rows — orphans that surface as missing studies weeks later.
Image Archives
Full backups of a multi-hundred-terabyte archive are impractical on any useful schedule. The realistic architecture is replication to a second site plus immutability, with the archive treated as append-mostly.
Two things to verify rather than assume:
- That replication is actually current. Replication lag is a silent RPO killer. Alert on it.
- That deletion propagates the way you expect. If a mistaken purge on the primary replicates instantly to the secondary, you have two copies of the mistake. Immutability windows and delayed-delete policies exist for exactly this.
Interface Engine Queues
Often forgotten and disproportionately painful. Messages sitting in a queue represent clinical events — admissions, orders, results — that have already happened in the real world. Lose them and reconciliation is manual.
Back up the engine's journal and queue storage, and know your replay procedure: which feeds can safely be replayed, which produce duplicates downstream, and how each receiving system handles a repeated message. That knowledge belongs in the runbook before you need it.
Configuration
The thing nobody backs up until they have needed it once: interface mappings, routing rules, hanging protocols, AE title configuration, user roles, dictionary customisation. Rebuilding these from memory after an outage can take longer than restoring the data. Export and version them.
Downtime Procedures: The Non-Technical Half
DR planning that stops at the technical layer fails clinically. While systems are down, care continues.
What needs to exist before the incident:
- Downtime forms, printed and physically located where staff will need them
- A downtime PACS or local cache on key workstations so recent imaging remains viewable
- Communication tree that works without the systems that are down — including without email
- A defined recovery-mode data entry process for backfilling what happened on paper
- A declaration authority — who decides this is an incident, and who decides it is over
The backfill process is the one most often missed. Systems come back and everyone is relieved; three days of paper records still have to be entered, reconciled, and validated, usually by staff who are already behind.
Testing That Produces Evidence
A recovery test that consists of confirming the backup job reported success is not a test.
Tiered testing that works:
| Frequency | Test | What it proves |
|---|---|---|
| Monthly | Restore a small number of individual studies and records | The data is readable and the process is documented |
| Quarterly | Restore a full system to an isolated environment | The backup chain is complete and the runbook is accurate |
| Annually | Full failover exercise with clinical participation | RTO is real and people know their roles |
Record for every test: what was restored, how long each phase took, what failed, and what changed as a result. The timing data is what converts an aspirational RTO into a measured one — and the first full test almost always reveals that the real RTO is several times the documented figure.
Test restores into an isolated environment. A recovery test that accidentally writes to production, or that brings a restored system up on the live network with the same AE titles and IP addresses, creates the incident it was meant to prevent.
Recovery Runbook Checklist
- RPO and RTO defined per system and signed off by clinical leadership
- Backup schedule verified to actually deliver the stated RPO
- At least one immutable or offline copy, outside the production credential domain
- Database and image archive backups verified consistent with each other
- Interface queues and replay procedure documented per feed
- System configuration exported and version-controlled
- Restore order documented across dependent systems
- Downtime forms and procedures distributed and current
- Contact list reachable without hospital email or telephony
- Restore tests scheduled, performed, and timed, with results recorded
Related Reading
- Healthcare Storage Tiering — why the archive tier is not a backup
- Healthcare IT Server & VM Sizing — the infrastructure underneath
- Monitoring Clinical Systems — catching failures before they become incidents
Need backup verification or archive-consistency reporting automated for your environment? Send us the details — we build reconciliation tooling against real PACS and database exports.
Related Articles
Monitoring Clinical Systems: The Alerts That Matter and the Ones That Wake You for Nothing
6 min read
Moving Clinical Systems to the Cloud: Architecture Patterns That Actually Work
6 min read
Hospital Network Design for Imaging Traffic: VLANs, Bandwidth, and QoS
7 min read