Monitoring Clinical Systems: The Alerts That Matter and the Ones That Wake You for Nothing

Most hospitals monitor their clinical systems the way they monitor file servers: CPU, memory, disk, ping. That tells you the server is running. It does not tell you that studies stopped arriving from CT two hours ago because an AE title changed.

The gap between "infrastructure is healthy" and "clinical workflow is working" is where outages live. Almost every significant clinical IT incident is discovered by a clinician phoning the service desk, not by monitoring — and that is a design failure, not an inevitability.

---

Monitor the Workflow, Not Just the Host

The useful reframing: for each clinical system, ask what a user is trying to accomplish, then monitor whether that specific thing is still happening.

The right-hand column is where the alerts belong.

---

Metrics That Predict Clinical Impact

Interface Engine

The single highest-value monitoring target in most hospitals, because it sits between everything.

• Queue depth per feed, with rate of change. A queue growing steadily is a downstream failure in progress. This is the earliest available warning for a large class of incidents.
• Message age at head of queue. More actionable than depth alone — a queue of 500 messages that clears in seconds is fine; one message stuck for twenty minutes is not.
• Negative acknowledgements (AE/AR) by feed. A rise in rejections signals a mapping or content problem, often after a change at either end.
• Time since last message, per feed. This is the alert that catches silent death. A feed that normally carries hundreds of messages an hour and has carried none for thirty minutes is broken, even though every process is running and every check is green.

That last metric deserves emphasis. Silent feeds are the most common undetected clinical IT failure. The system is up, no errors are logged, and nothing is flowing.

PACS and VNA

• Study arrival rate per modality AE, compared against that modality's normal pattern for the time of day. Alerting on absence is what catches a scanner that stopped sending.
• Failed or partial associations, by source. A rise indicates a network or configuration change.
• Study open time as measured by synthetic transaction, split by cache-hit and cache-miss.
• Prefetch success rate. Silent prefetch failure degrades experience without any error surfacing.
• Storage latency by tier, and free capacity with a projected days-to-full figure. Days-to-full is far more actionable than percentage used.

Databases

• Query latency at the 95th percentile, not the mean
• Cache hit ratio trend through the day
• Lock waits and blocking chains
• Replication lag, where replicas serve reads or DR
• Backup job success and verified restore-readiness

Modalities

Frequently outside IT monitoring entirely, and they are the origin of the imaging workflow. At minimum: reachability, DICOM echo (C-ECHO) success, and time since last study sent.

---

Synthetic Transactions

The strongest signal available is a robot doing what a user does, on a schedule.

Worth building:

• DICOM C-ECHO to every configured node, both directions. Cheap, and catches configuration and network faults immediately.
• DICOM C-FIND / C-MOVE for a known test study, measuring retrieval time. This exercises the actual read path including storage tiering.
• HL7 round-trip: send a test message through the interface engine to a test endpoint and confirm the ACK. Proves the whole chain.
• Application login and worklist load, scripted, with timing. Catches authentication, session, and database problems that infrastructure metrics miss entirely.

Two disciplines make synthetics safe: use dedicated test patient identifiers that are clearly non-clinical and excluded from reporting, and make sure the synthetic path exercises production components without writing clinical data.

---

Thresholds That Do Not Cause Fatigue

An alert that fires often and means nothing trains people to ignore the channel it arrives on. Alert fatigue is not a personal failing; it is a design outcome.

Principles that hold up:

Alert on symptoms, not causes. Page for "study arrivals from CT1 stopped", not for "CPU above 80%". High CPU with no clinical impact is not an emergency; zero arrivals with normal CPU is.

Use rate of change, not just absolute values. A queue at 200 messages may be normal at 08:00 and alarming at 03:00. Trend and time-of-day baselines beat static thresholds for anything with a daily rhythm.

Separate paging from ticketing. Not everything that deserves attention deserves a phone call at 3 a.m. Define two tiers explicitly:

Require duration before firing. Most transient blips resolve themselves. Requiring a condition to persist for several minutes eliminates a large share of noise without meaningfully delaying real detection.

Review fired alerts monthly. For each one: was it actionable? If an alert has fired twenty times and never resulted in an action, it should be deleted or re-tuned. Alert catalogues need pruning as much as they need additions.

---

The First Four Alerts to Build

If you have nothing today, these deliver the most coverage for the least effort:

1. Time since last message, per interface feed. Catches the silent failures nothing else catches.
2. Study arrival gap, per modality AE. The imaging equivalent, and the one clinicians would otherwise report to you.
3. Interface queue age at head. Detects downstream failure before it becomes a backlog.
4. Storage days-to-full, per tier. Turns a future emergency into a scheduled purchase.

Every one of these is workflow-level, and none require a large observability platform to implement.

---

Related Reading

• HL7 Interface Engine Configuration: 10 Best Practices — the system worth monitoring most closely
• Healthcare IT Server & VM Sizing — the metrics that inform sizing
• Backup and Disaster Recovery for Clinical Systems — monitoring backup verification
• HL7 Message Viewer — free tool for inspecting the messages your feeds carry

Need monitoring built around your own feeds and modality list? Send us the details — we build custom monitoring and reporting connectors for clinical systems.