Get Free Quote

A Measured Trial: How One Support Team Tested AIOps Claims

Healthcare IT

Consider an invented tale that carries a genuine method. A support-technology team at an imaginary regional care network, three analysts and a manager, decides in January to cut the noise from their monitoring system. They run the project like a small study: a baseline, a hypothesis, a trial, a result, and a ruling on what to keep. The network, the people and every figure that appears in the team’s own results are invented for illustration. Facts from outside the story, by contrast, come from the Ops Atlas, which xOps School publishes, and I flag them as such. Everything concerns systems that support appointments, messaging and records administration, and nothing concerns clinical care.

The baseline: counting before changing

The manager’s first instruction is to measure the status quo. Over a fortnight the analysts log every alert that wakes someone, then label each one after the fact: real problem, duplicate of another, symptom of something upstream, or nothing. In this illustration, about one alert in five proved to be a real, distinct problem. The rest were repeats, symptoms or noise. None of these numbers is real; they are sized to look like what a similar audit usually turns up.

The labelled fortnight becomes the team’s answer key. That decision turns out to be the most valuable of the whole project. The Atlas insists that grouping methods be judged by precision and recall against labelled incidents, and not by the size of the shrinking pile, and it suggests that around a hundred labelled incidents are needed before trying anything learned. The team’s fortnight yields thirty-odd, so they keep labelling as they go.

Hypothesis one: structure before statistics

The team writes its hypothesis on a whiteboard: most of the noise is structural, meaning inconsistent labels, repeats and missing dependency information, and can be removed without any machine learning. It is a prediction that could fail, which is what makes it a hypothesis. It also matches the Atlas ranking, where the first four practices are label contract, deduplication, grouping with suppression, and topology plus change correlation, and where only the fifth introduces anomaly detection.

They begin with labels. Alerts from the booking page, the patient messaging service and the records archive each get the same fields. They also split one overloaded label, “zone”, which meant a building in some alerts and a server group in others. The Atlas describes a related trap, a label meaning two things so that suppression silently hid a genuine outage, and the team treats the split as cheap insurance.

The trial of three correlation approaches

Meanwhile the manager arranges short evaluations of three ways of correlating alerts, so the team can compare them against its answer key rather than against sales slides. The Atlas helps with vocabulary here. It describes one commercial pattern that caps a correlated incident at 300 alerts, another that builds a fault tree over a discovered map of services, and a third, a cloud-provider diagnostic, that offers causes from just four categories: version changes, traffic increases, instance failures and full disks. The team does not copy any of them. It borrows those descriptions only to ask which causes each approach is able to name and which it cannot.

The results, again invented for illustration, show the usual trade-off. The cheapest rule-based approach has high precision and middling recall, because it only groups what the labels and the map explain. A time-window approach shrinks the pile most, but its precision falls, since it merges incidents that merely overlap in time. The map-based approach sits between them. The Atlas gives a hard example of the same trade-off: in its lab, simply grouping within a five-minute window compressed the alert pile 5.5-fold and still blended together incidents that were unrelated. The team’s analyst writes the conclusion on the wall: smaller is not better, correct is better.

Anyone wondering how a care network thinks about the other end of its digital operations, the patient-facing side, may like our overview of how telehealth services for eye care are described to patients; it covers the service side of what this team keeps running.

Hypothesis two: learning helps only at the margins

With structure in place, the second hypothesis is that a learned anomaly detector adds value for the thin slice of faults that no rule covers. The team chooses an isolation-forest detector, a staple of a popular Python toolkit, and runs it in shadow for six weeks, scoring events invisibly. It also tries a log-template parser, which the Atlas notes uses a similarity threshold of 0.4 in its worked example.

The detector finds some useful oddities and many harmless ones. A cluster of false positives lines up with a regional public holiday, when message volumes drop, and the team does what the Atlas did in its lab: it adds the holiday to a calendar and the false alerts vanish. The Atlas reports a parallel result from its own lab: a seasonal baseline spotted four of four planted faults, yet also gave 34 false alerts, 32 of them clustered on an undeclared holiday, and one entry in a calendar wiped out every one of those 32.

The ruling: keep the detector in shadow, promote it to a canary on one service next quarter, and add the research caveats to the team’s notes. The Atlas collects research with a sobering tone: one 2022 study found deep-learning log detectors less dependable than hoped, a 2024 paper showed simple template-rate counting to be competitive, and a 2024 comparison across 21 cause-finding methods named no overall winner. The team resolves to trust its own labelled history over any published ranking.

One more observation from the trial deserves space. The analysts noticed that their best results came on days when the on-call person had written a one-line note about what they saw. Those notes became the seed of a runbook library, and the library in turn improved the labels. Evidence-led work often discovers that the humble by-product, here a diary of human observations, is worth more than the headline experiment.

The automation they allowed, and the one they refused

One repair is approved, the clearing of old log files, capped in frequency, followed by verification, and escalating to a person whenever the verification fails. Another is refused: automatically restarting the messaging service. The manager’s reason is analytical. The restart would mask the signal the team needs to diagnose the underlying fault, and no one can show that it is safe to repeat. The Atlas supports the stance with its description of guarded remediation as approved, reversible and rate-limited, and with its repeated point that self-healing is a direction rather than something to buy.

Security gets a line item as well. The Atlas notes that an alert-management interface can accept requests from any web origin, so a single crafted request might silence an outage, and that the event source in a well-known automation collection speaks unauthenticated HTTP. The team binds its listener to loopback and requires authentication on every webhook.

Reading the results with proper scepticism

A good evidence-led story ends with limits. The team’s sample is small, its fortnight may not be typical, and its labels were made by the same people who built the rules. Those caveats go into the report. The Atlas adds one more: the tooling landscape churns, with several cloud anomaly services retired or closed to new customers within about a year, so the team writes its rules and labels in portable form and treats any engine as replaceable.

A final habit, echoing a simple monthly expense-tracking routine for students, is a recurring scorecard. The team publishes one each month, holds a weekly noise review and a quarterly platform review, and treats a rise in alerts that nobody can explain as a finding, not an inconvenience.

What the story illustrates

The team’s method is more portable than its numbers. Measure first. Label the incidents. Fix structure before adding statistics. Compare approaches against your own answer key. Keep new components in shadow and give automation fences. For the primary text behind the facts quoted here, see the Ops Atlas entry on AIOps, and remember that anything attributed to it above is its claim to be weighed, not settled science.