Get Free Quote

An IncidentOps Case Story: One Long Day for a Scheduling Platform Team

Healthcare IT

The following account is fiction. The organisation, the roles and the timings were invented to illustrate a point, and no real team or product is described. It follows a small technology group that maintains an appointment-scheduling and reminder platform used by clinic administrators across several regions. The case is deliberately about administration and software operations: how the group applied IncidentOps practice over one long, awkward day. It contains no clinical content of any kind.

The practices in the story are taken from the Ops Atlas, an educational reference on operations topics. Its entry on IncidentOps and its working methods stresses that the name is a loose one and that the substance is ordinary incident management: declare, coordinate, communicate, learn. We will see all four.

Background: a team that relied on one hero

Before the day in question, the group had a familiar pattern. A senior engineer, whom colleagues called the person who always knows, answered every serious alert. Everyone else deferred. The arrangement worked until she went on leave. In her absence, the platform’s reminder service began failing at midday, and for the first thirty minutes nobody was sure whether to treat it as an emergency. Messages in the team chat piled up. Two engineers independently restarted the same service. A manager asked for an estimate and was told “soon”.

This is what the Atlas calls the first rung of its maturity ladder: heroic effort, with results that depend on who happens to be available. The group decided that afternoon to climb at least one rung.

The day it happened again

Six weeks later, with a basic routine in place, the reminder service failed again. The on-call engineer, who had been trained as a coordinator only the week before, saw the alert at 10:12. Their written criteria said to declare when users could see the effect, so they declared at 10:14 with a one-line chat message. At 10:16 they named the hats: they would coordinate; a colleague would do the hands-on work; a product analyst would write messages; and a trainee would keep notes.

The first public message went out at 10:19, inside the five-minute target the Atlas recommends: “Reminder messages are delayed. We are investigating. Next update by 10:40.” The status label read Investigating.

Mitigation without a theory

At 10:30 the hands-on engineer noticed that the failure began right after the previous evening’s release. There was no explanation yet. The coordinator weighed two options, waiting for a full diagnosis or rolling back, and chose the rollback, which could be undone if wrong. By 10:42 reminders were flowing again. The analyst posted “Identified” and then, after half an hour of steady behaviour, “Monitoring”. This mirrors the Atlas’s principle that generic relief, such as rolling back, needs only the location of the fault and not its root cause.

At 10:48 a director joined the chat and asked for an immediate cause. The coordinator replied with the courteous question many teams use: “Do you wish to take command?” The director smiled, said no, and moved to the updates space. Because the team had agreed on two spaces, a small working channel and a wider updates channel, the working channel stayed quiet and the notes stayed readable.

The unexpected second act

At 11:20, the rollback exposed a second problem: a backlog of queued reminders began releasing all at once, overloading a downstream message gateway. The incident had grown. The group’s own criteria said to widen the response if another team was needed, and the gateway belonged to a different team. The coordinator raised the severity, and two liaisons joined, one from the gateway team and one from customer support.

With more than five people involved, the coordinator split the effort into two sub-teams with pre-agreed labels, each with its own lead. The Atlas, citing emergency-service practice and a widely used response guide, mentions the guideline of about five people per supervisor and the idea of naming the groups beforehand so no one wastes time inventing labels mid-incident. The sub-team leads reported to the coordinator in short, regular check-ins. The backlog was throttled by 12:05, and a spike in user complaints faded.

A handover across time zones

The coordinator had been at the keyboard since 10:14. At 14:30, with monitoring still running and a clean-up of the backlog under way, a colleague in an office several time zones east took the hat. The exchange was plain: “You are now the incident coordinator, okay?” “Yes, I have it.” The scribe’s notes made the transfer easy, and the team did not need a long briefing. The Atlas describes a real cloud incident that crossed continents mid-way and highlights how routine such a handover should be, provided it is spoken and acknowledged.

At 16:00 the analyst posted the closing message: the service was fully restored, delayed reminders had been delivered, and a small number of reminders had been dropped and would not be resent. The last sentence matters. The Atlas recommends stating plainly whether anything was lost.

The review

The first coordinator had named the review owner at 10:20, long before the incident ended. The review took place two business days later, run by an engineer from a different team who had not taken part. The team followed the habits the Atlas lists: look at conditions and not at individuals, keep the follow-up list brief, and tie every entry to a single person and a calendar date.

The review surfaced several system conditions. The release process had no automatic check that reminders were still being delivered after deployment. The backlog behaviour had never been tested against the gateway’s limits. The status page lived on the same infrastructure as the platform, which would have been a serious problem had the outage been bigger. The group wrote down three actions:

  1. Add a post-release check that sends a test reminder and fails the release if it is not delivered (owner and date attached).
  2. Test backlog release against the gateway’s limits in a staging environment (owner and date attached).
  3. Move the status page to an independent host (owner and date attached).

Nobody was blamed for the release. One participant suggested adding a fourth and fifth action. The facilitator declined; the Atlas warns that longer lists tend not to be finished. The group also agreed not to close any action by downgrading it, a tactic the Atlas lists among the anti-patterns.

The scribe’s log, in outline

Because a trainee kept notes, the review started from a shared timeline instead of competing memories. In outline it read like this:

  • 10:12 alert received; 10:14 incident declared; 10:16 hats assigned.
  • 10:19 first public message; 10:30 likely release identified; 10:42 rollback complete.
  • 11:20 backlog overload noticed; severity raised; liaisons join; sub-teams formed.
  • 12:05 throttling in place; 14:30 spoken handover; 16:00 closing message.

No entry in that outline reads as dramatic, and that plain quality is the whole point of keeping it. The Atlas argues that a good record lets a team measure the phases separately: time to notice, time to acknowledge, time to reduce harm and time to restore. For this single event, the group could see that detection and acknowledgement were quick and that most of the elapsed time came from the second problem, not the first. With only one data point, they resisted the urge to compute any average, and they noted that a handful of incidents tells you little about trends.

What changed in the IncidentOps routine afterwards

The group went on to train five engineers in total as coordinators, so no person was the hero. They scheduled a short rehearsal every quarter, and they stopped using a single recovery-time figure in their reports, preferring ranges and a count of actions completed on time. Engineers interested in reliability practice may enjoy our write-up of the reliability-engineering skills that keep cloud services steady, and those curious about monitoring can look at this piece about smarter monitoring and incident handling.

Lessons from the story

The first lesson is the speed of the declaration: two minutes after the alert, not thirty. The second is the separation of hats, particularly the coordinator who does not touch the fix. The third is the willingness to act without a cause. The fourth is the rhythm of public messages and the honest closing note. The fifth is a spoken handover. The sixth is a short, owned list of follow-ups. None required new technology. All required agreement beforehand and a little practice, and the invented group’s second bad day was, as a result, a manageable one.