Get Free Quote

The Reminder Model That Went Quiet: A Diagnosis Story

Healthcare IT

The front-office coordinator at a mid-sized eye hospital noticed it first, not through a chart but through a feeling. The reminder system that nudged people about upcoming visits had seemed sharp for months. Lately the day’s no-show list felt longer than the predictions said it would be. Nothing was broken in the usual sense. The system ran every morning, the messages went out, and no error appeared anywhere. This is the kind of quiet failure that MLOps is built to catch, and the story below, an invented composite for illustration, walks through how a team might trace it from symptom to cure. It concerns administration only and gives no clinical information.

The symptom: a good model that stopped being good

Predictive models do not usually fail loudly. They simply get less right while remaining perfectly available. The Ops Atlas entry on keeping machine-learning models correct in production says as much when it sets out monitoring in three layers, covering service health, incoming data and result quality. The team had only the first layer. The service level was green, so everyone assumed all was well.

Quality was hard to see for another reason. A prediction about a missed visit is only confirmed once the visit date passes, so the true answers lag behind. The Atlas describes this situation and recommends interim proxies, for example how predictions are distributed and how the predicted rate sits against a base rate once it matures. When the coordinator asked for that comparison, the predicted rate was visibly lower than what had actually been happening.

First suspect: the world changed

The natural guess was that patient behaviour had shifted. A new bus timetable, a change in clinic hours, a flu season in the schedule. The data lead agreed it was plausible but pointed out that the Atlas separates three possible causes of a fading model, and that jumping to the first would risk a pointless retrain. The three are a disagreement between the training and live pipelines, a defect upstream, and a real shift in the data. Only the last justifies retraining.

The team set up a table to test each in turn.

Second suspect: a mismatch between training and live inputs

The data lead ran a replay. She took a few hundred recent bookings, built the model’s inputs the way the training code does, built them again the way the live system does, and compared. The Atlas names this a parity replay and says the mismatch rate should be exactly zero. It was not. One input, the number of days since the last visit, was computed in training from the date of the appointment, but the live system computed it from the date the reminder was generated. The two differed by a day or more on most records.

That was skew, and it explained a good deal. The rule from the Rules of Machine Learning, cited in the entry, is to reuse code between the two paths and log the inputs that were actually served. The team had done neither. Nothing about the world had changed. Their own pipelines disagreed.

Third suspect: a timing leak

While tracing the skew, the data lead found a smaller issue in the training set. A field recording whether the person had rescheduled was filled in after the booking date but still ended up in the training rows for earlier dates. The model had effectively seen part of the answer. This sort of leakage is a recognised source of inflated results; the Atlas mentions a survey of published work in which such errors affected hundreds of papers across many fields, and it recommends held-out data fixed by date, plus features that respect the moment the prediction is made. A feature-store project mentioned in the entry offers a timestamp filter for point-in-time correctness, though it is explicit that such a store is not a general pipeline system.

The fix, in order

  1. Repair the pipeline, not the model. The live system’s date arithmetic was changed to match training, and a shared function replaced the two copies.
  2. Close the leak. The rescheduling field was removed from rows where it could not have been known at booking time.
  3. Retrain once, on purpose. With the pipelines aligned, the team trained a new version, recording the commit, data snapshot, settings and environment alongside it.
  4. Put the new version through a gate. It was measured against the incumbent using one shared hold-out sample, overall and by group, with a latency check. One group, first-time visitors, initially did worse, and the version was held back until the team corrected the cause.
  5. Release in stages. Shadow mode first, then a small share of reminder traffic.

The coordinator’s regular habit of writing down plans, much like the weekly routines in a piece about keeping a student’s monthly expense tracker going, turned out to be useful here, because each of these steps left a dated note.

Making sure it does not recur

The team added what had been missing. A parity replay now runs automatically with every release. A data check watches the incoming bookings for sudden changes in missing values and formats. A quality proxy appears on the same dashboard as service health, with alert limits calibrated on the hospital’s own history rather than a vendor default, which the entry says should not be treated as truth. They also wrote down a retraining policy listing triggers such as a schedule, new data and a sustained drop in the proxy, and agreed that nothing would be promoted without a human sign-off.

Finally, they rehearsed going back. On a quiet afternoon the production pointer was moved to the previous version and back again, and a plain fallback, a fixed reminder schedule, was confirmed to work. The Atlas recommends repeating such drills quarterly and insists that recovery means shifting traffic or an alias, not retraining.

What to take from the story

Diagnosis came before treatment. A plain-looking decline turned out to be a pipeline fault and a leak, not a change in the world, and retraining first would have baked both problems into a new model. Teams working on online services will find the same caution useful, for instance when reading about how care is delivered remotely in an overview of remote eye-care services, where reliable systems behind the scenes matter as much as the service itself.