Most hospitals discover their automation estate the unpleasant way: an incident reveals a script nobody remembered approving. The path out is mechanical rather than heroic, and it fits in ten steps. The framework behind them is the Ops Atlas AutomationOps entry, which treats every unattended job — the restarts, the purges, the failovers, the credential rotations, the configuration pushes across a whole fleet — as production equipment requiring an owner, a risk grade, and a set of guards. Move through the steps sequentially; every one builds on what the last produced. The whole sequence fits into a quarter of part-time effort — no procurement, no platform migration, no external consultancy required at any point.
Phase one: prepare the AutomationOps register (steps 1–3)
- Census the estate. List every job that runs unattended: crontabs, schedulers inside applications, the automations bundled with commercial systems, the scripts on individual workstations. Include the ones nobody will admit to. The census is complete when the list embarrasses someone.
- Name an owner for each entry. A current employee, by name, with a deputy. Any job without a candidate for owner is your first candidate for retirement.
- Write the register as code. Move the census into a versioned file — one entry per job stating its tier, limits, identity and review date. YAML is conventional. From this point the register can be reviewed line by line, compared across versions, and never again misplaced in a drawer.
The register deserves emphasis because everything downstream reads from it. Approvals, guard tests, audit questions — all of them consult the file, which is why the Atlas treats the declarative register as the spine of the whole practice. Two practical notes for step 1: census in person as well as in system — walk the departments, ask what “runs by itself”, and write down whatever makes someone pause; and date the census, because an estate map without a date is a rumour in six months. Most hospitals completing an honest census find between fifteen and forty unattended jobs, of which two or three genuinely frighten them. That fright is useful. It is the sensation of risk becoming visible, and it should be written into the register’s first entry.
Phase two: protect the estate (steps 4–8)
- Grade every job by consequence. Tier 0 for read-only work; tier 1 for a reversible change touching one target; tier 2 for the same kind of change landing on many at once; tier 3 for whatever is irreversible or destructive to data. The grade decides everything else.
- Fit the five guards, scaled to tier. Dry run before every change; refusal of empty or suspiciously changed target lists; batching with a canary slice for wide runs; a failure budget that halts the run automatically; kill switches at job and estate level. Tier 3 additionally requires second-person approval, soft deletes and staged rollout.
- Convert doing to ensuring. Where possible, rewrite jobs to seek a state — “ensure running” rather than “start” — so repeats are harmless. For inherently once-only actions (create, charge, send), require caller-supplied request IDs recorded atomically.
- Set the reach ceiling. Configure batch sizes explicitly: ordered groups, a maximum failure percentage per group, and a full stop when the ceiling trips. Wide jobs lose their ability to do wide damage.
- Test the design, not just the run. Add static checks — shell and playbook linters — plus behavioural tests that include an idempotence pass, so a job that stopped being safe to repeat fails loudly in the pipeline instead of quietly in production.
One rehearsal caveat belongs in writing: at least one widespread configuration tool omits command-level probes from check mode by default, so a clean prediction can still hide a damaging step. Rehearsals are evidence only when someone has read what they actually cover.
Two tuning details separate working guards from decorative ones. Failure budgets need a number chosen for the job’s own history, not a default: a job that has failed once in two years earns a budget of one, while a chatty polling job may need two failures within a single run before halting. And batch ceilings should be sized against recovery, not throughput — a ceiling of ten targets means any incident touches ten recoverable units, a number a duty manager can absorb at 3 a.m. without waking the whole building. Guard values belong in the register next to each job, with the date they were last revisited, so the next person tunes from history instead of guessing again.
Phase three: prove it works (steps 9–10)
- Drill the stop. Twice yearly for tiers 2 and 3, halt one job and the whole estate on purpose, during quiet hours. Measure time to stop. Write down every dependency the stop revealed — the whiteboard fed by the reminder job, the report the finance team reads — and give each a manual fallback.
- Keep the monthly scoreboard. Hours genuinely returned — each run’s saved minutes minus its tending minutes, summed and divided by sixty — plus success and human-rescue rates, induced-incident rate in DORA’s mould, guard trips, drill-measured stop time, ownership coverage against the 90% mark, and the tally of jobs still missing from the register. Publish all eight, monthly, to someone with authority to act on them.
How the AutomationOps phases compound
The sequence is deliberate. Preparation creates the record; protection uses the record to place guards where consequence justifies them; proof closes the loop with drills and numbers. Skip a phase and the later ones misfire — guards without a register protect the wrong jobs, and drills without measurement produce anecdotes instead of evidence. Teams that complete all three phases usually notice a subtler change: automation stops being discussed as a pile of conveniences and starts being discussed the way fleets, pharmacies and laboratories are — as infrastructure with known limits and named accountable people. Meetings shorten, because the register answers the questions that used to open them. Vendor conversations sharpen, because a buyer holding a register and a scoreboard asks questions sales collateral rarely preempts. And incidents, when they eventually occur, arrive smaller: a guard trips, a slice of ten fails instead of a fleet of four hundred, a run halts at its budget — and the post-incident review reads from the register instead of reconstructing history from memory.
Who should run the ten steps
Not a committee. One person runs the sequence — an IT lead, a senior technologist, a digitally fluent operations manager — with a standing hour of clinical-side attendance each week for the questions only clinicians can answer. Committees draft documents; a named individual completes steps, and this discipline progresses through completed steps. The clinical side matters more than technologists expect: only a senior nurse can say which report the night shift actually reads, and only the appointments lead knows which reminder wording reduces no-shows. The ten steps are administrative scaffolding; the judgement that fills them comes from both sides of the hospital.
What promotion looks like afterwards
Change in this discipline is earned by evidence, and the Atlas sketches five maturity levels for calibrating expectations: a climb from ad-hoc habits through shared and guarded practice to governed and, rarely, self-correcting operation. Most organisations stall at the shared rung — everything in one repository, goodwill everywhere, ownership nowhere — because sharing feels productive and asks little. Promotion to “guarded” is exactly what the ten steps above deliver; “governed” arrives when the register drives approvals automatically; “self-correcting” is the state where guard trips feed the register and the register tightens the guards. Aim for one level per quarter, promoted with the scoreboard as evidence, and expect the estate to keep its gains through staff turnover — because for the first time, the knowledge lives in a file rather than in someone’s head. The levels also give leadership a shared vocabulary for progress: “we are guarded, moving to governed” tells a board more than any activity report, and it commits the team to evidence rather than adjectives.
Hospital technology teams work at this intersection already, and several threads on this site extend the same ideas: AIOps observability for smarter monitoring and incident management covers the watching half that feeds candidates to this discipline, while robot fleet management for robotics software teams applies ownership thinking to machines that move. And for a clinical view of technology earning patient trust, LASIK hospitals, technology, safety and patient care makes the same argument the Atlas does: precision without governance is just speed with better handwriting.