Runbooks people actually follow

We had a good runbook library by the usual measures. Sixty documents, reviewed quarterly, linked from every alert. During an outage in March nobody opened any of them. Afterwards I asked why, expecting to hear that they were out of date. That was not the answer. They were accurate. They were just unusable at the moment they were needed, which is a different failure and one that a review cycle does not catch.

This is what I changed, and which changes actually held up over the following six months.

The reader is not who you imagined

The person reading a runbook during an incident is not a new starter calmly learning a system. They are someone with adrenaline, possibly woken up, holding a phone, already three minutes into something going wrong, and being asked for updates in a channel while they read. They have the reading comprehension of someone considerably less capable than themselves.

Almost every bad runbook is written for the wrong reader. It explains architecture before it says what to do. It uses the full formal name of every component. It assumes you will read it start to finish before acting. None of that survives contact with 3am.

The single most useful change was moving the first action to the first screen. Not the overview, not the prerequisites, not the escalation matrix. The first thing you do. Everything explanatory moved below it, and most of it turned out to be unnecessary once it was no longer in the way.

Commands must be copyable, not described

This was the biggest gap between what we thought we had written and what was actually there. A large share of our steps said things like "restart the ingestion service on the affected node". That is a description of an action, not an action. The reader now has to know the service name, the node, and the right way to restart it on that platform.

Compare:

Restart the ingestion service on the affected node.

against:

ssh ops@ingest-0{N}.prod.internal
sudo systemctl restart hopper-ingest
sudo systemctl status hopper-ingest --no-pager

The second one can be pasted. The first one requires reconstruction, and reconstruction under pressure is where mistakes happen. Every step that could be a literal command is now a literal command, with placeholders in an obvious format so it is clear what has to be substituted.

The related rule: if a command is destructive, the runbook shows the check that has to pass first, immediately above it, in the same block. Not in a warning callout three paragraphs up.

Say what success looks like

Our old runbooks told you what to do and stopped. They did not tell you how to know it had worked. So people ran the step, saw no error, and moved on, with no way of distinguishing "fixed" from "did nothing".

Every step now ends with an observable. Queue depth below a stated number. A specific log line. An HTTP 200 from a named endpoint. Something you can look at and get a yes or no from, with the actual expected value written down rather than left as "should look normal". Normal is exactly the thing you do not know at 3am.

This also exposed steps that we could not describe a success condition for, which was a useful signal on its own. Most of them turned out to be superstition, carried forward from an older version of the system, and we deleted them.

Write down the dead end

The part I expected to be controversial and was not. Runbooks record what worked. They almost never record what looked promising and was a waste of time. So every person hitting the problem for the first time spends the same twenty minutes on the same wrong path.

We added a short section at the end of each runbook for exactly this. No structure, no template, just a heading and a few lines. On the ingestion runbook it reads:

Restarting the consumer group looks like it should help and does not. The lag is upstream of it. Checked twice, once by me and once by Priya, six weeks apart.

That paragraph has probably saved more time than anything else on the page, and it took ninety seconds to write. It also gives people permission to record a negative result, which otherwise feels like admitting you wasted an afternoon.

What did not work

Two changes I was confident about and was wrong.

Templates. We built a strict template with required sections and a linter that rejected documents missing any of them. Compliance was high and quality went down. People filled in sections to satisfy the check, so we got a lot of prerequisites blocks reading "access to production" and overview blocks restating the title. The template made documents longer and less specific, which is the opposite of the goal. We kept a suggested shape and dropped the enforcement.

Quarterly reviews. Everyone signed off, nothing improved. The problem is that reviewing a document you are not currently using tells you whether it is accurate, not whether it is usable. We replaced the calendar with a rule: whoever uses a runbook during an incident edits it afterwards, the same day, while it is still fresh. Small, frequent, and driven by real use. Six months in, the documents that get used are noticeably better, and the ones nobody has touched turn out to be ones nobody needs.

The short version

If I were starting again with one page of guidance, it would be this. Put the first action first. Make every command copyable. State what success looks like with real values. Record the dead ends. Edit after using, not on a schedule. Let the documents nobody touches sit there, because their existence is telling you something too.