The Nearest Exit Might Be Behind You

The lead engineer was walking me through a rule he wanted to use while preparing an upcoming high-risk migration. Before anyone touched production, his plan was that every person on the team had to run the whole thing themselves, start to finish, alone, on a throwaway cluster. Not read the runbook. Not watch someone else do it on a Loom. Actually do it.

As a small engineering team, building discipline and practice, and avoiding a single point of failure are paramount for this department.

you practice so your body already knows

"I don't care if they fail. I just want them to see what it feels like." As a senior engineer, he knows the importance of practicing fire drills. If you never prepare and suddenly something goes sideways, you will panic.

Nobody does a fire drill because they can't find the exit. The exit is on a sign with an arrow pointing right at it. You do the drill so that when the alarm goes off for real, and your brain goes white, your legs already know where to walk.

Improv actually connects to this. The thing you drill over and over isn't the scene. You can't rehearse a scene that hasn't happened yet. You drill the fundamentals so when you walk onstage with nothing, no script, no plan, you succeed. The practice is the whole point. It's what lets you stay calm inside the unknown.

A migration runbook is the sign on the wall. Reading it tells you where the exit is. It does not tell you what it feels like when the command hangs for eight seconds, and your stomach drops because you can't tell if it's working or broken. You can't write that part down. You can only make someone practice it on a cluster that doesn't matter, so the first time their stomach drops isn't the time it truly does matter.

The engineer didn't assign the drill and wait for green checkmarks to roll in. He sat and watched someone do it live. Watched where they hesitated. Caught the questions they said out loud, the "wait, does this step come before or after" moments, and folded those gaps back into the process while they were still warm.

The questions people ask while they're doing the thing are nothing like the questions they ask while reading about it. Reading, you nod along. Doing, you hit the exact spot where the instructions assumed you already knew something you didn't. That spot is invisible until you actually watch someone or do it yourself. That's what a tabletop exercise is actually for. Not to prove the plan works. To find the place where the plan falls apart.

We say shared ownership a lot in product and engineering, and most of the time it means everyone's name is on the doc, and everyone nodded in the meeting. What it should mean is a team where each person has already been alone with the thing that breaks. So when the migration finally runs against production and something twitches, you don't have one person who knows and six people watching them sweat. Instead, you have a group of people who recognize the problem's shape because they are prepared.

The instinct on a small team is to lean on the two or three people who really know the system and call it covered. But two or three people who know is a single point of failure, and eventually it will fail. Engineering discipline isn't the shiny stuff. It's getting people ready for the coming failure, so they already know they can handle it when it shows up. Better they feel it now, on a cluster we're about to throw away, than the time they answer the alert that production has gone down.