Guardrails, not gates
Why the best experimentation programs automate safety checks instead of adding approval steps.

Most teams add process after something goes wrong. A bad release leads to a review board, and the review board leads to slower releases and the same number of bad ones. Gates feel safe because someone is watching. They rarely are, because the person approving a change has the least context about it.
What a gate gets wrong
A gate asks a person to predict whether a change is safe. A guardrail measures whether it is safe. The first depends on judgment under time pressure. The second depends on data that arrives whether or not anyone is looking.
What a guardrail is
A guardrail is a metric with a boundary that a change must not cross, whatever it does for the primary goal. Error rate, page load time, refund requests and day 2 return rate are common. When a boundary is breached, the rollout pauses on its own.
- Pick two to four guardrails per product area and use them on every test.
- Set bounds from historical variance, not gut feel.
- Choose metrics that move quickly. A guardrail that reports in a month is a postmortem.
- Name an owner who is paged when a rollout pauses.
Making guardrails trusted
Teams abandon guardrails that cry wolf. A good guardrail uses a non-inferiority test, which asks whether a metric got worse by more than a tolerance, rather than whether it changed at all. That cuts false alarms sharply.
The goal is that shipping a bad change is hard and shipping a good one is easy. Gates only make both hard.
Kerf engineering
Where to start
Take your last three incidents. For each, ask which metric would have shown the problem in the first hour of a rollout. That list is your first set of guardrails.
Then remove one approval step you no longer need, and keep an eye on how the guardrails behave. If nothing breaks, remove another.
Keep reading

The winner's curse in A/B testing
Why the lift you measured is usually larger than the lift you will get, and how to plan for it.

Feature flags are debt, so pay it down
How to keep a flag system from turning into a thousand-branch maze, with ownership, expiry and cleanup tooling.

An AI wrote our experiment readout, and we shipped it
What we learned building AI readouts: what the model is good at, where it needs hard limits, and how we check it.