Holdouts at scale: measuring what your program is really worth
A long-running holdout group answers the question every experimentation team gets asked: is all of this working?

Individual tests tell you whether a change helped. They do not tell you whether your program, taken as a whole, is worth the cost. For that you need a holdout.
What a holdout is
A holdout is a small slice of users, usually 1 to 5%, who never receive the winners of your experiments. They stay on the old experience while everyone else moves forward. The gap between the two groups over time is the real cumulative effect of your work.
Why it is harder than it sounds
- Holdout users must be excluded from every launch, which takes discipline in code.
- Interactions between wins can make the total smaller or larger than the sum of parts.
- A holdout that lasts a year accumulates software that is a year behind, which becomes hard to maintain.
How to run one
Assign the holdout by user id with a stable hash, and make it a first class targeting rule, so every flag excludes it by default. Reset it on a schedule, quarterly or twice a year, so the maintenance burden stays bounded.
kerf.holdouts.create({
key: "program-holdout",
percent: 2,
resetEvery: "6 months",
});Reading the result
Compare the holdout to the main population on your north star metric. Expect the confidence interval to be wide at first. Two percent of a large user base is still a lot of people, but effects of a few percent need months to show.
The holdout is the only number we could take to finance and defend line by line.
Head of Growth, customer
If the holdout shows no gap, that is also an answer. It usually means the tests are measuring short term novelty, and the team should lengthen its test windows.
Keep reading

Guardrails, not gates
Why the best experimentation programs automate safety checks instead of adding approval steps.

The winner's curse in A/B testing
Why the lift you measured is usually larger than the lift you will get, and how to plan for it.

Feature flags are debt, so pay it down
How to keep a flag system from turning into a thousand-branch maze, with ownership, expiry and cleanup tooling.