The winner's curse in A/B testing
Why the lift you measured is usually larger than the lift you will get, and how to plan for it.

A team runs twenty tests in a year. Eight win. Added together, the wins promise a 35% improvement in conversion. A year later the metric has moved 9%. Nobody lied. This is the winner's curse.
Where the curse comes from
Every measured lift is the true effect plus noise. When you pick the tests that cleared a significance bar, you pick the ones where the noise happened to be positive. The selection makes the average reported win larger than the real one.
The effect is worst when tests are underpowered. A small test can only reach significance if it measures a big number, so the tests that pass are exactly the ones that overshoot.
How big is it?
In a typical program with tests running at 50 to 70% power, the average winning lift is overstated by a quarter to a half. With very low power, the overstatement can exceed 100%.
What to do about it
- Size tests for the smallest effect you would act on, and run to that sample size.
- Shrink reported lifts toward zero before putting them in a forecast. Kerf shows a shrunk estimate beside the raw one.
- Rerun surprising wins. A holdout that keeps a small group on the old experience measures the true cumulative effect.
- Do not stop a test because it hit significance early unless you are using a sequential method built for it.
Sequential testing and the curse
Sequential methods let you stop early and keep valid error rates, but stopping early on a large effect inflates that effect even more. The answer is to report an interval, not just a point, and to treat the lower end as the planning number.
Plan with the bottom of the interval. If the bottom is still worth shipping, ship it.
Kerf data science
The curse never disappears. It becomes a known quantity, and a team that accounts for it makes forecasts that survive contact with the quarter.
Keep reading

Guardrails, not gates
Why the best experimentation programs automate safety checks instead of adding approval steps.

Feature flags are debt, so pay it down
How to keep a flag system from turning into a thousand-branch maze, with ownership, expiry and cleanup tooling.

An AI wrote our experiment readout, and we shipped it
What we learned building AI readouts: what the model is good at, where it needs hard limits, and how we check it.