Set up a holdout group
Keep a slice of users on the old experience so you can measure the cumulative effect of your program.

Why a holdout
Each test shows whether one change helped. A holdout shows whether the sum of your changes helped. It is the most honest way to answer a finance team that asks what the program is worth.
Choose the size
Two percent is a good default for a large product. Smaller products may need five. The group must be large enough to detect an effect of a few percent on your main metric within a quarter.
Create it
kerf.holdouts.create({
key: "program-holdout",
percent: 2,
exclude: ["security", "billing"],
resetEvery: "6 months",
});Make exclusion automatic
Every flag should skip holdout users by default. If engineers must remember to do it, they will forget on the one test that matters.
What to leave out
- Security fixes and legal changes should reach everyone.
- Bug fixes that you would never want to withhold.
- Anything that affects billing correctness.
Read it quarterly
Compare holdout and main groups on your north star with a confidence interval. Expect noise early, and do not react to the first month.
Reset on a schedule
Release the holdout back to the main population every six months and draw a new one. This keeps the old experience from drifting too far from what you can maintain.
More playbooks

Run your first checkout test
A step by step plan for testing a checkout change without risking revenue.

Test your pricing page
How to test plan layout, anchoring and copy on a pricing page without changing what anyone pays.

Improve onboarding activation
A playbook for finding your activation event and running a steady series of onboarding tests.