Experiments, done properly
A/B tests, multivariate tests and long term holdouts, with power worked out before you launch and sequential testing so you can look early without fooling yourself.
Pick the design that fits the question
A/B tests
One change, one control. The default for most decisions, with CUPED variance reduction to reach a result sooner.
Multivariate tests
Test headline, image and button together and see which combinations matter and which only look like they do.
Holdouts
Keep 5% of users on the old experience for a quarter to measure the cumulative effect of everything you shipped.
Painted door tests
Measure demand for a feature before you build it, with a button that leads to a waitlist.
Switchbacks
For marketplaces and pricing, alternate treatments by time window when user level assignment would leak.
Mutual exclusion
Put related tests in a layer so a user is only ever in one of them and results stay clean.
Statistics that don't need a statistician
Kerf runs sequential tests by default, so a peek at day three does not inflate your false positive rate. Confidence intervals are shown instead of a bare p value, and every result carries the checks that decide whether to trust it.
- Sequential testing with always valid p values
- CUPED using pre experiment behaviour
- Sample ratio mismatch alerts within an hour
- Bonferroni and Benjamini Hochberg corrections for many metrics
What teams see after switching
Launch a test
- 01
Write the hypothesis
State the metric you expect to move and by how much. Kerf uses it to size the test.
- 02
Plan the sample
See how many users and how many days you need, given your traffic and the smallest effect worth shipping.
- 03
Add guardrails
Pick the metrics that must not get worse. Kerf watches them for the whole run.
- 04
Launch and read
Start the test. The AI readout is drafted when it ends, with a ship, iterate or kill recommendation.
Experiment questions
Can I run tests on the server and client together?
Yes. Assignment is the same hash everywhere, so a user sees the same variant on the web, in the app and in an email.
How many metrics can one test have?
As many as you like. One is the primary, the rest are secondary or guardrails, and we correct for multiple comparisons.
Do you support Bayesian analysis?
Yes. Switch the engine per workspace or per test. Frequentist sequential is the default.

