Experimentation glossary
Twenty four terms, each in one sentence. Use it to settle an argument or onboard someone new.
Terms
A to F
- A/B test
A controlled experiment that splits users between two versions to see which performs better.
- Bucketing
The process of assigning a user to a variant, usually by hashing their id.
- Confidence interval
A range of values that likely contains the true effect, given the data.
- Control
The existing experience that a variant is compared against.
- CUPED
A technique that uses pre-experiment behavior to reduce variance and shorten tests.
- Exposure event
A record logged when a user first sees a variant.
- Feature flag
A switch in code that turns a feature on or off without a deploy.
G to M
- Guardrail metric
A metric that must not get worse, whatever happens to the goal.
- Holdout
A group kept on the old experience to measure cumulative impact.
- Hypothesis
A specific, testable claim about what a change will do and why.
- Kill switch
A flag that disables a feature immediately in an emergency.
- Lift
The relative change in a metric between a variant and control.
- Minimum detectable effect
The smallest change a test is designed to reliably detect.
N to S
- Novelty effect
A temporary change in behavior caused by something being new.
- Power
The probability that a test detects a real effect of a given size.
- p-value
The chance of seeing a result at least this extreme if there were no real effect.
- Progressive rollout
Releasing a change to a growing share of users in stages.
- Sample ratio mismatch
A split of users that differs from the planned allocation, a sign of a bug.
- Sequential testing
A method that allows results to be checked at any time without inflating error.
T to Z
- Targeting rule
A condition that decides which users receive a flag or variant.
- Variant
One version of the experience shown in a test.
- Variance
How much a metric varies between users, which drives how long a test takes.
- Winner's curse
The tendency for selected winning results to overstate the true effect.