An AI wrote our experiment readout, and we shipped it
What we learned building AI readouts: what the model is good at, where it needs hard limits, and how we check it.

A good experiment readout is a short document. It says what was tested, what happened, how sure we are and what to do next. Writing one takes an analyst an hour or two, and most teams skip it. We built a model that writes it for them.
What the model does not do
The model never computes a statistic. Every number in a readout comes from the stats engine, which is deterministic and tested. The model receives those numbers as structured input and writes about them. If it tries to state a figure that is not in the input, a checker rejects the draft.
What it does well
- Explains results in plain language for non-analysts.
- Notices that an effect differs by segment and says so.
- Reads linked replay metadata to describe what users did.
- Recommends ship, stop or extend, with a reason.
Where it needs limits
Early versions overstated confidence. A result with a wide interval would be described as promising. We fixed this by giving the model a fixed vocabulary tied to the interval, so a result that spans zero can only be described as inconclusive.
We also added a hard rule on multiple comparisons. If a test has been sliced twelve ways, the readout must say that a subgroup finding is a hypothesis, not a result.
How we check it
// every readout is validated before it is shown
const draft = await kerf.readouts.draft(experimentId);
const issues = kerf.readouts.validate(draft);
// issues: unsupported numbers, missing guardrail section,
// confidence language that contradicts the intervalWe sampled several hundred readouts and had analysts grade them. They disagreed with the recommendation in about 6% of cases, mostly on whether to extend a test or call it. That is close to the rate at which two analysts disagree with each other.
The best result was social. People started reading experiment results who never had before.
Kerf product team
The readout is a draft that a human can edit. We still think the human should sign off on anything that ships to a large audience.
Keep reading

Guardrails, not gates
Why the best experimentation programs automate safety checks instead of adding approval steps.

The winner's curse in A/B testing
Why the lift you measured is usually larger than the lift you will get, and how to plan for it.

Feature flags are debt, so pay it down
How to keep a flag system from turning into a thousand-branch maze, with ownership, expiry and cleanup tooling.