Making experimentation a Sprint deliverable, not an afterthought
Teams running feature flags and A/B tests alongside marketing technology often treat the experiment itself as separate from the Sprint's real work, something that gets configured after the code ships rather than planned as part of the same Increment. This module argues that an experiment's instrumentation, hypothesis, and success metric are part of the Definition of Done for any feature meant to be measured, because a feature shipped without a way to measure its actual impact is a guess dressed up as a delivery.
The module covers writing backlog items with a testable hypothesis attached from the start, refinement questions that force a stated success metric before a feature enters a Sprint, and the mechanics of flag hygiene, since a codebase littered with stale feature flags from experiments nobody remembered to clean up becomes a genuine maintenance burden and a source of confusing, contradictory behaviour. A scenario covers a marketing stakeholder who wants to launch a campaign this week that depends on a flag the engineering team has not yet built, forcing a negotiation about what is actually feasible inside the current Sprint versus what needs its own planning.
The back half of the module connects the dots between shipped code and business outcomes explicitly: reading experiment results correctly, including statistical significance and sample size concerns that marketing teams often skip past, and presenting those results at Sprint Review in terms of the metric that moved, not just the feature that shipped. The goal is a team that can answer, for any recently shipped feature, what changed in the business as a result, rather than only what changed in the codebase.
Mistakes teams make with this material
Building and releasing a feature meant to be tested without agreeing, before the Sprint starts, on the specific metric that will determine whether it worked, which makes any later result impossible to interpret with confidence.
Leaving stale flags from finished or abandoned experiments in the codebase, which creates confusing, hard-to-debug interactions between old and new behaviour and quietly becomes technical debt nobody owns.
Declaring a test result conclusive based on an inadequate sample size or before reaching statistical significance, which risks acting on noise as if it were a real signal.
Demoing new functionality without connecting it to the metric it was built to move, which leaves stakeholders unable to judge whether the work actually mattered to the business.
Questions people ask
What belongs in the Definition of Done for a feature that will be A/B tested?
Instrumentation for the agreed success metric, a stated hypothesis, and a plan for how the flag will be retired after the test concludes, all decided before the Sprint that builds the feature, not improvised afterward.
How do you handle feature flag sprawl?
Treat flag cleanup as its own backlog item with an owner and a deadline tied to the experiment's conclusion, rather than leaving it as an unowned side effect. A flag that outlives its experiment by more than a Sprint or two should be treated as debt.
How much traffic or sample size does an experiment actually need before you trust the result?
Enough to reach the statistical significance threshold the team agreed on before the test started, calculated against the expected effect size and baseline conversion rate, not simply until the number looks convincing to whoever is watching the dashboard.
How should a team present experiment results at Sprint Review?
In terms of the metric that moved and by how much, with the statistical confidence behind that number, rather than simply announcing the feature shipped. Stakeholders need to see the connection between the code and the business outcome to give useful feedback.
A question from this module's assessment
One sample question with the reasoning, so you can judge the level before you start. The rest of the assessment stays inside the module.
Which is the strongest shape for an experiment PBI?
- 'Ship it behind a flag and see'
- Hypothesis, primary metric, guardrails, minimum detectable effect, decision rule and flag owner all named up front
- 'Improve conversion'
- 'Add a new landing page'
Without the decision rule and the metric, the experiment produces an argument, not a decision.