1.Why Experimentation & MarTech Sprints is worth getting right
The module covers writing backlog items with a testable hypothesis attached from the start, refinement questions that force a stated success metric before a feature enters a Sprint, and the mechanics of flag hygiene, since a codebase littered with stale feature flags from experiments nobody remembered to clean up becomes a genuine maintenance burden and a source of confusing, contradictory behaviour. A scenario covers a marketing stakeholder who wants to launch a campaign this week that depends on a flag the engineering team has not yet built, forcing a negotiation about what is actually feasible inside the current Sprint versus what needs its own planning.
2.How it works in practice
The back half of the module connects the dots between shipped code and business outcomes explicitly: reading experiment results correctly, including statistical significance and sample size concerns that marketing teams often skip past, and presenting those results at Sprint Review in terms of the metric that moved, not just the feature that shipped. The goal is a team that can answer, for any recently shipped feature, what changed in the business as a result, rather than only what changed in the codebase.
3.Role reality
The gap between running experiments in principle and running them under Sprint pressure with a stakeholder waiting for a chart.
| Textbook theory | Delivery reality |
|---|---|
| The team ships stories every Sprint. | Shipping is not the point. Deciding whether the change moved a metric is the point. Ten stories that prove nothing is a wasted Sprint. |
| An experiment PBI is just a feature flag. | A real one names the hypothesis, the metric, the minimum detectable effect and the decision rule, before a single line is written. |
| More experiments running is progress. | More experiments closed with a decision is progress. Open experiments that never conclude are technical and organisational debt. |
| Guardrail metrics are optional detail. | Skip them and a conversion win that quietly breaks retention or trust looks like a success right up until it is not. |
| Flags can stay in the codebase indefinitely. | Every flag has a birthday and needs a death date. Undeleted flags are how companies drown in complexity nobody can explain. |
4.Core delivery pillars
Four disciplines that keep experimentation from becoming expensive reporting.
Name the hypothesis, the primary metric and how it will be judged in one sentence, so the experiment produces a decision instead of a debate.
A conversion win that damages retention or trust is not a win. Guardrail metrics are agreed before launch, not argued about after.
The flag is deleted, the events are archived, and the decision is recorded, the moment the experiment concludes, not weeks later.
A Sprint Goal like 'we learned X and shipped or killed accordingly' is honest. A Sprint Goal that only names the feature hides the real question.
5.Experimentation-specific metrics
The numbers that show whether the team is deciding things or just generating dashboards.
Ship, kill or default-on. Should approach total experiments started, with almost none left open indefinitely.
Net-positive forever is exactly how teams drown. Track both numbers side by side.
How often a primary metric win is later reversed because a guardrail regressed. Should fall as discipline improves.
The honest Retro question. A high count of charts with a low count of changed decisions means the tests are vanity.
6.Situations you will be asked to handle
The module puts you inside 2 decisions rather than asking you to recognise the right answer on a list. Each one is a situation practitioners meet, with several defensible options and consequences that follow from the one you pick. The scenarios below are the shape of the judgment the subject demands.
- Lesson 23.3: Game: pick the Sprint Goal for an experimentation iteration
- Lesson 23.4: Game: Definition of Done for experiments and MarTech
7.Common mistakes and why they fail
Shipping a feature without a stated success metric
Building and releasing a feature meant to be tested without agreeing, before the Sprint starts, on the specific metric that will determine whether it worked, which makes any later result impossible to interpret with confidence.
Letting feature flags accumulate without cleanup
Leaving stale flags from finished or abandoned experiments in the codebase, which creates confusing, hard-to-debug interactions between old and new behaviour and quietly becomes technical debt nobody owns.
Reading experiment results without checking statistical validity
Declaring a test result conclusive based on an inadequate sample size or before reaching statistical significance, which risks acting on noise as if it were a real signal.
Presenting Sprint Review purely in terms of features shipped
Demoing new functionality without connecting it to the metric it was built to move, which leaves stakeholders unable to judge whether the work actually mattered to the business.
8.Questions worth asking before you commit time to this
What belongs in the Definition of Done for a feature that will be A/B tested?
Instrumentation for the agreed success metric, a stated hypothesis, and a plan for how the flag will be retired after the test concludes, all decided before the Sprint that builds the feature, not improvised afterward.
How do you handle feature flag sprawl?
Treat flag cleanup as its own backlog item with an owner and a deadline tied to the experiment's conclusion, rather than leaving it as an unowned side effect. A flag that outlives its experiment by more than a Sprint or two should be treated as debt.
How much traffic or sample size does an experiment actually need before you trust the result?
Enough to reach the statistical significance threshold the team agreed on before the test started, calculated against the expected effect size and baseline conversion rate, not simply until the number looks convincing to whoever is watching the dashboard.
How should a team present experiment results at Sprint Review?
In terms of the metric that moved and by how much, with the statistical confidence behind that number, rather than simply announcing the feature shipped. Stakeholders need to see the connection between the code and the business outcome to give useful feedback.
9.What to remember
- Why experimentation teams cannot fake outcome-thinking the way output teams sometimes can
- What a real experiment PBI needs before it enters a Sprint
- A checklist for keeping flags and events from quietly drowning the platform
10.Where this sits in the Scrumling course
Turn feature flags, A/B tests and marketing telemetry into first-class Sprint content that connects code shipped to money moved.
About 35 minutes of lessons and decision scenarios.
- Lesson 23.1: Outcomes over outputs, actually
- Lesson 23.2: Writing PBIs for experiments
- Lesson 23.5: Feature flag hygiene and MarTech telemetry
- Lesson 23.6: Retros for experimentation teams
- Lesson 23.3: Game: pick the Sprint Goal for an experimentation iteration
- Lesson 23.4: Game: Definition of Done for experiments and MarTech
Assessment: Experimentation & MarTech Sprints quiz