Using EBM to Improve: Hypotheses and Experiments

Overview

• What you’ll learn: How to formulate improvement hypotheses, design experiments, define success criteria, interpret results, and create a culture of experimentation.
• Prerequisites: Lessons 40–45 (all EBM concepts).
• Estimated reading time: 15 minutes

Introduction

EBM without experimentation is just “evidence-based talking.” The real power of EBM comes from actually running experiments, measuring outcomes, and learning from results. This lesson teaches you how to go from “we should try something” to “we tested X, learned Y, and will do Z next.”

The Hypothesis Template

Every improvement starts with a hypothesis. Use this template:

We believe that [action/change]
will result in [expected outcome]
as measured by [specific metric]
We will know we succeeded when [success threshold]

Example: “We believe that adding WIP limits to our Kanban board will result in shorter cycle times, as measured by average cycle time per item. We will know we succeeded when average cycle time decreases by 20% over 3 Sprints.”

Designing Good Experiments

A good experiment is:

• Small: Implement the minimum change needed to test the hypothesis. Don’t change five things at once — you won’t know which one caused the result.
• Time-bound: Define how long the experiment will run (usually 1–3 Sprints).
• Measurable: You must be able to measure the outcome. If you can’t measure it, you can’t learn from it.
• Reversible: If the experiment fails, you should be able to revert without significant damage.

Interpreting Results

After the experiment period, inspect the data:

• Confirmed: The metric improved as hypothesized. Consider making the change permanent and run a new experiment.
• Disproved: The metric did not improve or got worse. This is not a failure — it is learning. Roll back the change and try a different approach.
• Inconclusive: Not enough data, too many confounding variables, or the change was too small to measure. Extend the experiment or redesign it.

Creating a Culture of Experimentation

The hardest part of EBM is not the framework — it is the culture. Experimentation requires:

• Safety to fail: If failed experiments are punished, nobody will experiment. Celebrate learning, not just success.
• Data literacy: People must understand what the data means and doesn’t mean.
• Patience: Results take time. One Sprint is rarely enough to see the impact of a process change.
• Discipline: Run one experiment at a time (or at least control for variables). Don’t change everything simultaneously.

Connecting Experiments to Goals

Every experiment should connect to your goal hierarchy: Tactical Goal (Sprint) → Intermediate Goal (quarter) → Strategic Goal (organization). If you cannot explain how an experiment connects to a strategic goal, it may not be worth running.

Key Takeaways

• Use the hypothesis template: “We believe [action] will result in [outcome] as measured by [metric].”
• Design experiments that are small, time-bound, measurable, and reversible.
• Disproved hypotheses are learning, not failure.
• Create safety to fail, data literacy, and discipline for experimentation.
• Connect every experiment to your goal hierarchy.