Results
How Improve counts visitors and conversions, decides when a result is significant, and keeps results up to date.
Open a test from the Monitor page to see its results. You get an overview, a table per variant, revenue and a funnel when your events support them, and a chart of results over time.
What gets counted
Results are built from two things: exposures (a visitor read the test's value) and the events you send. See Events for how to set them up.
- Visitors are everyone exposed to the test. That is the start step, your 100% baseline.
- A visitor belongs to the first variant they were exposed to. They stay there for the whole test.
- Only events sent after a visitor's first exposure count. Something they did before seeing the test can't be caused by it.
- Each visitor counts once per step. Ten purchases from one visitor is one conversion.
- If the test has an audience, only exposures that match it count.
The results table
One row per variant, with the number of visitors at each step and the conversion rate: conversions divided by visitors.
The first variant is the control. Every other variant is compared against it.
Significance
Improve compares each variant with the control using a two-proportion z-test at 95% confidence, on visitors and conversions.
- Significant means the difference with the control is unlikely to be chance. That works both ways: a variant can be significantly better or significantly worse.
- Best result in the overview is the significantly better variant with the highest conversion rate.
- No variant beats the control means some variants are significantly worse and none are better.
- No significant result yet means there isn't enough data to tell. Keep the test running.
The overview and the Monitor list show 🟢 when a variant beats the control, 🔴 when the only significant results are variants doing worse, and 🟠 while nothing is significant yet.
Open Stats for nerds for the direction, z-score and standard error of each comparison.
Don't stop a test the moment it turns significant. Results swing early on. Decide how long to run it up front. See What to A/B test.
Results over time
The chart shows, for each variant, how far its rate is from the control's, in percentage points. It is cumulative: each point includes everything up to that day. Pick the last 7, 30 or 90 days, or everything since the start.
Long tests are grouped into at most 31 points, so a test running for a year shows roughly one point per two weeks.
Revenue
When your conversion event carries a value, results also show total revenue and average order value per variant. Unlike conversions, every order counts here, including repeat purchases. See Revenue and order value.
Funnel
When a test has more than a start and a conversion event, results include a step-by-step funnel. See Funnels.
When results update
- Results are cached. Opening a test shows the cached result right away and recalculates in the background when it is older than 5 minutes.
- Sync analytics recalculates now.
- Changing a test's events recalculates right away, using the new events.
If your organization goes over its usage limit, results freeze until the limit clears. Events are still recorded, so nothing is lost and results catch up afterwards.
From an agent
The same results are available as JSON through the agent API. See Results.