Multi-variant tests
What multi-variant tests are, how they differ from A/B tests, and when the trade-off is worth it.
An A/B test changes one thing. A multi-variant test changes several things on the same page at once, and tests every combination.
How it works
You add several elements (called "variants" in the editor), each with its own list of values. Example:
- Headline: "Start free trial", "Try it free"
- Hero image:
product-shot.png,lifestyle-photo.png
Improve builds every combination, 2 x 2 = 4 here, and splits traffic evenly across them. Results show one row per combination in the Monitor page.
Combinations multiply fast. 3 elements with 3 values each is 3 x 3 x 3 = 27 combinations, each getting a shrinking slice of traffic. Add elements sparingly.
Multi-variant vs separate A/B tests
| Multi-variant test | Separate A/B tests | |
|---|---|---|
| Measures | Every combination, including interaction effects | Each element on its own |
| Traffic needed | Most (splits across all combinations) | Less (each test gets full traffic) |
| Best for | A coordinated set of changes shipped together | Independent changes, tested one at a time |
Two separate A/B tests tell you which headline wins and which image wins. They won't tell you if a headline performs differently depending on the image next to it. That's what a multi-variant test catches.
When it's convenient
- You suspect the elements interact (best headline depends on which image is shown, or vice versa).
- The elements are shipping together anyway, as one coordinated release.
- You have enough traffic to cover all the combinations. See What to A/B test.
When to use a plain A/B test instead
- You're testing one idea. See What to A/B test.
- Traffic is limited. A 2-variant A/B test reaches significance much faster.
- The elements are unrelated. Two separate A/B tests get you both answers faster.
Setting one up
- In Tests, create a test and set its type to Multi-Variant test.
- Add an element for each thing you want to vary, then add its values.
- Start with two elements before adding a third.
- Read results the same as any test, one row per combination. See Events for wiring up the conversion event.