Guides
Exclusion groups and holdouts
Keep visitors out of overlapping tests, and measure what a set of tests added up to against visitors held out of all of them.
Two tests that change the same page overlap: a visitor in both makes each result harder to read. An exclusion group keeps a visitor in at most one of its tests. A holdout measures what a set of tests added up to, by keeping some visitors out of all of them.
Both need the updated Improve SDK on the site. An older SDK is served grouped and held-out tests at 0% allocation, so it shows every visitor the default, and the dashboard says an SDK update is needed.
Exclusion groups
A group divides 10,000 buckets between its tests: a visitor's bucket falls in at most one test's range, so no visitor sees two tests of the group at once. Buckets no test holds are outside all of them.
- Adding a test: only a draft joins — a running test's visitors already sit outside any range. It takes free buckets equal to its share, and keeps them while it runs.
- Growing a share takes free buckets, possibly a new range; shrinking gives back from its last range, and the visitors in them see the default. A running test's other ranges never move, so starting or stopping one doesn't move anyone in another.
- Removing a test from the group frees its buckets; it runs on its own allocation again. Any test that isn't running can leave (finish it, archive it, or move it back to draft).
- A test is in at most one group. Flags don't join groups: a flag is a feature, and leaving a visitor without it because another test took them would be a bug.
The group page shows the bucket bar: each test's ranges, the free buckets, and each share in percent. Adding or changing a share that changes what is served follows approvals, and can be scheduled or ramped.
How a visitor is bucketed
The visitor id (numbers' obelism_visitor_id) is hashed with the group's salt, modulo 10,000. That is the same in the browser, on the server and in results. For tests in a group the bucket decides whether a visitor is in, every time and before any cookie, so a cookie left from another visitor id can't put someone in two tests of a group. Inside the test, a valid cookie still keeps the variant.
Only tests in a group or under a holdout use this. Every other test keeps the random assignment and cookies it had, so running tests don't reshuffle.
Exclusivity lasts as long as the visitor id does. If you set numbers' storage to session or none, visitors get a new id per tab or per page, and a group is exclusive only within that.
Holdouts
A holdout keeps a share of visitors out of every test it covers: they get each test's default value, and Improve counts them under the holdout so the result can compare them with everyone exposed to a test in its scope.
- Its scope is the tests started while it runs, or chosen tests that haven't started yet. Tests in a group or not; flags are outside holdouts.
- It starts now or later, and ends by hand or at a set time. Ending it releases its visitors to the tests that run then.
- Its share is at least 0.01% of visitors, by the same bucket hash with its own salt.
The result
The holdout's page compares its held-out visitors with everyone exposed to a test in its scope, on the holdout's goal events, with the same always-valid method as results: the answer to "what did our tests add up to", always valid to read. The first goal decides; further goals are shown as secondary and aren't corrected for multiple comparisons.
Two things to know when reading it:
- The audience rule is applied to both sides. A held-out visitor is counted before any test's audience is checked, so the result re-applies the rule from what was recorded: everyone when no scoped test has an audience, the shared audience when they all do, and otherwise a visitor counts when they match at least one — an approximation, since a held-out exposure doesn't say which test the visitor reached.
- Below full allocation, the exposed side leans toward visitors who reach several tests. A held-out visitor is counted before allocation is checked. The result reads cleanest with the tests in scope at full allocation.
Where to find them
Groups and Holdouts have their own entries in the dashboard's navigation, and a test's page shows its group, share and ranges — and a notice naming the holdouts that keep visitors out of it. Lists mark a subject that is in a group or under a holdout, and its result updates within about five minutes.
Agents and MCP
listGroups, getGroup, createGroup, renameGroup, deleteGroup, addTestToGroup, setTestGroupShare and removeTestFromGroup, and listHoldouts, getHoldout, createHoldout, updateHoldout, endHoldout and getHoldoutResult, in the agent API and as MCP tools. Share changes on a running test and holdout changes follow approvals; creating a holdout is an admin's in a reviewed environment.