Terminology

Tests

Tests is what we use to group all sorts of testing. Be it a simple AB test or a complicated multi-variant test. Improve doesn't judge with how you want to apply your tests.

AB tests

These type of tests are for optimizing specific parts of your platform. These answer questions like: "Would changing the CTA background color improve conversion?" or "If we show the newsletter popup one page later, does that improve the sign up rate?".

Implementation for these can be done using tests in Improve.

Feature Flags

Unlike tests, feature flags are intended for managed rollout of features. Say you have an e-commerce platform and want to release a new version of the cart page. You could develop and QA it for a few weeks and then roll it out, hoping no edge cases were missed. Feature flags can help you with this.

Flags make it possible to release to production in smaller batches. Setup a flag with 100% of the traffic to control, and release the first part of the new version behind the flag for the variation. This is now possible for your team to test by updating your cookie value.

After the entire feature is done and you're ready to get your first traffic you can start to increase the amount of traffic that gets the new version. If everything is operating smoothly you can set the traffic to 100% of the new version, and remove the control and feature flag. You've now successfully released a major part of the platform while always being able to roll back and avoiding a single, one-time major release 🎉.

Audience

An audience is a group of visitors, these can be bucketed by a few parameters like; OS, device type and such. Most parameters come from the visitor's user agent, so they resolve anywhere. Location (country) is the exception: the browser can't read it from the user agent, so country targeting only resolves during server-side / middleware bucketing (see the Next.js SDK and the server SDK's geo argument).

Analytic

These are events being sent from your platform to Improve and can be used to track the visitor on what they are doing. Events can be tied to tests to see if visitors accomplish the goal you want them to achieve. The percentage of visitors reaching your set goal is called the conversion.