Terminology

Tests

Tests is what we use to group all sorts of testing. Be it a simple AB test or a complicated multi-variant test. Improve doesn't judge with how you want to apply your tests.

AB tests

These type of tests are for optimizing specific parts of your platform. These answer questions along; "Would changing the CTA background color improve conversion?" or "If we show the newsletter popup one page later, does that improve the sign up rate?".

Implementation for these can be done using tests in Improve.

Feature Flags

Unlike tests, feature flags are intended for managed rollout of features. Say you have an e-commerce platform and want to release a new version of the cart page. You could develop and QA it for a few weeks and then roll out it out hoping no edge cases were missed. Feature flags can help you with this.

Flags make it possible to release to production in smaller batches. Setup a flag with 100% of the traffic to control, and release the first part of the new version behind the flag for the variation. This is now possible for your team to test by updating your cookie value.

After the entire feature is done and you're ready to get your fist traffic you can start to increase the amount of traffic that gets the new version. If everything is smooth operating you can set the traffic to 100% of the new version, and remove the control and feature flag. You've now successfully release a major part of the platform while always being able to roll back and no one time major release 🎉.

Audience

An audience is a group of visitors, these can be bucketed by a few parameters like; OS, device type and such.

Analytic

These are events being send from your platform to Improve and can be used to track the visitor on what they are doing. Events can be tied to tests to see if visitors accomplish the goal you want them to achieve. The percentage of visitors reaching your set goal is called the conversion.