Skip to content
Products
/
/
/
About A/B testing
Last updated

About A/B Testing

Introduction Video

Overview

A/B testing helps marketers test treatments (for example, different marketing Calls to Action in an email) to determine which treatments perform better. With Treasure Data's A/B Testing, you can compare one to eight versions of something to determine which performs better.

To accomplish this test, at a minimum, you need one test case and a control group to validate the results. A control group is important because it provides a benchmark against which to measure performance. A control group highlights what works as much as what doesn't.

Journey's support the following A/B test capabilities:

  • Unlimited A/B tests : There is no maximum number of A/B tests that you can add to a journey stage.
  • A/B test nesting : You can add an A/B test to an A/B test. Journey Orchestration supports 4 layers of A/B test nesting.
  • A/B test variants : Each A/B test can have up to eight variants with a single control group.
  • Customizable variant percentages : By default, percentages are split equally among variants. However, you can add specific percentages for each variant.
  • Unique ID field for consistency : (Optional) Add a Unique ID field (attribute) to make A/B test assignment more consistent over time.
  • Test group members' activation : Select to send the A/B test and A/B test group names to an activation destination within an activation.
  • Variant membership assignment : After an A/B test is created and launched, Treasure Data saves the following information with the parent segment for individual profiles:
    • Profile ID
      • cdp_customer_id
      • Stable ID (Unique ID in A/B testing)
      • Journey
      • Journey stage
      • A/B test
        • Unique ID
        • Name
      • A/B test variant
        • Variant ID
        • Name

How it Works

Let's assume you have created your A/B tests and launched them within a live Journey. When a parent segment refreshes and the profiles move in the journey flow, any profiles in an A/B test step that are not assigned to a variant are assigned a variant. Journey Orchestration keeps these variant assignments static after they are assigned unless a profile drops out of the stage based on the following criteria:

  • If a profile drops out of a journey stage, the profile is removed from being a member of all the A/B tests within that journey stage. For example, if Beth is in both A/B test A and B variants in stage 1, if she drops out of stage 1, Beth is removed from both Test A and Test B.
  • If a profile is assigned a variant, drops out of the journey, and then re-enters the journey, the profile should stay with the same variant assignment previously assigned.

See also Journey FAQs.

Previous page
Next page