A/B Experiments

The "A/B Experiments" module makes it possible on the one hand to test configurations of Makaira in a data-driven manner and on the other hand to play out different versions in the frontend. For example, different buyboxes or different page structures can be tested against each other.

For this purpose, the visitors to your website are divided into two groups - an A group, which receives the original setting, and a B group, which receives a different configuration. We also refer to this as the A and B scenarios.

By simultaneously tagging the users in Makaira Tracking, the optimal configuration can then be found. The following metrics are available for detailed evaluation:

• Orders
• Conversions
• Bounce rate
• Visit duration
• Turnover

📘

Currently, in addition to pure frontend tests, the ranking mix and machine learning/personalization are available for A/B testing. Other Makaira features will be further developed bit by bit for A/B test compatibility.

Requirements

• Use of a current version of Makaira Connect (from version 2.6.4) or Makaira Connect Legacy (from version 2021.2.3) or the current Makaira Storefront
• Makaira tracking is active (see Tracking -> Integrationen bzw. A/B-Testing-Tag für Google Tag Manager)

A/B test setup

• Switch to "A/B experiments" in the makaira
• Create a new A/B experiment
• Store all necessary test information
• Select traffic distribution (minimum 1% must remain with the original traffic).
  ATTENTION: Traffic distribution can not be changed after setup, so that the data can be evaluated correctly!
• Save.

Configuration of the "B scenario"

📘

This section is only relevant if you want to A/B test a Makaira configuration

The default configuration, which you can store in Makaira as usual, is always the "A" scenario

You can edit the "B scenario" by going to the A/B experiment list and clicking on "Configure". Makaira now switches to the configuration interface for the "B scenario" - recognizable by a yellow frame. Here you can only change the configuration that you have selected for the A/B testing, the menu is thinned out accordingly.

When you are done with the configuration, you can return by clicking "Finish editing" at the top.

Start of the A/B experiment

🚧

Before you start the A/B experiment, check again whether all settings in the original (A variant) and in the B variant (see "Configuration of the B scenario") are stored correctly. Subsequent changes can lead to falsified test results!

📘

Currently only one experiment can run per instance!

When you are ready, click on "Start Experiment". Now your customers will be served A or B scenarios based on the traffic classification you made.

You can always tell if an A/B experiment is running because a yellow badge is played in the menu bar at the top.

When you think you have collected enough data, you can stop the experiment by clicking the "Stop experiment" button. From this moment on, only the original (the A-scenario) will be played.

Evaluation of the A/B experiment

As long as an A/B experiment is running and after its completion, you can view the experiment results at any time. Just click on the small statistics icon in the A/B experiment overview.

Here you can choose according to which parameters you want to display the results:

• Orders
• Conversions
• Bounce rate
• Visit duration
• Turnover

All metrics are collected at all times, as long as tracking is set up correctly (see Prerequisites).

A/B testing in the frontend

Storefront

A helper function is available in the storefront that allows elements to be displayed or hidden based on the group assigned to the user.

This requires that the experiment IDs and variant IDs to be used are known at implementation.

Example:

function ExperimentTestComponent() {
  const { isInExperiment } = useAbTesting()
  
  if (isInExperiment({ id: 99, variation: 'Original' })) {
    return <span>Original implementation</span>
  } else if (isInExperiment({ id: 99, variation: 'B' })) {{
    return <span>Implementation of testcase B</span>
  } else if (isInExperiment({ id: 99, variation: 'C' })) {{
    return <span>Implementation of testcase C</span>
  }
}