SMS
      Back to home
      1. Recart Help Center
      2. SMS

      Popup A/B Testing

      Run an experiment and display two or more different pop-ups concurrently to test which one works best with your clients.

      Table of Contents

      What is A/B Testing? 

      • A/B testing, also known as split testing, is a method widely used in digital marketing to assess the effectiveness of different variations of a webpage or elements within it.
      • When it comes to both desktop and mobile popups, A/B testing becomes a valuable tool for optimizing user engagement and conversion rates.
        • It involves presenting two or more versions of a popup to different segments of your audience and analyzing the performance of each variant.
      • This allows you to determine which popup design, content, or call-to-action resonates best with your target audience, leading to informed decisions that enhance the overall user experience and maximize desired outcomes, such as increased sign-ups, sales, or other conversions.
      • You are not limited by how many popups you’d like to test at the same time.
        • However, please note that each popup you’re testing will be taking a portion of your incoming traffic- which means, the more popups you test at the same time, the smaller volume of traffic will go into each popup’s testing data. This may ultimately impact how you determine what is statistically relevant.

      How to create a popup experiment

      You can create a popup experiment to test different versions against your current active popup.

      Desktop example

      Current Active Popup New Popup to Test Against
      • 10-second entry delay
      • 4-second entry delay
      • lightbox layout
      • fullscreen
      • no teaser step
      • teaser step
      • one-step opt-in
      • two-step opt-in
      • reply in SMS as a consent
      • in popup confirmation (with 1-time code)
      • close icon 0-second delay
      • close icon 2-second delay
      • 10% discount offer
      • 20% discount offer

        VS

        Create a new popup


        1. Go to your Opt-in Tools and click Create new

        2. Create your new popup and edit it with the differences you would like to compare

        3. In Settings, toggle A/B Testing to Active and click Create new

          AB testing - Turn on under settings-1

        4. Name the experiment, select the start and end date & time, and click Add

          AB testing - name experiment

        5. The A/B Experiment will now show → Click Save as INACTIVE

          • as you already have an active popup, it is important to save this one as inactive, otherwise, you will have 2 popups showing at the same time which is a bad user experience

          • this will allow your inactive popup to be automatically activated when the experiment begins

            AB testing - set as inactive

        Update your currently active popup

        1. Go to your current active popup, toggle the A/B Testing to Active, and select the experiment you previously created.

          • Click Save as ACTIVE
          • NOTE: You will be able to see which opt-in tools are participating in the experiment

            AB testing - current with test active

        2. Your popups will be visible in your opt-in tools. 

          • Your original active popup will still be active
          • The inactive popup will remain so until the experiment begins
          • Once the experiment begins, the inactive popup will automatically be activated by Recart

            AB testing - activeinactive

        3. Once the experiment is live and throughout the running time, you will see:

          1. all popups involved in the experiment are Active

          2. a green dot next to the experiment name under the participating popups


        ❗Important things to note❗

        • You need to create separate A/B tests for mobile and desktop
          • TEST 1  → desktop 1 vs desktop 2 
          • TEST 2 → mobile 1 vs mobile 2
        • As mentioned above, the inactive popup will automatically be activated once the experiment begins.
        • Once the experiment is complete, the losing popup will automatically be deactivated and the winning popup will remain active.
        • The determining stats will only be pulled for the period that the experiment is running.
        • You can run multiple experiments at the same time, for example, you can have two different experiments running for both mobile and desktop concurrently.
        • If you would like to test two completely new popups, you will first need to create one and activate it before proceeding with the above experiment steps.

        View the results of the popup A/B test

        1. Once your A/B testing experiment has ended, the results will be displayed as a banner on your Opt-in Tools Page

          1. You can check the results here

            AB12

        2. You can compare the:

          1. impressions

          2. opt-in rates

          3. opt-ins

          4. new subscriptions


            NOTE: This will only reflect data collected during the running time of the experiment.

        3. Recart will calculate the popup that performed the best and the winner will automatically remain active, while the inferior popup is automatically deactivated.

          • You can still manually activate or deactivate these if you choose

            AB13

        4. You can see the results of all of your experiments at any time by using the filter function

          • Go to the filter drop-down menu

          • Under A/B testing select the experiment you would like to view

          • Click Apply

            AB14

          • The results of your experiment will be shown on this page

            AB15

        Configuring 3rd-Party A/B Testing for Recart Popups

        Recart allows you to integrate external A/B testing tools to optimize popup performance beyond our built-in A/B testing feature. To successfully implement external testing, three key pieces of information needs to be provided.

        What Information Is Required?

        To configure external A/B testing, you would need the following details:

        1. JavaScript Value to Access A/B Testing

        This is a special variable or function stored in the website’s code that helps determine which popup variant to display. If you are unsure, you can provide documentation for your A/B testing tool so our developers can confirm the correct value.

        • If a simple variable is used → Recart will check whether its value matches the popup’s Variant ID. If it does, that popup will be shown.
        • If a function is used → Recart will run the function using the Experiment ID. If the function returns a result matching the Variant ID, that popup will be displayed.

        📌 Note: If the function produces a different type of result, a custom solution may be required, which would involve additional development time. 

         

        Examples of JavaScript Variables from Popular A/B Testing Tools:

        A/B Testing Tool JavaScript Value Example
        Intelligems window.igData.getTestGroup
        Outsmartly parent.APP_CONFIG.getAssignedVariation
        VWO window.vwo

        2. Variant ID

        • This is the unique identifier that represents a popup as a participant in an A/B test.
        • It ensures that a specific popup is displayed to a specific group of users in the experiment.

        3. Experiment ID (Optional)

        • The Experiment ID refers to the entire A/B test rather than an individual popup.
        • While optional, providing this ID may help in cases where multiple experiments are running simultaneously.

        Additional Setting: Default Visibility

        What happens if no result is received during evaluation?

        It is important to specify whether the popup should be shown by default or hidden if no A/B test result is returned.

         

        Example: Configuring Intelligems for Recart

        To retrieve the correct settings from Intelligems, follow these steps:

        1. Open the A/B Test in Intelligems

        • Click on the three dots in the upper-right corner of the test.
        • Select Show Info from the dropdown menu.

        2. Retrieve the Necessary IDs

        • Locate the Experiment ID and Variant ID for the test.

        3. Use These Values in Recart

        • Ensure the Variant ID matches the value expected by Recart.
        • If needed, adjust the default visibility setting based on your preference.

        Where to Configure A/B Testing in Recart?

        To set up A/B testing for your popups in Recart:

        1. Log in to the Recart App.
        2. Navigate to Opt-in Tools and select the popup you want to configure.
        3. Under Settings, go to A/B Testing Configuration section.
        4. Enter the JavaScript value, Variant ID, and Experiment ID (if applicable).
        5. Set the Default Visibility option.
        6. Click Save & Apply.

        Once configured, Recart will automatically determine which popup variant to display based on the A/B testing setup.

         

        In case you need further assistance, feel free to contact our support team, we are happy to help. 😊