Popup A/B Testing

Explains how to create a popup A/B test with the A/B test wizard, choose what counts as a win, and read the results.

Test two or more popups against each other to see which one brings in more opt-ins, orders or revenue. You set up the whole test in one place — the variants, the traffic split, what counts as a win and when it runs — from the Popup A/B Tests page. Before you start, you'll need at least two popups to compare; they can be active, inactive or drafts.

What you can test

You can test elements like:

• Entry timing
• Layout
• Incentives
• Consent method
• Opt-in flow

Each additional variant splits your traffic further, which can affect how quickly you reach statistically significant results.

Example: one change per test

Change one thing per test. If you change the delay and the offer at the same time, you can't tell which change caused the difference. Each row below is a separate test:

What you're testing	Current popup	New popup variant
Entry timing	10-second delay	4-second delay
Layout	Lightbox	Fullscreen
Opt-in flow	One-step opt-in	Two-step opt-in
Consent method	Reply in SMS	In-popup confirmation
Incentive	10% offer	20% offer

Create an A/B test

1. Go to Opt-in Tools, then the Popup A/B Tests tab
2. Click Create new to open the A/B test wizard
3. Enter a Name for the test and add any Tags

   
4. Under Test participants, click + Add variant and select whether you want to test Mobile or Desktop

   
5. Select the popups you want to compare
6. Choose the Traffic split — default is evenly split between variants — click on manual if you'd like to update this

   
7. Choose the Primary metric — what counts as a win

   
8. Set the Start date and End date

   
9. Check the Pages and Geo targeting for the test

   
10. Add any Notes

    
11. Click Create — or Save as draft to finish later

If your account had a test running or scheduled when the wizard launched, it moves over to the wizard once that test ends. If you don't see Create new on the Popup A/B Tests page yet, contact the Recart Support team.

Additional notes per step

Test participants

• Click + Add variant to choose from your active, inactive and draft popups. You can filter by channel and status, or search by name
• When you add the first variant, you'll choose whether the test is for desktop or mobile. Run separate tests for each device
• The first popup you add becomes the Control. To change it, open the … menu on another variant and select Set as control
• A test needs at least 2 participants and can include up to 8 popups
• A popup that's already part of another in-progress, scheduled or draft test can't be added
• To test a brand-new design, create it as a draft popup in the popup editor first, then add it here

Draft popups can join a test and are activated automatically when the test starts. If a draft is missing something it needs — for example a welcome flow message — you'll see a warning on its row, and you'll need to fix it before you can schedule the test. Once a draft popup has been activated by a test, it can be set to inactive later but can't go back to draft.

Traffic split

• Auto splits traffic evenly across all variants and updates as you add or remove them
• Manual lets you set a percentage for each variant. The percentages must add up to exactly 100%

Primary metric

Choose the one metric that decides the winner: SMS opt-in rate, Email opt-in rate, Order conversion rate, Revenue per visitor, Average order value, Total revenue or Total orders. It's marked as the Primary metric on the results page. Every other metric is still tracked and shown.

Choosing a primary metric that fits your test

• Using a manual split? Avoid Total revenue and Total orders — a variant with more traffic will win on totals regardless of quality. Use a rate instead, like Revenue per visitor or Order conversion rate
• SMS-only or email-only variants? Don't pick the opt-in rate for the channel they don't collect — it will always be 0
• Variants collect different channels? Pick a metric every variant can produce, or a revenue metric, which works for any channel mix

Pages and Geo targeting

The test has its own Pages and Geo targeting, which applies to every variant while the test runs.

• By default, it's copied from the control popup. You can also import the targeting from any other variant
• The country list only shows countries that every variant can target
• While the test is running, each participating popup's own Pages and Geotargeting settings are locked, with a note that they're being overridden by the test
• When the test ends, each popup goes back to its own targeting automatically

Device targeting, Visitor Targeting and trigger settings (delay, exit intent, frequency limits) stay with each popup.

Notes and tags

• Tags are required to schedule a test. Choose from tags like Copy, CTA, Discount test, Layout test, Teaser test or BFCM, so you can filter your tests later — for example, to find every CTA test you ran before BFCM
• Notes are optional. Use them to record your hypothesis or why you ran the test

Save as draft

Click Save as draft if you're not ready to schedule — for example, if one of your variant popups isn't finished. A draft test doesn't activate any popups or change any targeting. Open it from the Popup A/B Tests list to keep editing.

Test statuses

A test moves from Draft to Scheduled to In Progress to Ended. A scheduled test can also be Canceled.

Status	What you can still change	What opens when you click it
Draft	Everything	The wizard
Scheduled	Name, notes, tags, start date, end date	The wizard
In Progress	Name, notes, tags, end date (extend, or shorten to a future date)	The results page
Ended	Name, notes	The results page
Canceled	Name, notes	The results page

Variants, traffic split, primary metric and targeting are locked once a test is scheduled, so the results stay reliable. To change them, end the test and create a new one.

• Cancel a scheduled test to release its popups. Nothing ran, so each popup goes back to its own targeting and keeps its own active or inactive status
• End an in-progress test to stop it. The confirmation shows what will happen to each popup, and you can change any popup by hand afterwards

Viewing your results

Click any in-progress, ended or canceled test on the Popup A/B Tests page to open its results. You'll see each variant side by side against the control, with your primary metric marked, across:

• Impressions
• Opt-ins
• Opt-in rates
• Bounce rate — lower is better, so a variant that beats the control shows in green
• Orders
• Revenue per visitor
• Average order value
• Conversion rate

The results page also includes graphs of opt-in rates, conversions and revenue over time, and an opt-in funnel for each variant.

The Popup A/B Tests list shows all your tests, with filters for status and tags and a search by name. Tests are sorted in-progress first, then scheduled, draft, ended and canceled.

Configuring 3rd-Party A/B Testing

If you're using an external A/B testing tool like Intelligems or Convert, you can connect it to a popup so your testing tool decides which popup each visitor sees. This is set up in the popup's own settings, not in the A/B test wizard, and these tests don't appear on the Popup A/B Tests page.

Where to find the settings

1. Go to Opt-in tools and open the popup you want to use in your test
2. Click Edit, then click Settings in the left menu
3. Scroll down to Third-party A/B testing and switch it to Active
4. Fill in the fields described below
5. Click Save at the top right

What to fill in

The numbers match the screenshot below.

1. Javascript value to access A/B testing is the place on your website where your testing tool keeps the group each visitor is in. See Values for common tools for the value to use
2. Variant ID is the ID your testing tool gives to the group this popup belongs to
3. Experiment ID (optional) is the ID of the whole test. Some tools need it and some don't, as shown in the table below

Default visibility is what Recart does if it can't work out which group a visitor is in. Choose Hide popup or Show popup.

The Third-party A/B testing panel in the popup settings, with numbers marking the Javascript value, Variant ID and Experiment ID fields

How Recart uses these values

• Your testing tool stores the visitor's group: Recart compares it with the Variant ID. If they match, the popup shows
• Your testing tool has to be asked: Recart asks using your Experiment ID, then compares the answer with the Variant ID. If they match, the popup shows
• Nothing comes back: Recart uses your Default visibility setting

The Variant ID has to match exactly what your testing tool returns for that group.

Values for common tools

Tool	Javascript value	Experiment ID field
Intelligems	window.igData.getTestGroup	Required. Fill it in
Convert	window.convert.currentData.experiences.<your-experiment-id>	Leave it empty

Using a different tool, such as VWO? Contact the Recart Support team and share your tool's documentation. We'll confirm the right value for you.

Example: Intelligems

1. In Intelligems, open your A/B test and click the three dots in the upper-right corner

   

   The three dots menu in the upper-right corner of an Intelligems test
2. Click Show Info

   

   The Show Info option in the Intelligems menu
3. Copy the Experiment ID. Then copy the ID of the group this popup belongs to (for example New Group 1 ID). Use the long ID, not the short For GA one. Each popup in your test gets the ID of its own group

   

   The Intelligems information window showing the Experiment ID and the group IDs
4. In Recart, open the popup's Third-party A/B testing settings and switch it to Active
5. Enter window.igData.getTestGroup as the Javascript value, the group ID as the Variant ID, and the Intelligems Experiment ID as the Experiment ID. The Experiment ID is required for Intelligems
6. Choose your Default visibility, then click Save

Example: Convert

1. In Convert, open your project, go to the Experiences tab and open your experience. The number next to A/B - ID is your experiment ID

   

   The experiment ID shown at the top of a Convert experience
2. Click the three-dot menu next to the variation you want and select Preview

   

   The three-dot menu next to a variation in Convert
3. In the window that opens, the variant ID is the last number in the preview link, after convert_v=

   

   The Convert preview link with the variant ID highlighted at the end
4. In Recart, open the popup's Third-party A/B testing settings and switch it to Active
5. Add your experiment ID to the end of the Javascript value, so it reads window.convert.currentData.experiences. followed by your experiment ID. Enter the variant ID as the Variant ID and leave the Experiment ID field empty. Convert needs the experiment ID as part of the Javascript value

   

   The Third-party A/B testing panel set up for Convert, with the Experiment ID field left empty
6. Choose your Default visibility, then click Save

Troubleshooting

Getting an error when downloading your A/B test results?

If you see an error page when trying to export your A/B test results as an XLSX file, check whether your test name contains any special characters (such as +, &, or %). These can cause the export to fail.

To fix this, rename your test to remove the special character — for example, change "Summer + Fall" to "Summer and Fall" — then try downloading the export again.

If you have any questions, do not hesitate to contact the Recart Support team. We'd be happy to help!