How to Run a Popup Holdout Test
Learn how to set up a popup holdout test in Recart to measure the true revenue impact of your popups compared to showing no popup at all.
What Is a Popup Holdout Test?
A popup holdout test is a type of A/B test where one variant — the "holdout" — is configured so that it never actually displays to visitors. Traffic is split between your live popup and the invisible holdout, letting you measure the true incremental impact of running a popup versus showing nothing at all.
This answers a key question: Is this popup actually driving revenue, or would visitors have converted anyway?
How It Works
When a holdout test is active, Recart splits traffic between the live popup and the holdout variant just like any standard A/B test. The difference is that the holdout popup is hidden on the backend — visitors assigned to it never see anything.
Revenue attribution works the same for both variants. Recart links each visitor's session to their cart and orders using Shopify data, so revenue can be attributed regardless of whether a popup was shown or whether the visitor opted in to SMS or email.
This means you can directly compare revenue per visitor and conversion rate between visitors who saw the popup and those who saw nothing.
Important: Holdout tests require a backend configuration by Recart before launch. Please make sure you've reached out to your dedicated CSM or the Recart Support team to get this set up before proceeding.
Step-by-Step Setup
1. Choose the popup to test
Decide which popup you want to measure the impact of. For best results, use a freshly created popup rather than one that's already been running — active popups accumulate ongoing revenue attribution, which can skew results when compared against a new variant.
2. Duplicate the popup
- In the Recart dashboard, go to Opt-in Tools → Popups
- Find the popup you want to test and duplicate it
- This duplicate will be your holdout variant — keep it in Draft status and do not activate it
3. Reach out to your CSM or the Recart Support team
Contact your dedicated CSM or reach out to the Recart Support team to request the holdout configuration. Include the following:
- Your Site ID
- The link or ID of the popup you want to use for the holdout
Once we've completed the backend setup, we'll get back to you to confirm — and we'll launch the test on your behalf.
4. Wait for confirmation
Do not launch the A/B test until you've received confirmation from the Recart team that the holdout has been configured.
How Long Should the Test Run?
Revenue-focused tests need more time than opt-in rate tests to produce reliable results. Run the test for a minimum of 2–4 weeks. Revenue data has higher day-to-day variance, so tests that run for only a few days will produce unreliable results. Wait for statistical significance before drawing conclusions.
Interpreting Results
The core comparison is revenue per visitor and conversion rate between the two groups. If the live popup group outperforms the holdout, the popup is driving incremental value.
Note that the holdout variant will show impression data (used for attribution) but no opt-ins, since no popup is ever displayed.
Discount effectiveness
A common concern is that visitors who opt in would have purchased anyway — meaning the popup is offering discounts unnecessarily rather than driving incremental revenue. This is exactly what a holdout test is designed to answer.
By comparing the conversion rate and revenue per visitor of the popup group against the holdout (no popup) group, you can see whether the popup is genuinely influencing purchases or simply discounting visitors who were already going to convert. If the two groups perform similarly, that's a signal to reassess the discount strategy.
If the popup group outperforms, the popup is earning its keep.
Checklist
- Choose the popup to test
- Reach out to your CSM or the Recart Support team with the specific popup you’d like to test
- Wait for confirmation from the Recart team
- Run the test for at least 2–4 weeks
- Compare revenue per visitor and conversion rate between variants
Need Help?
If you have any questions or need support setting up a holdout A/B test, feel free to reach out to us at support@recart.com — we're happy to help.