> For the complete documentation index, see [llms.txt](https://help.recart.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.recart.com/list-growth/how-to-run-a-popup-holdout-test.md). # Popup Holdout Testing Learn how to set up a popup holdout test in Recart to measure the true revenue impact of your popups compared to showing no popup at all. * [What Is a Popup Holdout Test?](#what-is-a-popup-holdout-test) * [How It Works](#how-it-works) * [Step-by-Step Setup](#step-by-step-setup) * [How Long Should the Test Run?](#how-long-should-the-test-run) * [Interpreting Results](#interpreting-results) * [Checklist](#checklist) ### What Is a Popup Holdout Test? A popup holdout test is a type of A/B test where one variant — the "holdout" — is configured so that it never actually displays to visitors. Traffic is split between your live popup and the invisible holdout, letting you measure the true incremental impact of running a popup versus showing nothing at all. This answers a key question: **Is this popup actually driving revenue, or would visitors have converted anyway?** ### How It Works When a holdout test is active, Recart splits traffic between the live popup and the holdout variant just like any standard A/B test. The difference is that the holdout popup is hidden on the backend — visitors assigned to it never see anything. Revenue attribution works the same for both variants. Recart links each visitor's session to their cart and orders using Shopify data, so revenue can be attributed regardless of whether a popup was shown or whether the visitor opted in to SMS or email. This means you can directly compare revenue per visitor and conversion rate between visitors who saw the popup and those who saw nothing. **Important:** Holdout tests require a backend configuration by Recart before launch. Please make sure you've reached out to your dedicated CSM or the Recart Support team to get this set up before proceeding. ### Step-by-Step Setup #### 1. Choose the popup to test Decide which popup you want to measure the impact of. For best results, use a freshly created popup rather than one that's already been running — active popups accumulate ongoing revenue attribution, which can skew results when compared against a new variant. #### 2. Duplicate the popup 1. In the Recart dashboard, go to **Opt-in Tools → Popups** 2. Find the popup you want to test and duplicate it 3. This duplicate will be your holdout variant — keep it in **Draft** status and do not activate it #### 3. Reach out to your CSM or the Recart Support team Contact your dedicated CSM or reach out to the Recart Support team to request the holdout configuration. Include the following: * Your **Site ID** * The **link or ID of the popup** you want to use for the holdout Once we've completed the backend setup, we'll get back to you to confirm — and we'll launch the test on your behalf. #### 4. Wait for confirmation Do not launch the A/B test until you've received confirmation from the Recart team that the holdout has been configured. ### How Long Should the Test Run? Revenue-focused tests need more time than opt-in rate tests to produce reliable results. **Run the test for a minimum of 2–4 weeks.** Revenue data has higher day-to-day variance, so tests that run for only a few days will produce unreliable results. Wait for statistical significance before drawing conclusions. ### Interpreting Results The core comparison is **revenue per visitor** and **conversion rate** between the two groups. If the live popup group outperforms the holdout, the popup is driving incremental value. Note that the holdout variant will show impression data (used for attribution) but no opt-ins, since no popup is ever displayed. #### Discount effectiveness A common concern is that visitors who opt in would have purchased anyway — meaning the popup is offering discounts unnecessarily rather than driving incremental revenue. This is exactly what a holdout test is designed to answer. By comparing the conversion rate and revenue per visitor of the popup group against the holdout (no popup) group, you can see whether the popup is genuinely influencing purchases or simply discounting visitors who were already going to convert. If the two groups perform similarly, that's a signal to reassess the discount strategy. If the popup group outperforms, the popup is earning its keep. ### Checklist * Choose the popup to test * Reach out to your CSM or the Recart Support team with the specific popup you’d like to test * Wait for confirmation from the Recart team * Run the test for at least 2–4 weeks * Compare revenue per visitor and conversion rate between variants #### #### Need Help? If you have any questions or need support setting up a holdout A/B test, feel free to reach out to us at [**support@recart.com**](mailto:support@recart.com) — we're happy to help. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://help.recart.com/list-growth/how-to-run-a-popup-holdout-test.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.