Popup vs no popup. Measure what it adds.

Every popup tool reports the revenue its signups bring in. None of that tells you whether those shoppers would have bought anyway, or whether the popup put some of them off. The only way to know is to hide the popup from a random share of visitors and compare orders. This guide covers what published holdout tests found, how to set one up, how many visitors it takes, and how to read a result that says no difference.

By the numbers

  • +10.4%order conversion rate for Simple Modern's popup group against a 50/50 no-popup holdout over 30 days, at 99% confidenceRecart Conversion Lab, Simple Modern popup vs no popup holdout
  • +7%revenue per visitor for True Classic's best discount popup against no popup; Juvenon saw +2% and Perfect White Tee was flatRecart Conversion Lab, no popup vs discount popup incrementality test
  • ~86%of Wisepops holdout tests that reached a clear verdict, where the campaign beat showing nothingWisepops, incrementality report 2026
  • 17%more revenue from visitors who saw a welcome popup than from a control group, for a popup credited with over $200,000Wisepops, the revenue attribution problem

These are holdout tests published by popup vendors, about their own customers. Treat them as reasons to test your own store, not as a forecast.

What a popup vs no popup test is

Visitors are split at random. One group sees your popup as usual; the other sees no popup at all, so nobody in it can sign up through it. After enough time, you compare orders per visitor in each group. Because the split is random, the difference is what the popup caused, including any sales it cost. Most A/B tests compare two versions of a popup on signups. This one compares a popup with nothing, so it has to be decided on orders or revenue: the no-popup group has no signups to count.

Attributed revenue is not caused revenue

Popup dashboards credit the popup with orders placed by subscribers within an attribution window. Some of those shoppers were buying anyway. Wisepops describes a store whose welcome popup was credited with over $200,000 in about six months; a 50% holdout showed the visitors who saw it generated 17% more revenue than those who didn't, which Wisepops calls "nowhere near the original $200,000". At a luxury fashion brand, a 14-day window credited the popup with 12% of monthly revenue; at 3 days it fell to around 5%. Wisepops recommends windows of 3-5 days for most ecommerce popups. The window changes the headline; only a holdout measures the effect.

What published holdout tests found

Recart ran a 50/50 holdout at Simple Modern for 30 days across hundreds of thousands of sessions. Order conversion rate was +10.4% for the popup group (desktop +14.2%, mobile +9.2%) at 99% confidence, and revenue per visitor was +29.5% on desktop and +8.9% on mobile. In a separate Recart write-up, True Classic's best discount popup lifted revenue per visitor +7% and gross profit per visitor +2% against no popup; Juvenon's 10% off popup gave +2%, and Perfect White Tee's was flat (+0%). Only True Classic's result is described as having "full statistical power"; the page gives no confidence levels for the other two.

The largest set of holdout tests

Wisepops counted 247 experiments since January 2024 that compared a campaign with a no-popup control group, of which 204 concluded. When a holdout test reached a clear verdict, the campaign beat showing nothing about 86% of the time, with a median gain of roughly +20% on the campaign's own goal. Conversion gains were more modest, about +1% to +8% across the middle range. Wisepops calls the median "still a small sample, so treat it as directional", and its chart labels the win rate as about 21 decided tests. The report gives no confidence levels.

Do popups hurt conversion?

In the published tests, rarely. Recart says that across dozens of tests, revenue per visitor "was either neutral or actually leaning towards the popup version", and in its lab write-up, Perfect White Tee's flat revenue per visitor came with over 13,000 new subscribers. But every figure here comes from a popup vendor writing about its own customers, and tests that showed a loss are less likely to be published. A full-screen popup on arrival, a discount given to shoppers who were already buying, or a popup that blocks the cart can all cost orders. The honest answer for your store is a test.

Choose the metric

Decide on orders per assigned visitor: every visitor in each group counts, whether or not the popup ever opened for them. This intent-to-treat comparison keeps the groups random; counting only visitors who saw the popup would compare engaged shoppers with everyone. Revenue per visitor captures discount cost and order size but swings with a few large orders, so it needs far more traffic. Count orders for a fixed window after each visitor is assigned, so late buyers in either group are treated the same.

How many visitors you need

Popups usually move orders by a few percent, and small differences need large samples. These are estimates from the standard two-proportion formula at 95% confidence (two-sided) and 80% power: n per group = (1.96 + 0.84)² × [p1(1 − p1) + p2(1 − p2)] / (p2 − p1)². At a 4% order rate, detecting a 10% relative lift (4.0% to 4.4%) needs about 39,472 visitors per group, a 20% lift about 10,313 and a 5% lift about 154,301. At a 2% order rate, a 10% lift needs about 80,679 per group. The smaller group sets the pace: with 10% of visitors in the no-popup group, putting 39,472 in it takes 394,720 visitors in total.

How long to run it and how to split

Run whole weeks, so weekdays and weekends are balanced, and do not stop the moment the numbers look good. Wisepops suggests planning "beyond two weeks for lower-traffic goals", and Recart says revenue tests "usually need at least $1M in revenue to gain statistical power". A 50/50 split answers fastest. A smaller no-popup share, such as 10%, risks fewer lost signups while the test runs but takes far longer; Wisepops says 5% to 10% is enough only on high-traffic pages. Leave everything else alone during the test: a new offer or a sale halfway through muddies the result.

When the answer is no difference

A test that ends with no reliable difference has not proved the popup does nothing. It means the effect, if any, is smaller than your traffic can detect. That is common, since popups often change orders by only a few percent. It still tells you something useful: the popup is not costing a measurable share of orders, and it keeps growing your list, whose later purchases fall outside a same-session or short-window test. Keep the popup if those subscribers are worth having, or run longer for a tighter answer.

How other popup tools handle it

Klaviyo's sign-up form A/B tests have no no-popup arm: in a 2023 community thread, its community manager said a holdout "isn't a native feature", and its optimization tests compare timing settings against the current form, for accounts with at least 400,000 profiles. Alia's help center has merchants duplicate a popup, remove its triggers and A/B test the two, judging by site conversion rate and bounce rate. Wisepops offers an optional control group that sees no variant, conclusive at 95% confidence. Recart documents an invisible popup variant on a 50/50 split, or a third-party testing tool.

Run it in Veya

On a live form's Optimize tab, click New test and choose Popup vs no popup. Choose order rate or revenue per visitor, and the share of visitors who see no popup: 50%, 30%, 20% or 10%. Visitors in that group see none of the test's forms while your other popups keep showing; each visitor keeps their group. Orders count for every assigned visitor within 30 days of assignment by default. Veya gives a verdict only after 7 days, 1,000 assigned visitors per group and 30 buyers across the two, then lets you keep the popup or keep no popup, which pauses the form. The help doc walks through each step.

Questions

Do popups hurt conversion rates?

Published holdout tests mostly found popups neutral or positive for orders: Recart measured +10.4% order conversion at Simple Modern at 99% confidence, and flat revenue per visitor at Perfect White Tee. These are vendor-published results, so test your own store by hiding the popup from a random share of visitors and comparing orders per visitor.

What is a popup holdout test?

A test where a random share of visitors never sees a popup, while everyone else sees it as usual. Comparing orders or revenue per visitor between the two groups shows what the popup caused, which attributed revenue cannot, because attributed revenue includes shoppers who would have bought anyway.

How long should a popup vs no popup test run?

At least a week, in whole weeks, and until each group has enough visitors to detect the difference you care about. At a 4% order rate, a 10% lift needs about 39,472 visitors per group by the standard two-proportion estimate. Wisepops suggests planning beyond two weeks for lower-traffic goals.

What share of visitors should see no popup?

A 50/50 split reaches an answer fastest. A smaller holdout, such as 10%, gives up fewer signups while the test runs, but the smaller group sets the pace, so the test takes far longer. Wisepops says 5% to 10% is enough only on high-traffic pages.

Should I judge the test on signups?

No. Visitors who see no popup cannot sign up through it, so signups always favor the popup. Judge it on orders per assigned visitor, or on revenue per visitor if you have far more traffic.

Sources

Figures checked November 6, 2026.

Your next popup?
We’ll build it for you.

14-day free trial on every plan. Custom design and revisions included.