By the numbers
- 26.1%false positive rate when a test stops at the first significant result, against the intended 5%Evan Miller, how not to run an A/B test
- 7 daysminimum test length, one full business cycle, recommended by OptimizelyOptimizely, how long to run an experiment
- 1,000displays per version Popupsmart advises as a floor for a popup test, more for small gapsPopupsmart, 28,674 campaigns
- 1.6 millionusers needed in Kohavi's worked example to detect a 5% revenue change, against under 500,000 for conversionKohavi, Henne and Sommerfield, KDD 2007
Vendor figures are averages across many stores, not controlled tests. Use them to decide what to test first.
What a popup A/B test can tell you
A test answers one question: does version B do better than version A for visitors like yours? Visitors are split at random, so the two groups differ only by chance and by the change you made. That is why a test should change one thing at a time, such as the headline, the button, the size or the trigger. Change five things and a win tells you nothing about which one worked. Keep the offer the same too, unless the offer is what you are testing, because changing the discount changes what you give away, not only how you ask.
Choose one metric before you start
Decide in advance what counts as winning. Kohavi, Henne and Sommerfield's guide to online experiments warns that picking the metric afterwards raises the risk of finding results that look significant by chance. For a popup, signup rate per view is the usual choice, and it needs the least traffic. A popup that brings in more signups can still bring in fewer real customers, so stores with enough traffic can decide by net-new subscribers, orders or revenue instead. Veya lets you pick one of those four when you start a test, and that metric alone decides it.
How many visitors you need
Sample size depends on your current rate and the smallest lift you care about. Evan Miller's rule of thumb is n = 16 × p(1 - p) / δ² per version, where p is the current rate and δ the absolute change. Wisepops puts the average signup popup at 4.82% in 2026. Plugging that in, detecting a 10% relative lift, from 4.82% to about 5.30%, takes more than 31,500 views per version by that formula; a 20% lift takes nearly 7,900. Halving the lift you want to detect quadruples the visitors needed, so small stores should test bold changes.
Revenue needs far more traffic
Most visitors spend nothing and a few spend a lot, so revenue is noisy. In Kohavi's worked example, a store where 5% of visitors buy needs over 1.6 million users to detect a 5% change in revenue, but under 500,000 to detect a 5% change in conversion, and 30,400 for a 20% change in conversion. Veya reflects this. A revenue per visitor test needs 5,000 assigned visitors and 30 orders in each version before any verdict, about ten times a signup test, and caps each buyer's revenue at the 99th percentile so one huge order cannot decide it.
Why peeking gives false winners
A significance test at 95% is wrong about one time in twenty only if you look once, at the planned end. Look every day and stop at the first significant result, and the odds of crowning a version that is no better climb fast. Evan Miller's simulation of a change that does nothing found 26.1% of such tests declared a winner, more than five times the intended 5%. His advice is to decide the sample size in advance and wait. If you must look several times, each look needs a stricter bar: with 10 looks, about 1.0% per look keeps the overall rate near 5%.
How long to run a popup test
Run at least one full week, even with plenty of traffic. Optimizely recommends a minimum of one business cycle, 7 days, to capture every kind of visitor behavior, and Kohavi's guide recommends a week or two, then continuing in whole weeks, because weekend visitors often behave differently. So the answer is the time needed to reach your planned sample size, rounded up to whole weeks, and never less than one. Veya gives no verdict before 7 days, whatever the counts say, and its autopilot checks results only at the end of each full week.
Even splits finish sooner
Sending most visitors to the current form feels safer, but it slows the test. Kohavi's guide shows a test that needs 5 days at 50/50 needing 25 days at 95/5, and over 125 days at 99/1, which it calls too long for a reliable result. Splitting traffic unevenly protects you from a bad challenger at the cost of time. In Veya's autopilot you choose the share of visitors who see the challenger, from 10% to 50%, with 50% as the default. A lower share is reasonable when the challenger is a bigger risk, if you have the traffic to wait.
How Veya splits traffic
Veya assigns each visitor in the browser by hashing their visitor ID with the test's ID and mapping the result onto the versions' weights, so no server call is needed and the split is stable. With preferences consent, the same visitor sees the same version across visits; without it, for the session. Veya picks the version before checking whether the visitor qualifies to see it, because falling back to another version would bias the test. Versions in one test share a frequency cap, so a new version is not treated as a new popup. A manual test compares 2 to 4 live forms.
How Veya calls a winner
Results show as collecting until the test has run 7 days and each version has its minimum sample: 500 views for signup rate; 1,000 assigned visitors and 30 net-new subscribers or buyers across the pair for those metrics; 5,000 visitors and 30 orders each for revenue. Rates are compared with a two-proportion z-test, revenue with Welch's t-test. A challenger wins when its difference is significant at 5%, divided by the number of challengers, so testing three versions does not triple the false winners. Outcome metrics count every assigned visitor, whether or not they saw the popup.
What the Veya autopilot does
The autopilot keeps one live form and tests one challenger at a time, each changing one thing: a benefit-led headline, a first-person button, a bolder offer, a different size, timing, exit intent, a scroll trigger or smart re-trigger, or your own idea. You approve each challenger's design before it runs, and it keeps the live form's offer. Results are checked once a week, and the 5% bar is split across the planned looks: with the default 28 day limit, a win must hold at p < 0.0125. In a signup test, a challenger with under a 1% chance of beating the live form is switched off early, after 3 days and 300 views each. Winners are never called early.
Brief Veya
Open the form's Optimize tab. For one test, click New test, describe what version B should change, and choose Decide the winner by. You can also launch a new form with a test from day one by filling in the A/B test field when you request it. For ongoing testing, switch on Autopilot, set the challenger's share of visitors and the longest test, from 14 to 56 days, and choose whether a significant winner goes live without asking you. An autopilot test that reaches its limit without enough data ends with your current form kept.
Questions
How long should I run a popup A/B test?
At least one full week, and in whole weeks after that, so every day of the week is counted equally. Then keep going until each version reaches the sample size you planned before starting. Do not stop early because one version looks ahead: early leads often fade, and stopping at the first significant result produces many false winners.
How many visitors do I need to A/B test a signup form?
It depends on your signup rate and the lift you want to detect. At the 4.82% average signup rate Wisepops reports, Evan Miller's rule of thumb puts a 20% relative lift at nearly 7,900 views per version, and a 10% lift at more than 31,500. Popupsmart suggests at least 1,000 displays per version as a floor. Revenue tests need far more.
Can I stop an A/B test as soon as it reaches 95% significance?
Not if you have been checking along the way. Each check is another chance for random noise to cross the line, and Evan Miller's simulation found that stopping at the first significant result flagged a winner 26.1% of the time when there was no real difference. Either fix the sample size in advance, or use a method that sets a stricter bar for each look.
What should I A/B test on a popup first?
Start with changes big enough to show up in your traffic: the headline and offer framing, the trigger and its timing, or the popup's size. Small effects need large samples to detect, so on modest traffic the bold changes are the ones a test can actually measure. Keep the discount the same between versions unless you mean to test it, so the result is about the design.
Should I decide a popup test by signups or by revenue?
Signup rate per view needs the least traffic and suits most stores. It can reward a version that attracts people who never buy, so if each version can reach 1,000 assigned visitors and 30 new subscribers or buyers, net-new subscribers or order rate are a better check. Revenue per visitor is the most direct measure but the noisiest: in Veya it needs 5,000 visitors and 30 orders per version before a verdict.
Sources
- Evan Miller, how not to run an A/B test
- Kohavi, Henne and Sommerfield, practical guide to controlled experiments on the web (KDD 2007)
- Optimizely, how long to run an experiment
- Wisepops, 20+ popup statistics 2026 (email signup popups, about 1 billion displays)
- Popupsmart, do exit-intent popups work? (28,674 campaigns, January 2024 to September 2026)
Figures checked November 6, 2026.