Test one change. Wait for the answer.

An A/B test shows two versions of a popup to two random groups of visitors and compares the results. It is the only reliable way to know whether a new headline, trigger or layout helps. It is also easy to fool yourself with one. This guide covers what to measure, how many visitors you need, how long to run, why checking early misleads, and how Veya runs tests and its autopilot.

By the numbers

Vendor figures are averages across many stores, not controlled tests. Use them to decide what to test first.

What a popup A/B test can tell you

A test answers one question: does version B do better than version A for visitors like yours? Visitors are split at random, so the two groups differ only by chance and by the change you made. That is why a test should change one thing at a time, such as the headline, the button, the size or the trigger. Change five things and a win tells you nothing about which one worked. Keep the offer the same too, unless the offer is what you are testing, because changing the discount changes what you give away, not only how you ask.

Choose one metric before you start

Decide in advance what counts as winning. Kohavi, Henne and Sommerfield's guide to online experiments warns that picking the metric afterwards raises the risk of finding results that look significant by chance. For a popup, signup rate per view is the usual choice, and it needs the least traffic. A popup that brings in more signups can still bring in fewer real customers, so stores with enough traffic can decide by net-new subscribers, orders or revenue instead. Veya lets you pick one of those four when you start a test, and that metric alone decides it.

How many visitors you need

Sample size depends on your current rate and the smallest lift you care about. Evan Miller's rule of thumb is n = 16 × p(1 - p) / δ² per version, where p is the current rate and δ the absolute change. Wisepops puts the average signup popup at 4.82% in 2026. Plugging that in, detecting a 10% relative lift, from 4.82% to about 5.30%, takes more than 31,500 views per version by that formula; a 20% lift takes nearly 7,900. Halving the lift you want to detect quadruples the visitors needed, so small stores should test bold changes.

Revenue needs far more traffic

Most visitors spend nothing and a few spend a lot, so revenue is noisy. In Kohavi's worked example, a store where 5% of visitors buy needs over 1.6 million users to detect a 5% change in revenue, but under 500,000 to detect a 5% change in conversion, and 30,400 for a 20% change in conversion. Veya reflects this. A revenue per visitor test needs 5,000 assigned visitors and 30 orders in each version before any verdict, about ten times a signup test, and caps each buyer's revenue at the 99th percentile so one huge order cannot decide it.

Why peeking gives false winners

A significance test at 95% is wrong about one time in twenty only if you look once, at the planned end. Look every day and stop at the first significant result, and the odds of crowning a version that is no better climb fast. Evan Miller's simulation of a change that does nothing found 26.1% of such tests declared a winner, more than five times the intended 5%. His advice is to decide the sample size in advance and wait. If you must look several times, each look needs a stricter bar: with 10 looks, about 1.0% per look keeps the overall rate near 5%.

How long to run a popup test

Run at least one full week, even with plenty of traffic. Optimizely recommends a minimum of one business cycle, 7 days, to capture every kind of visitor behavior, and Kohavi's guide recommends a week or two, then continuing in whole weeks, because weekend visitors often behave differently. So the answer is the time needed to reach your planned sample size, rounded up to whole weeks, and never less than one. Veya gives no verdict before 7 days, whatever the counts say, and its autopilot checks results only at the end of each full week.

Even splits finish sooner

Sending most visitors to the current form feels safer, but it slows the test. Kohavi's guide shows a test that needs 5 days at 50/50 needing 25 days at 95/5, and over 125 days at 99/1, which it calls too long for a reliable result. Splitting traffic unevenly protects you from a bad challenger at the cost of time. In Veya's autopilot you choose the share of visitors who see the challenger, from 10% to 50%, with 50% as the default. A lower share is reasonable when the challenger is a bigger risk, if you have the traffic to wait.

How Veya splits traffic

Veya assigns each visitor in the browser by hashing their visitor ID with the test's ID and mapping the result onto the versions' weights, so no server call is needed and the split is stable. With preferences consent, the same visitor sees the same version across visits; without it, for the session. Veya picks the version before checking whether the visitor qualifies to see it, because falling back to another version would bias the test. Versions in one test share a frequency cap, so a new version is not treated as a new popup. A manual test compares 2 to 4 live forms.

How Veya calls a winner

Results show as collecting until the test has run 7 days and each version has its minimum sample: 500 views for signup rate; 1,000 assigned visitors and 30 net-new subscribers or buyers across the pair for those metrics; 5,000 visitors and 30 orders each for revenue. Rates are compared with a two-proportion z-test, revenue with Welch's t-test. A challenger wins when its difference is significant at 5%, divided by the number of challengers, so testing three versions does not triple the false winners. Outcome metrics count every assigned visitor, whether or not they saw the popup.

What the Veya autopilot does

The autopilot keeps one live form and tests one challenger at a time, each changing one thing: a benefit-led headline, a first-person button, a bolder offer, a different size, timing, exit intent, a scroll trigger or smart re-trigger, or your own idea. You approve each challenger's design before it runs, and it keeps the live form's offer. Results are checked once a week, and the 5% bar is split across the planned looks: with the default 28 day limit, a win must hold at p < 0.0125. In a signup test, a challenger with under a 1% chance of beating the live form is switched off early, after 3 days and 300 views each. Winners are never called early.

Brief Veya

Open the form's Optimize tab. For one test, click New test, describe what version B should change, and choose Decide the winner by. You can also launch a new form with a test from day one by filling in the A/B test field when you request it. For ongoing testing, switch on Autopilot, set the challenger's share of visitors and the longest test, from 14 to 56 days, and choose whether a significant winner goes live without asking you. An autopilot test that reaches its limit without enough data ends with your current form kept.

Questions

How long should I run a popup A/B test?

At least one full week, and in whole weeks after that, so every day of the week is counted equally. Then keep going until each version reaches the sample size you planned before starting. Do not stop early because one version looks ahead: early leads often fade, and stopping at the first significant result produces many false winners.

How many visitors do I need to A/B test a signup form?

It depends on your signup rate and the lift you want to detect. At the 4.82% average signup rate Wisepops reports, Evan Miller's rule of thumb puts a 20% relative lift at nearly 7,900 views per version, and a 10% lift at more than 31,500. Popupsmart suggests at least 1,000 displays per version as a floor. Revenue tests need far more.

Can I stop an A/B test as soon as it reaches 95% significance?

Not if you have been checking along the way. Each check is another chance for random noise to cross the line, and Evan Miller's simulation found that stopping at the first significant result flagged a winner 26.1% of the time when there was no real difference. Either fix the sample size in advance, or use a method that sets a stricter bar for each look.

What should I A/B test on a popup first?

Start with changes big enough to show up in your traffic: the headline and offer framing, the trigger and its timing, or the popup's size. Small effects need large samples to detect, so on modest traffic the bold changes are the ones a test can actually measure. Keep the discount the same between versions unless you mean to test it, so the result is about the design.

Should I decide a popup test by signups or by revenue?

Signup rate per view needs the least traffic and suits most stores. It can reward a version that attracts people who never buy, so if each version can reach 1,000 assigned visitors and 30 new subscribers or buyers, net-new subscribers or order rate are a better check. Revenue per visitor is the most direct measure but the noisiest: in Veya it needs 5,000 visitors and 30 orders per version before a verdict.

Sources

Figures checked November 6, 2026.

Your next popup?
We’ll build it for you.

14-day free trial on every plan. Custom design and revisions included.