Test a popup against no popup
Find out whether a popup adds orders by showing it to some visitors and no popup to the rest, then keep the popup or switch it off.
A popup collects signups, but signups alone don't tell you whether it changes how many visitors buy. A Popup vs no popup test answers that: some visitors see none of the test's forms, everyone else sees your form, and Veya compares how often each group orders.
For why this test matters and what other stores have found, read the popup vs no popup guide.
When to use it
- You want to know whether your popup earns its place, not only how many signups it collects.
- Someone on your team worries the popup costs sales, or you want proof before adding an offer.
- You changed your offer or design and want to check the popup still helps.
The test needs traffic. If your store has a few hundred visitors a day, expect it to run for weeks. See how long it takes.
Start the test
The form must be live.
- Open the form and go to its Optimize tab.
- Under Tests, click New test.
- Choose Popup vs no popup. The dialog is titled Test “your form” against no popup.
- Check Test name. It starts as the form's name followed by "popup vs no popup".
- Under Decide the winner by, choose Order rate (per assigned visitor) or Revenue per visitor (needs far more traffic). Visitors who see no popup can't sign up through it, so the test is decided on orders, never signups.
- Under Visitors who see no popup, choose 50%, 30%, 20% or 10%. A 50% split answers fastest. A smaller share risks fewer signups while the test runs, but takes longer.
- Click Start test.
Who sees what
- Visitors in the no-popup group see none of the forms in this test. Your other popups still show to them as usual.
- Everyone else sees the form, with its usual audience, trigger and frequency rules.
- Each visitor keeps their group for the whole test, on every visit.
Orders count for every assigned visitor, whether or not they ever saw a popup, within your store's attribution window after they were assigned (30 days by default). This is called intent-to-treat: it compares the two groups as they were split, so visitors who left before the popup opened count too.
Read the results
The test's card on Optimize shows each group.
- The No popup row is the control. It reads Sees none of the forms in this test, because it has no views or signups.
- The form's row shows its views, signups and conversion rate as usual.
- The outcomes table shows orders and revenue per assigned visitor for both groups. Its last column, vs. no popup, shows the difference with a range, marked significant when the difference is reliable.
When the test has enough data, the card gives a verdict:
- Showing “your form” does better than showing no popup: the popup adds orders. Click Keep this one on the form's row to keep it and stop the test.
- Showing no popup does better than showing this form: the popup costs orders. Click Keep no popup to stop the test. Veya asks you to confirm, then pauses every form in the test.
- No reliable difference: see what no difference means.
You can end the test at any time with Keep this one, Keep no popup or Stop test.
How long it takes
A verdict needs all of these:
- at least 7 days, so every weekday is included
- at least 1,000 assigned visitors in each group
- at least 30 buyers across the two groups
Deciding by revenue per visitor needs 5,000 visitors and 30 orders in each group, because a few big orders can swing revenue.
These are the minimums before Veya will say anything. Popups usually change orders by a few percent, and a small difference takes far more visitors to detect: at a 4% order rate, a 10% lift needs roughly 39,000 visitors per group. The guide shows the estimate.
What no difference means
"No reliable difference" means the test could not tell the two groups apart on orders. It does not prove the popup does nothing: the effect may be smaller than this much traffic can detect. It is a common result.
If you get it:
- The popup is not costing you a measurable number of orders, and it still grows your list. Signups you collect today can buy later, after the attribution window.
- Keep the popup if list growth is worth it to you, or run the test longer for a clearer answer.
How it differs from the holdout
Analytics also has a store-wide Holdout under Sitewide impact. They answer different questions:
| Popup vs no popup | Holdout | |
|---|---|---|
| What it measures | One form against no form | All your popups against none |
| Who sees no popup | 10% to 50% of visitors, chosen in the test | 0% to 20% of visitors, set under Hold out |
| Other popups | Still show to the no-popup group | Never show to held-out visitors |
| How it ends | A verdict, then Keep this one or Keep no popup | Runs until you set it to 0% |
Use the holdout to measure what popups add to your store overall. Use Popup vs no popup to decide about one form.
Agents
Agents connected over MCP can run the same test: the experiments tool's start action with noPopupWeight set to the share of visitors who see no popup. See A/B tests for other kinds of test.
Your next popup?
We’ll build it for you.
14-day free trial on every plan. Custom design and revisions included.