What does statistical significance mean in an ad test?

Every ad test starts from a default assumption, called the null hypothesis, that the two ads perform the same and any gap in the results is noise. A significance test asks how surprising the observed gap would be if that assumption were true.

The p-value is the answer. A p-value of 0.05 means a gap at least this large would turn up about 5% of the time if the ads were truly the same. The significance level, written as alpha, is the risk you accept in advance of declaring a difference that is not there. Most testers set it at 5%.

Some tools report confidence instead. It estimates how likely the better-looking ad really is better, or how likely you would see the same winner again. Meta's A/B test uses the second version.

Two limits apply to all of these numbers. The American Statistical Association's 2016 statement says a p-value does not measure the probability that a hypothesis is true, and that decisions should not rest only on whether a p-value passes a threshold. It also says significance does not measure how big an effect is.

Purchases are rare events, and rare events are noisy. A few buyers more or less can flip a test. The figures below are illustrative, made up to show the effect rather than taken from any real account. In every row, Ad B converts 50% better than Ad A.

In the first row, Ad B looks like a clear winner, with half as many sales again. Yet a gap that size would appear about one time in three if the two ads were identical. Five buyers decide the whole result. At four times the traffic, the same conversion rates pass the usual 5% bar. At ten times the traffic, chance is a poor explanation.

The p-values come from a standard two-proportion test, and the right-hand column from a simple Bayesian comparison of the two rates. You can run your own numbers in the A/B test calculator.

Look at the first row again. An 84% chance that B is better sounds convincing next to a p-value of 0.32, yet both describe the same thin data. Probability-style figures look strong earlier than p-values do, which matters when you read Meta's results.

Meta picks the winning version by comparing each version's cost per result for the campaign's objective, for example cost per purchase. It then reports a confidence percentage. Meta describes this as the likelihood that you would get the same results if you ran the test again, and says it simulates possible outcomes tens of thousands of times, based on the data from the test, to see how often the winner would have won.

Meta also says how to read the number. Below a threshold that depends on how many versions you tested, it treats the versions as having performed similarly:

Above 75%, whatever the number of versions, Meta says it considers the results very clear. Results start to show once at least 100 events have been recorded, but Meta says to wait until the test has ended before judging, and to rerun a low-confidence test with a longer schedule or a higher budget.

A 75% result still leaves room for error. By Meta's own definition, it means about a one in four chance that a rerun would not produce the same winner. The conventional bar in statistics is a p-value below 0.05, so treat a result in the 70s as a lean rather than a verdict, especially when it decides where a large budget goes.

Peeking means checking a running test and stopping as soon as one ad looks significant. It feels efficient, but it breaks the maths. A 5% significance level assumes you look once, at a sample size fixed in advance. Each extra look is another chance for noise to cross the line, so a test checked every day and stopped at the first significant reading declares false winners far more often than 5% of the time.

The statisticians Ramesh Johari, Leo Pekelis and David Walsh showed that conventional p-values and confidence intervals become unreliable when people decide when to stop by continuously monitoring a test, and they developed sequential methods that stay valid under monitoring. Meta's guidance points the same way. It shows interim results but tells advertisers to judge a test only once it has ended.

Ad tests add their own early bias. In Meta's creative test, for example, an ad that clears review first can take the first day's budget before the others go live. Set the length and the decision rule before launch, and let the test finish.

The fix for peeking is to decide the sample before the test starts. That decision rests on the smallest difference worth detecting. In the standard sample size formulas, such as those in the NIST engineering statistics handbook, that difference sits squared in the denominator, so halving the smallest difference you care about roughly quadruples the data you need.

Take the illustrative rates from the table again. To detect a lift from a 0.5% to a 0.75% conversion rate, with a 5% significance level and an 80% chance of catching a real lift, a standard calculation calls for about 15,600 visitors per ad. At those rates that is roughly 80 purchases for Ad A and 120 for Ad B. A test that stops at a dozen purchases each cannot reliably detect a lift of that size.

Meta builds the same idea into some of its tests. Where it shows an estimated power before launch, it recommends 80% or higher, and the levers are more budget or a longer schedule. If your budget cannot get there, test bigger differences, such as two different concepts rather than two button colours, because large gaps show up in smaller samples. The guide to testing budgets turns this into spend per variation.

A test with enough data can detect a difference too small to matter. In a very large test, say 10,000 purchases per ad, a 3% relative gap in conversion rate can pass the 5% bar. If the winning ad also costs more to deliver, its cost per purchase may be no better. The ASA statement makes the general point. Significance does not measure the size of an effect or the importance of a result.

So read a result in two steps. First, ask whether the gap is likely to be real, using the p-value or Meta's confidence. Then ask whether it is big enough to change what you spend, using cost per purchase or cost per lead. Only a result that passes both earns a budget shift.