Mockingbird

A/B test calculator for ads

An A/B test calculator tells you whether the difference between two ads is real or could be chance. Enter each ad's base (impressions, clicks or landing page views) and conversions, and it returns the conversion rates, the relative lift, a p-value from a two-proportion z-test, and the chance that B beats A.

A p-value under 0.05, or a chance to beat above about 95%, is the usual bar before you move the budget.

Ad A

Ad B

Chance B beats A

99.4%

From a Bayesian comparison of the two conversion rates.

Relative lift

29.5%

A converts at 1.40%, B at 1.81%.

p-value

0.0111

Two-sided z-test on the two proportions.

Result

Significant

Under 0.05. The difference is unlikely to be chance.

Use the same base for both ads: impressions if you are testing click-through, clicks or landing page views if you are testing conversion. Worked out in your browser.

How it works

The p-value comes from a pooled two-proportion z-test. It answers: if the ads were really the same, how often would a gap this big appear by chance?

The chance to beat compares the two conversion rates as probability distributions, which is easier to act on: 97% means B is very likely the better ad.

Common mistakes

  • Calling a test after a day. Early leads flip all the time.

  • Mixing bases, for example impressions for one ad and clicks for the other.

  • Testing ads that differ in five ways, so you never learn which change mattered. See /guides/ad-creative-testing.

Frequently asked questions

What is statistical significance in ad testing?
It means the gap between two ads is unlikely to be random. The usual threshold is a p-value under 0.05, which means a gap that big would appear by chance less than 5% of the time.
Should I use Meta's A/B test feature or this calculator?
Meta's A/B test splits the audience for you and is the cleaner test. This calculator is for checking results you already have, for example two ads running in the same ad set.
What base should I use?
Impressions when you are testing whether people click, clicks or landing page views when you are testing whether they buy.

Keep reading