What is Statistical significance?

Statistical significance is the judgement that a difference in test results is unlikely to be chance alone. Testers set a significance level in advance, commonly 5%, and compute a p-value, which is how often a gap at least as large would appear if the versions were really the same. A p-value below the level counts as significant. In ad testing it shows whether one ad's better result is likely to hold.

Significance depends on sample size, so tests with few purchases rarely reach it. It does not measure how big or valuable a difference is, a point the American Statistical Association made in its 2016 statement. Meta's A/B test reports a related figure, a confidence percentage, which estimates how likely the same winner would come out on top if the test were repeated.