Calculate the statistical significance of your A/B split conversion test. Simply add the number of visitors and conversions and click calculate.
Calculate your statistical significance
Significant result!
Variant B’s conversion rate (2.35)% was higher than variant A’s conversion rate (2.00)%. You can be 95% confident that variant B will perform better than variant A.
p value
0.0082
SRM alert
Assuming you intented to have a 50% / 50% split, a Sample Ratio Mismatch (SRM) check indicates there might be a problem with your distribution.
What is the statistical significance?
Statistical significance helps you to understand whether a test result can be explained by pure chance or real interest. When a result is significant, it means that it is caused by something other than chance. When a result is not significant, it means that your sample data do not represent a reliable outcome. Your result is likely to be caused by pure chance.
Why should you A/B test for statistical significance?
When you A/B test the conversion performance of your website or ad campaign, it is crucial to ensure that your sample size reflects your wider audience. Testing your results for satistical significance will help you to rule out that your results were the outcome of a lucky (or unlucky) coincidence.
Use my above A/B significance calculator to be confident that your conversion results are the outcome of a real impact of your A vs. B version and not based on pure chance.
How to calculate statistical significance
Simply pop in your A/B testing results in the above calculator. Enter your visitor and conversion count of your two versions A and B.
You can also select at which confidence level you would like to get the results for (e.g., 90, 95 or 99). Selecting a 90% confidence level means that you can be 90% confident that variant B has performed better (or worse) than variant A.
You’ll also get the p-value displayed for your data set as a reference. It describes how significant your results are in relation to the null hypothesis, which states that there is no relationship between the two variables of your A/B test. In other words, it assumes that one variable does not affect the other.
The smaller the p-value, the less likely it is that your data are explainable by chance alone. As a result, it refutes the null hypothesis. You typically want to have a p-value below 0.05, but ideally even below 0.03.
What is SRM and why do you get an alert?
SRM stands for Sample Ratio Mismatch. It measures the split between your sample test control group A and experimental group B. Usually, tests are performed with an equal 50/50 split allocation between both groups.
A sample ratio mismatch alert means that the observed traffic split does not equal the expected traffic split. It will be displayed if your sample group size A deviates ±5% from your sample group size B.
Your results are still going to be displayed but may be less reliable due to the comparison of two groups with different sample sizes. Always try to compare fairly equal sample sizes of your control group A and your experimental group B.
What to do if your results aren’t significant
So what if your A/B split testing results aren’t significant? First, be assured that it is more common than not. Second, don’t be disappointed. It simply means that your variant B did not perform better or worse than variant A because of the changes you’ve made. This can be a powerful insight.
If your p-value is higher than .05, you should stick to your current version A, as the performance of variant B is likely to be driven by pure chance. If you don’t know how to explain this, you may want to ask your visitors directly. Consider running surveys or focus groups to get a better understanding of your page’s performance.