Statistical testing is a critical tool in data analysis, helping us determine whether observed patterns are likely to be genuine or just the result of random variation. However, when multiple comparisons are made, the risk of encountering a false positive increases. This is where the Bonferroni correction comes in.
Bonferroni Correction
The Bonferroni correction is a method used to address the problem of multiple comparisons. When conducting multiple statistical tests simultaneously, the chance of obtaining at least one significant result due to random chance increases. The Bonferroni correction adjusts the significance level to account for the number of tests being performed.
Process Breakdown
- Determine the Number of Tests (N): Identify the number of comparisons or tests you are conducting.
- Adjust the Significance Level (α): Divide the desired overall significance level (typically 0.05) by the number of tests. This gives you the new significance threshold for each individual test.
If your original significance level is α and you are conducting N tests, the Bonferroni-adjusted significance level is α/N.
Example
Imagine you are running a marketing campaign, and you want to see how well it performs over a month. To do this, you collect data every week for four weeks.
Steps
- Number of Exposures: This means you track how many times people see your campaign each week. For example, if one person sees the campaign three times in a week, that counts as three exposures.
- Conversion Rates: This is the percentage of people who make a purchase after seeing your campaign. If 100 people see your campaign and 5 of them make a purchase, your conversion rate is 5%.
Example Breakdown
- Week 1: You collect data and find out that your campaign was seen a total of 1,000 times, and 50 people made a purchase. This gives you a conversion rate of 5%.
- Week 2: The campaign was seen 1,200 times, and 72 people made a purchase, resulting in a 6% conversion rate.
- Week 3: The campaign had 1,300 exposures, with 91 purchases, leading to a 7% conversion rate.
- Week 4: There were 1,500 exposures, and 120 purchases, giving you an 8% conversion rate.
Determining Significant Differences
Now, you want to find out if these differences in conversion rates from week to week are just due to random chance or if they are significant (meaning there is likely a real difference in performance between the weeks). Understanding if these differences are significant helps you know whether your campaign is actually improving, staying the same, or if any changes you make are having a real impact.
Since you are comparing multiple weeks (Week 1 vs. Week 2, Week 1 vs. Week 3, etc.), you need to make sure your results are not misleading. This is where the Bonferroni correction helps by adjusting the threshold for what you consider statistically significant.
In simple terms, you are making sure that the differences you see are not just by chance but are likely due to real changes in your campaign’s effectiveness. You want to determine if there are significant differences in conversion rates across these weeks.
Apply the Bonferroni Correction
- Hypotheses:
- Null Hypothesis (H0): There is no difference in conversion rates between the weeks.
- Alternative Hypothesis (H1): There is a difference in conversion rates between the weeks.
- Conduct Comparisons: You perform pairwise comparisons between the conversion rates for each week. For four weeks, there are (4 : 2) =6 pairwise comparisons.
- Adjust the Significance Level: If your original significance level (α) is 0.05, the Bonferroni-adjusted significance level for each test is 0.05/6≈0.0083.
- Perform the Tests:
- Week 1 vs. Week 2
- Week 1 vs. Week 3
- Week 1 vs. Week 4
- Week 2 vs. Week 3
- Week 2 vs. Week 4
- Week 3 vs. Week 4
For each pairwise comparison, you calculate the p-value. If a p-value is less than 0.0083, the difference is considered statistically significant.
Practical Example
Assume you have the following conversion rates over four weeks:
- Week 1: 5%
- Week 2: 6%
- Week 3: 7%
- Week 4: 8%
After performing pairwise comparisons, you get the following p-values:
- Week 1 vs. Week 2: p = 0.15
- Week 1 vs. Week 3: p = 0.02
- Week 1 vs. Week 4: p = 0.005
- Week 2 vs. Week 3: p = 0.25
- Week 2 vs. Week 4: p = 0.01
- Week 3 vs. Week 4: p = 0.04
Applying the Bonferroni correction (α = 0.0083), only the comparison between Week 1 and Week 4 (p = 0.005) shows a statistically significant difference. The other comparisons do not meet the stricter threshold.
Conclusion
The Bonferroni correction is a straightforward and effective way to mitigate the risk of false positives in multiple comparisons. By adjusting the significance level based on the number of tests, it ensures that the results are more reliable. In our example, despite several pairwise comparisons, only the difference between Week 1 and Week 4 conversions is statistically significant after correction, highlighting the importance of this method in data analysis.
Need help with statistical testing — contact us
