Back to Glossary
Glossary
False Positive (Type I Error)
Incorrectly declaring a winner when there is no real difference between variants.
What is a False Positive?
A false positive (also known as a Type I error) happens when your test results show that Variant B is better than Variant A, but in reality, the difference was just due to random chance.
The Risk of False Positives
If you implement a "winner" that is actually a false positive, you might spend time and money on a change that provides no real benefit, or worse, actually hurts your performance in the long run.
How to Avoid Them
- Higher Confidence Level: Using 95% instead of 90% reduces the risk.
- Adequate Sample Size: Don't stop tests too early.
- Bayesian Methods: Bayesian tools (like runab) provide an "Expected Loss" metric that specifically quantifies the risk of a false positive.