Back to Glossary
Glossary
Sample Size
The total number of visitors or participants required for your A/B test to produce reliable results.
What is Sample Size?
Sample size refers to the number of users who participate in your experiment. In A/B testing, having an adequate sample size is crucial for ensuring that your results are statistically valid.
Why Sample Size Matters
- Accuracy: Small sample sizes are prone to "noise" and random fluctuations.
- Power: You need enough users to detect a difference between variants (especially if the difference is small).
- Confidence: The larger the sample, the more confident you can be in the results.
Factors Affecting Sample Size
- Baseline Conversion Rate: If your current conversion rate is very low, you'll need more traffic to see a change.
- Minimum Detectable Effect (MDE): The smaller the improvement you want to detect, the more users you need.
- Statistical Power: Usually set to 80%, this is the probability of detecting an effect if there is one.
How runab Handles Sample Size
Unlike traditional tools that require you to calculate sample size upfront, runab's Bayesian engine allows you to monitor results as they come in, providing clear "Probability to Win" and "Expected Loss" metrics regardless of your current sample size.