Shipt Sample Size Calculator

Binary Metric Sample Size Calculator

Highlighted Features	Description
Multiple Testing Correction	A/B/n tests with more than one treatment with the Bonferroni testing correction.
Support Unequal Split A/B	Experiments where the A and B groups are not equally split.
Comprehensive Configuration	Configurations for Significance, Statistical Power, MDE, Lift, and Cohen's Distance.
One- or Two-sided Test	Configurations for one- or two-sided tests.
Binary Metric	Configurations for Binary Metrics (e.g. conversion rate, retention rate).
Continuous Metric	Configurations for Continuous Metrics (e.g. dollar amount, number of visitors, order volume).
Ratio Metric	Configurations for Ratio Metrics (e.g. $tips / $order value).
Visualization	Plots of minimum required sample sizes over lifts to decide the optimal size.

Alternative Hypothesis Testing

Plot Sample Size By Lift

Confidence Level (1-α) %

Statistical Power (1-β) %

Number Of Groups

Multiple Testing Correction

Baseline Binary Metric Rate %

Minimum Detectable Effect (Y_t-Y_c) %

Data Allocation
(By Default Equal Split 50:50)

Input parameter	Output
Minimum sample size for control group
Minimum sample size for each treatment group
Total required sample size of control and treatment (If we have more than 2 groups, this number is sum of control and only one treatment that is used to measure the MDE.)
Lift (the difference between two groups over the control)
MDE (the expected difference between treatment group and control group)
Cohen’s Distance (the effect size used to indicate the standardized difference between treatment group mean and control group mean)