Computing Minimum Sample Size for A/B Tests in Statsmodels: How and Why

Introduction

There is currently no good resource on how Statsmodels computes the minimum sample size.

It is critical to calculate the minimum sample size required before conducting an A/B test. A popular way to do it is by calling the tt_ind_solve_power function in Python's Statsmodels package, but there are currently 2 gaps when it comes to understanding how it works:

1. There are many great articles (e.g. by Stan Nsky, TDS 2019) explaining what the parameters mean and provide examples of function calls. However, they do not explain how the function actually computes the sample size and why the procedure is correct.
2. There are also many great articles (e.g. by Mintao Wei, TDS 2023) that explain the statistical derivation based on a z-test for proportions such as conversion rates, which is also a popular choice for many online sample size calculators (e.g. Evan Miller's Calculator). However, this is not the method used by Statsmodels and results can differ.

This is important for data scientists because Statsmodels is commonly used to compute sample size in Python.

Data scientists frequently use Statsmodels to get the minimum sample size, but may not be aware that it employs a different method from what most articles describe and what most online calculators use. It is essential to understand how the function works so that we can trust its results.

This article bridges the gap by explaining how Statsmodels actually works.

This article aims to make the novel contribution of explaining how tt_ind_solve_power actually computes the sample size, why the procedure is correct and what benefits it brings over closed-form solutions. [1]

Part 1: It will first explain how sample size is computed and why the procedure is correct in two steps:

1. Show the statistical derivation for sample size calculations.
2. Write a stripped-down version of tt_ind_solve_power that is an exact implementation of the statistical derivation and produces the same output as the original function

Part 2: Following which, it will explain two benefits it brings over closed-form solutions:

1. Benefits to generalizability
2. Benefits to statistical intuition

---

Part 1: How Statsmodels computes minimum sample size and why it is correct

1.1. Showing the statistical derivation for sample size calculations

Core Idea

A general A/B test is an unpaired two-sample t-test. Rather than using a closed-form solution, Statsmodels obtains the minimum sample size in two steps:

1. For a given sample size, compute the associated power of the test.
2. Run a numerical optimization algorithm to find the sample size that returns the target power of the test.

Notation and Concepts

These are some terms we will use throughout the article:

• n: minimum required sample size. n = n_1 + n_2
• n_1, n_2: minimum required sample size for the treatment and control group, respectively
• ratio: n_2 = n_1 * ratio, where for a 50:50 allocation, ratio = 1
• p: p-value

• Tags: A B Testing Data Science Hands On Tutorials Product Analytics Sample Size

  Add Fav