Holdout Calculator

What a holdout is. A random group of people you leave out of a send. They're like everyone else in every other way, so the difference between them and the people who got the message is what the message did.

How the tool decides. It compares the two conversion rates with a two-proportion test, the standard way to tell whether two rates really differ. When you plan, it finds the smallest holdout with an 80% chance of spotting the uplift you're after (that chance is called power), at the confidence level you choose. When you read a result, it gives the range the true difference most likely sits in. If the whole range is above zero, the uplift is very likely real.

Its limits. It assumes people were split at random and that each person converts once at most. Revenue is an estimate, because testing it properly needs the figures for each person, not just the totals.