The challenge: deciding which variant wins on an experiment when the dataset is small by design.

An A/B test puts two versions on the market (a page, an offer, an experience) and asks which performs better before a full commitment to either. Observations accumulate from the first day, but on limited traffic variance stays high, confidence intervals stay wide, and a directional lean can appear while classical methods still wait for statistical significance. At ZyG, a platform that identifies high-potential brands and executes everything needed for eCom scale,  it is often necessary to make decisions long before statistical significance is achieved. For example, a brand with 1,000 sessions a day testing conversion rates across two landing pages will need an allocation decision on a timeline that doesn’t allow for statistical significance. The question is how to decide on samples that are small by design, without overstating the impact and without treating the absence of significance as indefinite deferral.

The traditional approach

For decades, A/B testing was built on frequentist statistics. A common tool is the Z-test, comparing two groups and asking whether an observed gap is likely random noise or a real difference. The output most people know is the p-value: if A and B were actually the same, how likely is a gap this large? Cross a threshold, and the test yields statistical significance: permission to call a winner. That framework assumes enough observations to reach a conclusion at the scheduled endpoint. Until then, the prescribed action is to wait. For high-traffic tests with long runways, that works well. This is why the Z-test became the industry default.

Where that tradition meets a different reality

In practice, experiments often run alongside the decisions they are meant to inform. Significance may not arrive for weeks. The framework appropriately withholds a conclusion given the sample, yet the decisions an experiment is meant to inform cannot always be deferred. That is why we needed statistical methods built for this use case.

A different way to read the same experiment

Bayesian statistics reframes the question. Rather than asking whether a difference is statistically significant it asks: given the observations so far, how probable is it that one option is better, and how large is the gap likely to be? As data accumulate, the method updates a posterior on the true gap and reads off win probability and a credible interval. Binary outcomes use a Beta-Binomial model, continuous outcomes a Normal-Normal comparison, from the first observation forward rather than waiting for the test to end.

The method is built to work on small samples: a read is available even when the numbers are still low, with moderate win probability and a wide credible interval. As the sample grows, the estimate becomes more accurate, win probability rises when a real gap is present, and the interval tightens.

To estimate whether more users at the current traffic rate would raise win probability and sharpen the read enough to decide, Monte Carlo preposterior projection simulates future traffic and projects the read forward from the current posterior. Rather than a single forecast, it draws many plausible paths of future observations, runs the same Bayesian update on each path, and estimates how often the read would sharpen enough to settle the question.

What we built at ZyG

A/B testing is the clearest surface: two variants, one allocation choice, observations in real time. At ZyG that is how every experiment is decided: win probability, credible interval, and ship, stop, or keep testing on the metric the test was opened to judge, against thresholds agreed before launch, updated as traffic arrives.