A/B Test Duration and Sample Size Guide: Know When a Conversion Experiment Is Reliable
A/B testingCROexperimentationsample sizeconversion rates

A/B Test Duration and Sample Size Guide: Know When a Conversion Experiment Is Reliable

CClicker Cloud Editorial Team
2026-11-13
7 min read

Estimate A/B test sample size and duration, set stopping rules, and check significance, seasonality, and validity before choosing a winner.

Knowing how long to run an A/B test is less about waiting for a fixed number of days and more about collecting enough representative data to make a dependable decision. This guide explains how to estimate A/B test sample size and duration, set stopping rules, check statistical significance, and document the result before choosing a winner.

Overview

An A/B test compares a control version with one or more alternatives against a defined outcome, such as a form submission, checkout completion, trial start, or product purchase. The test is useful only when the conversion event is measured consistently and the observed difference is large enough to distinguish from normal variation.

There is no universal answer to “how long should I run an A/B test?” A suitable duration depends on your baseline conversion rate, expected improvement, eligible traffic, allocation between variants, number of variants, and the natural rhythm of your business. A test with a small expected improvement may require substantially more visitors than a test designed to detect a large change.

Duration is therefore an output, not the main input. Estimate the required sample size first, then divide it by the number of eligible visitors or users expected per day. Treat that result as a planning estimate rather than a promise. You still need to account for weekday and weekend behavior, campaign changes, delayed conversions, and technical issues.

Before starting, define one primary conversion event. Supporting metrics, such as button clicks, scroll depth, or revenue per visitor, can help explain the result, but changing the main success metric after seeing the data weakens the test decision. For event measurement, see the GA4 event tracking plan template and the website click tracking guide.

How to estimate

Use this repeatable workflow for an A/B test duration estimate:

  1. Record the baseline conversion rate. Use a recent, stable period for the same audience and page. If the baseline is 4%, that means approximately 4 conversions per 100 eligible visitors.
  2. Choose the minimum detectable effect. Decide the smallest improvement that would justify implementation. You can express it as a relative lift or an absolute percentage-point change. For example, moving from 4% to 5% is a one-percentage-point increase and a 25% relative lift.
  3. Set the decision criteria before launch. Choose the significance threshold, desired statistical power, allocation, and primary metric in advance. Your experimentation platform may use a particular statistical method, so use its terminology and calculation settings consistently.
  4. Estimate the required sample size per variant. A sample-size calculator usually needs the baseline rate, target rate, significance level, power, and number of variants. If you do not have a calculator available, treat any rough estimate as directional rather than as a launch decision.
  5. Convert sample size into duration. Add the required visitors across all variants and divide by the expected number of eligible visitors per day. If each variant needs 2,000 visitors and there are two variants, the planned total is 4,000 visitors. At 400 eligible visitors per day, the arithmetic estimate is 10 days.
  6. Apply a representativeness check. Avoid ending immediately when the arithmetic target is reached if the test has not covered the normal traffic pattern. A calendar period that includes only unusual campaign traffic may not represent ordinary behavior.

A simple planning formula is:

Estimated duration = total required eligible visitors ÷ average eligible visitors per day.

This formula is intentionally simple. It does not correct for sample-ratio mismatch, tracking loss, delayed conversions, or changes in traffic quality. Those checks belong in the validity review.

Planning situationSample-size implicationDuration implication
Large expected differenceOften fewer visitors are needed to detect the effectMay be shorter, subject to validity checks
Small expected differenceUsually requires more visitorsMay require a longer test or more traffic
Low baseline conversion rateConversions accumulate slowlyPlan for more time or a higher-volume page
Several variantsTotal traffic is divided among more experiencesEach variant may take longer to reach its target
Strong weekly or seasonal variationSample must cover relevant traffic patternsDo not rely only on a short, unusual period

Inputs and assumptions

The quality of an A/B test duration calculator depends on the quality of its inputs. Document these values in a short experiment brief before publishing the variation:

  • Unit of analysis: Decide whether the test counts users, sessions, accounts, or another stable unit. Do not switch units during the experiment.
  • Eligible traffic: Count only visitors who could actually experience the tested page and complete the primary action. Exclude traffic that is not part of the intended audience when your setup supports that distinction.
  • Baseline rate: Use the conversion rate for the same funnel step, audience, device mix, and measurement definition. A sitewide rate may not be suitable for a landing-page test.
  • Minimum detectable effect: Set a practical threshold. An effect can be statistically detectable but too small to justify the implementation cost, risk, or maintenance.
  • Traffic allocation: Record the planned split, such as an even control-and-variant allocation. Confirm that the observed allocation is close to the plan.
  • Conversion window: If users can convert hours or days after the initial visit, allow the chosen attribution window to mature before making a final decision.
  • Test interference: Note concurrent campaigns, pricing changes, outages, audience exclusions, or other experiments that could affect the result.

A useful brief can be as short as this:

Hypothesis: Changing the primary CTA will increase completed sign-ups without reducing qualified traffic. Primary metric: completed sign-up per eligible user. Baseline: 4%. Minimum worthwhile change: 1 percentage point. Variants: control and one treatment. Stopping rule: reach the planned sample, check validity, allow delayed conversions to mature, then review the preselected decision criteria.

For prioritizing what to test next, use the CRO test prioritization template. For funnel context, review the conversion funnel analysis guide.

Worked examples

Example 1: Estimating calendar duration

A landing page receives an average of 600 eligible visitors per day. The control converts at 4%, and the team wants to detect a meaningful increase to 5%. The experiment has two equally allocated variants. If the selected calculator indicates that each variant needs 2,000 eligible visitors, the total planned sample is 4,000 visitors. At 600 visitors per day, the arithmetic duration is about 6.7 days.

The team should not automatically stop on day seven. It should confirm that both variants received the intended traffic, the sign-up event fired correctly, and the period was not dominated by an isolated campaign. If the business has a strong weekday pattern, extending the test to cover a complete representative cycle may be more appropriate than stopping at the first mathematical threshold.

Example 2: Why low traffic changes the answer

Suppose the same test receives only 80 eligible visitors per day. The sample requirement has not changed simply because traffic is lower, so 4,000 total visitors would take about 50 days. That duration may be reasonable, but it may also reveal that this page is not an efficient testing surface for a small expected change. Options include testing a higher-volume step, selecting a larger practical change, improving measurement first, or accepting that the experiment will need more time.

Example 3: A result that needs investigation

After reaching the planned sample, the variant appears to win, but the variant received 60% of visitors instead of the intended 50%. Before choosing a winner, investigate the allocation problem, deployment history, duplicate users, and event collection. A favorable conversion difference does not compensate for an invalid or biased experiment. Record the issue and, if necessary, rerun the test after correcting the implementation.

When to recalculate

Recalculate the A/B test sample size or duration whenever an important input changes. That includes a material shift in baseline conversion rate, a revised minimum worthwhile effect, a new audience, a different traffic allocation, additional variants, or a change in the primary conversion event.

Revisit the estimate before every new experiment and after major changes to the page, funnel, analytics implementation, campaign mix, or product flow. Also review it when traffic volume or conversion volume moves enough to alter the expected calendar duration. Keep the original estimate with the final result so later tests can be compared against the assumptions that shaped the decision.

At the end of the test, use this practical checklist:

  1. Confirm the planned sample and conversion window were reached.
  2. Check traffic allocation and investigate sample-ratio mismatch.
  3. Verify that the primary event, variants, and audience targeting worked as intended.
  4. Review statistical significance using the method provided by your testing platform or analysis process.
  5. Check absolute change, relative lift, uncertainty, and practical business value—not only a win or loss label.
  6. Look for weekday, device, audience, and novelty effects that may require a follow-up test.
  7. Document whether to implement, iterate, rerun, or stop the idea.

Use the A/B test duration calculator guide as a planning reference, then update the inputs when your traffic, baseline, or testing goal changes. A reliable experiment is not merely one that ran for a certain number of days; it is one whose design, measurement, sample, and decision rules were defined before the result was known.

Related Topics

#A/B testing#CRO#experimentation#sample size#conversion rates
C

Clicker Cloud Editorial Team

SEO and Analytics Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.