What is the belief, and where does it come from?
The belief is that A/B testing is for large stores, and that a store with modest traffic should not bother because tests will never reach significance. It comes from experience and from arithmetic. Store owners who ran a test on a small lift watched it sit inconclusive for weeks, and anyone who has put a low baseline conversion rate and a small target lift into a sample-size calculator has seen a number in the tens of thousands per variant.
The belief deserves respect, because the calculator is right about the small lift. The question is whether the small lift was the right thing to test.
What is true in the belief?
What is true is that the sample size a test needs rises steeply as the effect it must detect gets smaller and as the baseline rate gets lower. A store converting at a low rate that wants to detect a modest relative lift needs many thousands of visitors in each variant, and at a few hundred visitors a day that is months. It is also true that a test run for months crosses seasons and campaigns, which contaminates it. For small effects at small stores, the belief is correct and the honest advice is not to run that test.
It is also true that a store with very little traffic is better served by qualitative research and considered redesign than by a testing program, and the Conversion FTE says so rather than proposing tests that cannot conclude.
What does traffic actually decide?
Traffic decides the smallest effect a store can detect within a time it is willing to wait, not whether it can test at all. The same calculator that says a small lift needs tens of thousands per variant says a large lift needs far fewer. A small store therefore has a different testing strategy, not no testing strategy: test bold changes whose effect, if real, would be large, and test them on the step with the most traffic, usually the product pages or the cart, where the sample accrues fastest.
The Conversion FTE on ftes.ai calculates the sample size from the store's own baseline and the owner's minimum detectable effect before every proposal, and states the expected duration plainly. A proposal that would take four months says four months, and the owner decides whether that hypothesis is worth it or whether a bolder one should go first.
Why is peeking more dangerous at low traffic?
Peeking, reading a running test and stopping when it looks significant, inflates false positives at any traffic level, and it is more dangerous at low traffic because the test runs longer and there are more mornings to be tempted. Early in a test the numbers swing widely; a variant that is ahead on day nine is often behind on day twenty. Stopping on the good day records a winner that was noise.
The protection is a stop rule set before launch: the test is read at the planned sample size and not before. The Conversion FTE enforces this mechanically; the morning brief shows a running test's planned read date, not its interim result, so the temptation does not arrive.
What should a small store test first?
A small store should test changes that could plausibly move the number a lot, on the step with the most traffic, judged on revenue per visitor. Candidates are the ones the drop-off map and the store's own signals point at: shipping cost shown in the cart instead of at checkout, a guest checkout option, a product page that answers delivery and returns above the fold, or a bold simplification of a cluttered template. Small copy tweaks are the wrong first test at low traffic, however appealing they look.
The store should also accept a larger minimum detectable effect and, where it can, a lower confidence threshold with the risk stated, rather than pretending a test can detect what its traffic cannot.
How can a store check for itself?
A store can check the belief with a sample-size calculator and its own numbers. Enter the baseline conversion rate on the step to be tested, the smallest lift worth detecting, and the confidence threshold; read the visitors per variant; divide by daily visitors to that step. If the answer is weeks, the test is feasible. If it is months, raise the minimum effect or pick a bolder change, and read the answer again. The exercise takes minutes and ends the argument with arithmetic rather than opinion.
What remains true is that some stores are too small for any test to conclude in a season. For those stores, the honest path is research and redesign, and the Conversion FTE's proposal will say that instead of proposing a test.