Question 1: was the metric fixed before the test started?
The first question to ask of any A/B test result is whether the metric it reports on was chosen before the test launched. A test launched to move revenue per visitor and reported on add-to-cart rate because that is what moved has changed the question after seeing the answer, and among many metrics one will always have moved by chance. The Conversion FTE on ftes.ai records the metric in the proposal the owner approves, and the result is reported on that metric, with the others shown as diagnostics and labelled as such.
A report that leads with a metric different from the one the test was designed for is the first sign of a story.
Question 2: was it read at the planned sample size?
The second question is whether the result was read at the sample size calculated before launch, and not before. Reading a test daily and stopping when it crosses a threshold inflates false winners badly, because early numbers swing widely and a threshold crossed on a good day is noise recorded as a finding. The proposal the owner approved on ftes.ai states the sample size and the planned read date; the result shows both, and a test read early is not reported as a result at all.
The same question applies to tests that ran past their sample size while someone waited for a better number. Extending a test to reach significance is peeking in slow motion.
Question 3: what is the interval, not the point estimate?
The third question is what the confidence interval says, because the point estimate is one number from a range the data supports. A reported lift with an interval that runs from a small loss to a large gain is not a win; it is uncertainty, and the honest reading is that the test could not tell. An interval entirely above zero is a win at the threshold chosen; an interval that includes zero is a loss, reported as such.
The Conversion FTE reports the interval on every result and names the outcome from it. A labelled worked example, illustrative only: a variant shows a two percent lift in revenue per visitor with an interval from minus one to plus five. The point estimate is positive; the interval includes zero; the result is a loss, and the report says so.
Question 4: what did the change cost?
The fourth question is what the change cost in the metrics the headline does not show. A lift in conversion rate from a discount costs order value and margin; a lift in order value from a shipping threshold costs shipping margin; a lift on one device may cost the other. The decomposition of revenue per visitor into rate and order value shows the first; margin per visitor, with the cost inputs stated, shows the second; segment results by device show the third.
A result that reports only the headline is asking not to be read closely. The Conversion FTE reports the decomposition and the segments on every result, and margin where the change touched cost.
Question 5: what would change your mind?
The fifth question is what evidence would overturn the result, and whether it has been sought. A winning test that fails to replicate, a lift that disappears in the segment the change was aimed at, a result that coincided with a campaign or a season: any of these should move a finding back to a hypothesis. The Conversion FTE names, on every winning result, the check it proposes next: a replication on a later period, a segment read, or a holdout on the change once it is live.
A store that never overturns a result has stopped testing and started believing. The fifth question keeps the discipline honest.
How do you apply the frame to any report?
The frame applies to any A/B test report, from a testing tool, an agency or a Digital FTE. Read the five questions against the report. A metric that was not fixed in advance, a read before sample size, a point estimate without an interval, a headline without its costs, or a win with no proposed check each disqualifies the result as a finding, and the honest label for it is a story. Two or more together mean the report is not worth the time to read further.
On ftes.ai the frame is built into the result: the metric and sample size from the approved proposal, the interval and outcome, the decomposition and segments, and the next check. That is what reporting honestly means in practice, and it is why the Conversion FTE names losses without being asked.