Reading A/B results

Where to find your results

Open a site from Sites, then choose Performance under Results. You get one block per published campaign, and inside it one card per goal that counts in A/B results.

The Range and Show controls at the top change the numbers in the tiles and the daily trend. The verdict, the confidence and the sample size always belong to the current run, whatever range you pick.

How an A/B test works here

• A campaign starts without an A/B test: every matching visitor sees the personalized version, and there is no original to compare against. To start one, open A/B settings in the campaign header and turn on Run as A/B test.
• The holdback is the share of matching visitors who see your original page. You can pick 10%, 20%, 30%, 50%, and 50% is the recommended start.
• Every campaign has its own A/B setting and holdback, changed under A/B settings in the campaign header. The site-wide switch A/B testing turns experiments off for the whole site.
• A visitor stays in the same group for as long as their browser keeps the stored version: until the tab is closed with Session, for up to 30 days with Persistent. Within that time, they do not flip between versions from page to page. See Snippet behavior for the Variant persistence setting, including None.
• A lower holdback sends more visitors to the personalized page, but it makes results slower: a 10% holdback needs roughly three times the total visitors of a 50/50 split to reach the same certainty.
• With a lower holdback, the Original and the Personalized version each have their own planned sample size, and both reach it at about the same time: at a 10% holdback, for example, Original might read 4,000 of 7,938 visitors while Personalized reads 36,000 of 71,438 visitors.

Reading a campaign block

The header of each campaign tells you the state of its experiment:

• Running, Counting (no original to compare against: adfit measures the personalized version on its own), Ended, Ended: plan changed, Ended: measurement turned off, Ended: subscription paused, Ended: payment problem, Ended: subscription canceled, Page changed, Not started yet (A/B testing is on for the site and this campaign, but it has not had a run yet) or A/B test off, followed by the run number, its start date and the holdback when there is a run. A run without an original shows A/B test off: counting all visitors instead of the holdback.
• Run history lists earlier runs with the reason they ended (for example A/B settings changed or Targeting changed) and their result.
• A/B settings and, where it applies, Restart experiment.

A run in the history can carry a note. Each one flags a comparison to read with a little care:

• Holdback raised: returning visitors may already have seen the personalized page: you increased the share that sees the original, so some of those visitors had already seen the personalized page under the earlier setting.
• Visitors before this run saw the personalized page: before this run the campaign showed the personalized version to everyone, so its returning visitors are not meeting it for the first time.
• Site key rotated: returning visitors were re-randomized: after a new site key, returning visitors are assigned to a group again from scratch.
• Restarted after your subscription pause: returning visitors may already have seen the personalized page or Restarted after your payment went through: returning visitors may already have seen the personalized page: the earlier run ended because your subscription was paused or because of a payment problem, and started again once that was resolved.
• Restarted with the pages your plan still includes: returning visitors may already have seen the personalized page or Restarted after an earlier plan change: returning visitors may already have seen the personalized page: a plan change reduced how many of the campaign's pages your new plan includes, so the test restarted with the pages that still fit.

If a change to a smaller plan doesn't go through, the run history can show a run that ended with Ended: plan changed, although your plan stayed the same. After it you see either a very short run, or a new run with the note Restarted after an earlier plan change: returning visitors may already have seen the personalized page. The results of all these runs are valid, and the test continues in a new run.

Reading a goal card

• Two tiles, Original and Personalized, each with its conversion rate, its visitors and its conversions.
• The lift: how much better or worse the personalized version did, in percent, followed by Likely range (95%). The range matters more than the single number: if it spans zero, the difference could still be nothing.
• Confidence with a mark at the threshold this read has to clear: 99.9% for an early read, 95% at the final read.
• Sample: how many visitors the Original and the Personalized version have and, once adfit has planned it, the sample size this run needs for each. At the final read it shows the sample the result is based on.
• By page and By source for the numbers per page and per traffic source, and Daily trend for the day by day chart, which also has a Show as table view.
• In the daily trend, days without any data show as a hatched band marked no data, and the lines break there. Show as table marks such a day as (no data).
• Without an A/B test, a goal card shows only the Personalized tile, with no verdict, lift, confidence or sample, and its Daily trend, Show as table, By page and By source list only Personalized.

Cards for goals other than your primary one carry the badge Indicative: same calculation, but they do not decide the verdict. They never get a final read, so they show no Early read or Final read badge, and their threshold stays at 99.9%.

What confidence means

An example: at 97% confidence, a difference this big would show up in fewer than 3 out of 100 tests where the two versions are in truth exactly equal. That is a good reason to believe the difference is real. It is not a statement that the personalized version has a 97% chance of being the better page, and adfit never claims that.

Read the Likely range (95%) next to the lift together with the confidence. It shows the range of true effects that fit your data, and it is usually wider than people expect.

What each verdict means

• Collecting data: not enough visitors or conversions yet. adfit shows what is still missing and no confidence at all.
• Leaning personalized or Leaning original: one version is ahead, but it has not cleared the threshold for this read. This is a direction, not a result.
• Personalized wins or Original wins: the difference cleared the threshold that applies: 99.9% for an early read, 95% at the final read.
• No clear difference: the run reached its planned sample size without a significant difference. That is a real answer, not a failure: it means any effect is likely smaller than the run was built to detect.
• Results unreliable: adfit hides the verdict because something is off with the data, for example an uneven visitor split or a changed page. The visitors, conversions and conversion rate of each version stay visible; lift and confidence are hidden, and the card says why.

Early read and final read

Next to the verdict of your primary goal, a badge shows Early read or Final read. Early read means there is no final read yet. While a run is still going, an early read can still flip. When a run has ended, the tooltip on the badge reads: Early read: this run ended before reaching a final read. Its early read can no longer flip because of new data: a run that has ended takes in no new visitors or conversions. Until adfit's daily check has covered the day the run ended and every day it received visitors, the read can still change, for example into a final read if the test already has enough data.

A final read needs two things: both versions reach their planned sample sizes, and the test has run for at least 7 days with enough traffic, counted in whole weeks. A day has enough traffic when it has at least 10% of the visitors of an average day with visitors in this test. Days with less, for example while your ads are paused, and days excluded for unusual traffic do not count. adfit checks both every day and gives the final read on the first day both are true, not only at the end of a week. Final read means both conditions are met, and that read is the one adfit stands behind.

The two reads do not use the same bar, and that is the point. For an early read, adfit only calls a winner at 99.9% confidence. At the final read, 95% is enough. The mark on the confidence bar always shows the threshold that applies right now, so the bar and the verdict never tell you different things. Above the bar, the card shows both numbers side by side, for example Confidence 99.4% on the left and Threshold 99.9% on the right.

After the final read, adfit freezes the verdict: visitors keep arriving and the live numbers keep moving (they also include conversions that arrive late), but they no longer change the verdict of that run. If you change the triggers of your primary goal while the run is going, adfit removes the final read and counts again from the next full day.

If your page changes after the final read, that frozen result still describes the period it was measured in, and adfit flags the run as changed. Update the baseline and start a new run to measure the page you have now.

When adfit shows no confidence yet

Below 100 visitors per version, or fewer than about 10 conversions in total at a 50% holdback (more at lower holdbacks), no confidence is shown. Percentages at this size swing wildly, and a number that moves by ten points because two more people converted would only mislead you.

The card tells you exactly what is missing, for example 62 of 100 visitors per version and 4 of 10 conversions in total needed.

How long a test takes

adfit plans a sample size per run from your original's conversion rate and a minimum effect of 20%, and it never calls a read final before the test has run for at least 7 days with enough traffic, counted in whole weeks, so one strong weekday cannot decide your test.

• The lower your conversion rate, the more visitors the run needs. At a 2% to 3% conversion rate and a 50/50 split, a run is a matter of weeks, not days.
• A lower holdback stretches it further, because the original group grows slowly and it is the smaller group that limits the answer.
• Until the original group has enough conversions, the target is an estimate and adfit labels it as one. After that it stays fixed while the run collects data, so new visitors do not move the finish line. Changing your primary goal or its triggers, or turning its Tracking off and on again, while the run is going can start the count again from the next full day, with a new target.

What the original group sees

Visitors in the original group get your page exactly as you built it: adfit applies no change to it. It still decides which group the visitor belongs to before anything on the page is touched, and it still counts that visitor, which is why the comparison has two sides at all.

To avoid a visible flicker, adfit briefly holds back the elements a campaign may change before it knows which version to show. That hold is capped by your safety reveal timeout, described on Snippet behavior, and the original content is revealed at the latest when the timeout is reached.

When a conversion counts

• The visitor must have been assigned to the campaign first. A conversion from someone who never matched is not attributed.
• adfit counts one conversion per visitor, goal and run, however often the goal fires.
• Conversions are attributed while the experiment is running and within your attribution window: the same page with None, up to 24 hours while the tab stays open with Session, or up to 30 days with Persistent. Conversions after an experiment ended are not attributed.
• A visitor who arrives through two different campaigns counts in both experiments.

Visitors adfit cannot recognize for long

Without a consent signal, adfit measures a visitor with a hash that changes daily, so a visitor who comes back on a later day may be counted again, as a new visitor. Which version they see is a separate matter: it is kept in their browser for as long as your Variant persistence setting remembers it. With Session or None, a returning visitor who matches again may end up in the other group. With Persistent, they can be counted again in the same group; see Visitors measured without a stored id.

In your results, visits without a stored id that come from the same IP address and the same browser details count as one visitor on the day of the visit and the day after (UTC), even across several windows or tabs, private windows included. In the consent mode, a visitor who accepted tracking is recognized by a random id stored in their browser instead, for the tests they reach after accepting; a private window does not share that id, so visits there count as a separate visitor.

If your site asks for tracking consent and you connect it, once a visitor accepts, adfit can recognize them with a stored id for up to 30 days instead. Consent changes how long adfit recognizes a visitor for measurement, not which version they see. See Consent and visitor tracking.

Notices that pause or change a result

Uneven visitor split

If visitors are split noticeably differently from your holdback, the campaign header shows the note Uneven visitor split with the measured split and a Check installation link. While the run is going and the primary goal has no final read yet, the note reads, for example: Visitors split 44.0% / 56.0% instead of about 50% / 50%. adfit hides the verdict while the split is uneven. If it persists, check your snippet installation.

• If the primary goal already has its final read, the split no longer changes that read. Verdicts without a final read stay hidden while the split is uneven.
• When the run has ended, the note only names the split of that run and asks you to check your snippet installation. It says nothing about the verdict.

Where a verdict is hidden, the goal card keeps the visitors, conversions and conversion rate of each version and says why, for example: Lift and confidence are hidden because of an uneven visitor split.

If adfit finds the uneven split only after the final read, the final verdict stays, and the card of the primary goal adds: Uneven visitor split, not found at the final read. The verdict stays. Later numbers may be affected.

The original changed

If your page changes underneath a running experiment, the comparison no longer means what it says: some visitors saw one original, others saw another. adfit flags this as Original changed: seen by visitors (detected in real visits) or Original may have changed: server scan, and marks the run Page changed.

Update the baseline on the campaign's pages, then press Restart experiment in Performance. Updating the baseline ends the running test (adfit asks you to confirm first) and does not restart it on its own. adfit never refreshes a baseline by itself, and that is deliberate: silently adopting a changed page in the middle of a run would rewrite the comparison you already collected.

If the run has already ended, the campaign header notes when the original changed and that the run's results since then are unreliable. It does not ask you to update the baseline or restart; Restart experiment appears only where a new run is possible.

If adfit finds the change only after the final read, the final verdict stays, and the card of the primary goal adds: Page change found after the final read. The verdict stays. Later numbers may be affected. If the visitor split is uneven as well, the card shows the split warning instead.

Unusual traffic on some days

When a day looks like automated or otherwise unusual traffic, adfit takes that day out of the results and notes Unusual traffic on 2 days excluded from the results. The experiment keeps running, and you do not need to restart it. Excluded days are dimmed in the daily trend.

Reduced coverage

On very large traffic spikes adfit stops counting new visitors for a while to protect the measurement. The card then notes that coverage is reduced. Your personalizations are unaffected.

Paused, inactive or switched off

• Pageview limit reached: personalization is paused, so no new data comes in. If the test already has enough data, its final read can still appear. See Pageview usage for when it starts again.
• This site is paused: no new data comes in while the site is paused. If the test already has enough data, its final read can still appear.
• No active plan: no new data comes in while your subscription is inactive. Your tests do not keep running.
• Measurement is off: adfit is not measuring on this site at all. Personalizations keep running.
• A/B testing is off: every matching visitor gets the personalized version, so there is no original to compare against. adfit still counts personalized visitors and conversions.

Reading the result caveats

Caveats can appear in three places on a run: the campaign header of the Performance screen, a personalization's own results card, and a run history row for an earlier run. Wherever they appear, each caveat says something about the numbers shown that you need in order to read the result correctly.

An earlier run showed more visitors the personalized page

The caveat reads: Some visitors in the original group may have seen the personalized page in an earlier run, so this result may look better or worse than it really is.

This caveat can appear when an earlier run of the same campaign, in the 30 days before this run started, showed the personalized page to more visitors, for example a run without an A/B test or one with a smaller holdback. Some visitors placed in the original group of this run may have already seen the personalized page back then, so the original group is not a clean, unexposed baseline, and the result can look better or worse than it really is.

Letting the run continue reduces how much that earlier exposure matters over time; for a page with major changes, restarting the experiment gives you a clean comparison instead.

This caveat follows from the settings of those earlier runs, whether they ran without an A/B test or at a smaller holdback, never from adfit recognizing the same individual visitor across runs. See also the run history notes under Reading a campaign block.

A gap in the data

The caveat reads: This result combines visitors from before and after a gap of several days without data.

A gap is a stretch of at least 7 days in a row without a day that had meaningful traffic, for example while your ads were paused. A result that spans a gap like that is harder to read: the visitors from before and after the gap are combined into the same numbers, even though the traffic and the audience around the gap can differ. See Unusual traffic on some days and the gap marker in the daily trend under Reading a goal card.

Visitors measured without a stored id

The caveat reads: Visitors measured without a stored id may be counted again on a later day in the same group. Where that happens, the confidence and likely range allow for it, using an assumption about how often it does.

This caveat is about visitors, not about sites: it applies to any visitor adfit measures without a stored id, in the cookieless measurement path.

Group membership for those visitors is held as an entry in the browser's own storage. With Variant persistence set to Persistent, that entry lasts up to 30 days, so a visitor who returns on a later day can be counted again in the same group. Where that happens, confidence and the likely range include that, using an assumption about how often it occurs, which is why such a test needs more time than one with Variant persistence set to Session. See Visitors adfit cannot recognize for long.

Results and your pageview allowance

A pageview counts only when adfit decides what a visitor sees: a personalization is applied, or the original is shown to a visitor in an A/B test. All other traffic is free.

Visitors in the original group of a running A/B test are part of that: adfit decided what they see and measured them, so they use your monthly allowance just like personalized visitors do. The page you open through a campaign's test link is the exception on the results side: it is left out of your results, although it still counts as a pageview when a personalization is applied. Pages you click on to from there are normal visits and can be counted in your results.

Results are indicative

adfit gives you a well defined statistical read: a planned sample size, a stated threshold for each read (99.9% for an early read, 95% at the final read), and a final read that later visitors and conversions do not change (see Early read and final read). It is not proof.

• adfit filters bots, caps how many new visitors it counts and drops days that look manipulated, but no measurement on the open web is immune to someone deliberately sending fake traffic.
• Seasonality, your own campaigns and changes elsewhere on your site all move conversion rates. A test measures the difference between two versions in the same period, not why the period looked the way it did.

Related

• Goals: define what counts as a conversion.
• Why some visitors see the original: the reasons behind an unexpected original page.
• Targeting and the test link: who matches a campaign, and how to preview it yourself.