Why "I asked ChatGPT once" is not a measurement

Here's a scene that plays out weekly in marketing teams everywhere. Someone asks ChatGPT "what's the best expense management tool for startups?" The company shows up. Screenshot, Slack, celebration. Two weeks later someone else asks. The company doesn't show up. Panic, a meeting, a hastily commissioned "AI visibility audit".

Nothing happened between those two moments. Both people observed one draw from a random process and mistook it for a fact. It's the measurement equivalent of flipping a coin once, seeing heads, and announcing the coin always lands heads.

This post is the statistics you need — and honestly, it's not much — to stop doing that. No proofs, just the three ideas that turn anecdotes into measurements: where the randomness comes from, what a confidence interval actually tells you, and how many samples buy how much certainty.

Where the randomness comes from

An LLM doesn't retrieve the answer to your question; it generates an answer, token by token, by sampling from a probability distribution over possible next words. Three separate mechanisms inject variance:

• Sampling temperature. Production assistants don't always pick the single most likely next token — they sample, with a "temperature" that deliberately preserves variety. Your brand might be the third-most-likely name to appear in a recommendation slot: present in many possible answers, absent in many others.
• Retrieval variance. Engines that browse (ChatGPT with search, Gemini, Perplexity) fetch sources at answer time. Which pages come back shifts with index updates, ranking ties, even latency. Different sources in, different brands out.
• Platform drift. Model versions, system prompts and safety layers change without notice. The distribution itself moves under you — which is a reason to sample more, not less, since you need to distinguish drift from noise.

None of this is a flaw to be worked around. It's the nature of the instrument. Sensible measurement doesn't ask "is my brand in the answer?" — it asks "what fraction of possible answers contain my brand?" That fraction — the mention rate — is a real, stable property of the engine at a point in time. Any single run is just one noisy vote on it.

The coin-flip model of AI visibility

Formally, each run of a prompt is a Bernoulli trial: your brand appears (success) with some unknown probability p, or doesn't. Repeat runs are like flips of a weighted coin whose bias you're trying to estimate. This is the same statistical object as a conversion rate or an A/B test — machinery marketers already trust.

The coin analogy makes the core problem vivid. Suppose the truth is that your brand appears in 30% of possible answers. Run the prompt 10 times and simple binomial arithmetic says you'll see it 3 times on average — but seeing it 1 time or 5 times is entirely unremarkable. Two people can run the same honest test on the same day and come back with "10%" and "50%".

Confidence intervals, without the incantations

A 95% confidence interval is the honest error bar around your observed rate: the range of true mention rates that are plausibly consistent with what you saw. Report "31% (95% CI: 25–38%)" and you've said something checkable. Report "31%" alone and you've hidden the uncertainty that decides whether the number means anything.

For proportions, the textbook ±1.96√(p̂(1−p̂)/n) formula (the "Wald interval") misbehaves exactly where AI visibility work lives: small samples and rates near 0% or 100%, where it produces intervals that are too narrow and can even cross below zero. The fix has been known since 1927 — the Wilson score interval¹ — and modern statistical guidance recommends it over Wald for essentially all practical use.²

The intuition: instead of assuming your observed rate is the truth and building an error bar around it, Wilson asks "which true rates would make my observation unsurprising?" A side effect is that it gently pulls small-sample estimates toward 50%, which is exactly the humility a 10-run test deserves. Two examples, computed properly:

• Brand mentioned in 3 of 10 runs: observed 30%, 95% Wilson interval ≈ 11% to 60%. The data is consistent with your brand being a nonentity and with it appearing in most answers. You've learned almost nothing.
• Brand mentioned in 30 of 100 runs: same observed 30%, interval ≈ 21.9% to 39.6%. Now you know something: you're a real but minority presence, and next quarter's number will be comparable.

How many runs buy how much certainty

The table below shows honest 95% Wilson intervals for an observed mention rate of 30% at increasing sample sizes. (A "sample" here is one answer — so 25 prompts × 8 runs = 200 samples for a suite-level rate.)

Two things to internalize from this table. First, the brutal square-root law: going from 100 to 400 samples — four times the cost — only halves the interval. There is no clever trick around this; precision is purchased with samples. Second, the practical sweet spot: for tracking purposes, 150–400 answers per engine per wave gives intervals tight enough (±5–8 points) to see real quarter-over-quarter movement without heroic volume.

Detecting a real change (the question you actually care about)

Tracking programs rarely care about the level; they care about the delta. Did the content overhaul move our mention rate? Here the math gets more demanding, because both the before and after numbers carry noise, and their uncertainties compound.

Standard power analysis for comparing two proportions gives the benchmark: to detect a change from 30% to 40% — a big, campaign-justifying jump — with 95% confidence and 80% power, you need roughly 356 sampled answers in each wave. Smaller true changes need dramatically more: detecting 30%→35% pushes the requirement past 1,300 per wave. If your before-and-after waves were 50 answers each, a "we went from 28% to 36%!" slide is indistinguishable from noise — the intervals overlap almost completely.

Three comparison pitfalls that survive even decent sample sizes:

1. Changing the instrument mid-experiment. If the engine shipped a model update between your waves, you measured the update, not your campaign. Track engine/model versions alongside every wave, and hold out a set of prompts you didn't optimize for as a control.
2. Peeking and stopping. Running samples until the number looks good, then stopping, inflates false positives — the same sin as stopping an A/B test the first day it goes green. Fix wave sizes in advance.
3. Twenty metrics, one lucky winner. Track 20 prompts independently and one will show a "significant" change by chance. Judge the suite-level rate first; treat single-prompt movements as hypotheses, not results.

This is also why measured optimism is warranted: real, detectable effects exist. The GEO study measured visibility lifts of up to ~40% from content changes across ~10,000 queries³ — effects of that size clear the statistical bar comfortably at practical sample volumes. The point of rigor isn't pessimism; it's making sure the wins you report are wins.

Putting it into practice

You don't need a statistician on staff — you need four habits:

• Never report a rate without its sample size and 95% interval
• Fix your prompt suite and wave size before measuring, and keep them constant across waves
• Aim for 150–400 answers per engine per wave; treat anything under ~50 as directional only
• Call a change real only when the before/after intervals separate (or a two-proportion test says so) — and check nothing changed about the engine itself

Do this and something pleasant happens: your AI visibility numbers become boring. They stop swinging 20 points between reports, stakeholders stop panicking over single screenshots, and when a number finally does move outside its interval, everyone knows it means something.

Frequently asked questions