How MentionBeat measures AI visibility

1Collection: real engines, real answers

Every measurement response comes from the real engine's own API, called with your tracked prompt at a realistic sampling temperature. We do not infer AI answers from Google rankings, and we never silently substitute one model for another.

• Grounded vs. parametric answers are measured and reported separately. A brand can dominate a model's parametric memory yet vanish when the engine searches the live web — collapsing the two hides the difference that matters.
• Citations come from each provider's native metadata (Perplexity citation lists, OpenAI URL annotations, Gemini grounding chunks, xAI citations, SERP references) — not regex-guessed from answer text.
• Every call records cost, latency, tokens and the provider-reported model version, so engine-version drift is visible instead of mysterious.

2Sampling: repetition and honesty about variance

LLMs are non-deterministic; a single query is an anecdote, not a measurement.

• Each (prompt × engine) cell runs multiple repetitions (default 5) at realistic temperature — never one-shot snapshots.
• Prompt sets span an intent taxonomy (informational, commercial, comparison, recommendation, problem/solution), personas and locales, then get human approval.
• Interrupted runs resume idempotently; failed calls are retried and error rates are reported on the run.

3Scoring: a calibrated judge, not keyword matching

Responses are scored by an LLM judge that extracts brand mentions, position, sentiment, recommendation strength, cited domains and factual claims. Because judges are also LLMs, we treat the judge itself as an instrument to calibrate:

• Consensus mode: multiple independent judge passes per response with majority voting, recording the agreement fraction.
• Human calibration: judge output is validated against human-labeled gold sets — Cohen's κ, sensitivity/specificity, and Rogan–Gladen bias correction applied to mention rates, so a judge that over- or under-detects doesn't silently skew your trend.
• A deterministic literal-match safety net catches misses, with provenance recorded — you can always see why something was counted.

4Statistics: confidence intervals or it didn't happen

• Headline metrics carry 95% confidence intervals from a cluster bootstrap (2,000 resamples) that resamples prompts, not individual responses — repetitions of the same prompt are correlated, and treating them as independent understates uncertainty (the standard mistake in this category).
• Small slices use Wilson intervals; every rate is shown with its interval.
• Trend deltas are flagged “likely real” only when intervals don't overlap. We'd rather tell you “this wiggle is noise” than sell a dashboard of noise.
• Share of voice: simple mention share and position-weighted (a #1 recommendation counts more than a footnote), de-duplicated per response.
• Demand-weighted and unweighted metrics use the same bootstrap; modeled weights are labeled as estimates.
• A power calculator recommends repetition counts before you spend budget.

5Accuracy intelligence: grounded in your approved facts

We don't just count mentions — we fact-check what engines say about you against your single source of truth (SSOT): a provenance-tracked store of approved product facts, each with its source and human-review status.

• Claims that contradict an approved fact are reported as hallucinations, bound to the specific fact they contradict — directly actionable.
• Claims we can't verify are reported separately (unverifiable ≠ false).
• Cited domains are classified own / competitor / third-party, mapping your off-site corroboration footprint.

6Experiments: measured lift, not vibes

Content changes are evaluated as baseline → treatment experiments: per-metric lift with confidence intervals, significant only when intervals separate. A confound guard flags experiments where the model set, repetitions, profile version, or provider model version changed mid-experiment — those are marked confounded instead of quietly reported as wins.

7Live / modeled honesty

8Known limitations (the part most vendors skip)

• API vs. UI divergence. We query engine APIs; consumer web UIs can route to different configurations. A browser-capture adapter with a published API-vs-UI reconciliation report is planned. Until then: our results measure the engines' API-served answers, stated plainly.
• No consumer panel. We don't have clickstream data of real user prompts; prompt sets are generated + human-curated, and demand weights are estimates unless you supply real volumes.
• Geographic sampling is currently locale-of-prompt, not IP-of-origin; region-parameterized querying is in development (SERP engines already accept country/language parameters).
• SERP-captured engines depend on the SERP provider's rendering; absence of an AI answer for a query is itself recorded as a data point.
• Non-determinism is managed, not eliminated. Repetition + intervals bound it; they don't make an LLM deterministic. Treat point estimates as centers of intervals.