When two tools give the same brand different figures, that doesn't mean one of them is wrong. Each figure rests on choices: which questions are asked, which engine answers them, how many answers are collected and how they are counted.
This post walks through those choices, using Unwoven AI's methodology as the example. The software is being finished for its first pilot customers, and no collection has run for a customer yet, so the parts about it describe how it is designed to measure.
Answers change from run to run
AI answers change from one run to the next. Ask the same question twice and the answer can name different brands or cite different sources. So any figure is worked out from a sample of answers, not read off one fixed answer.
How many answers are collected
A tool has to decide how often to ask each question and how many answers to collect each time.
Unwoven AI's software is designed to run each panel of questions on a schedule, daily or weekly, at a local time. In each run, it is designed to collect three answers per question from Google AI Overviews and one answer per question from OpenAI API (research mode). These numbers are fixed; you don't choose them.
A tool that collects one answer where another collects three is working from a different sample, so its figures can differ.
Engines stay separate
Google AI Overviews and OpenAI API (research mode) are kept apart. Results are never pooled into one blended score.
They are also reached in different ways. Google AI Overviews answers are collected through a third-party data provider, and OpenAI API (research mode) answers through an official API. OpenAI API (research mode) answers are not answers from the consumer chat app.
A tool that blends engines into one score, or collects from an app instead of an API, is measuring something different, so its figures can differ for that reason alone.
How mentions and citations are counted
A mention is a brand named in an answer. A citation is a source the answer gives, and it counts for you when it is a domain you have approved as your own.
The figures are worked out from counted answers: normal answers, refusals and, for Google AI Overviews, searches that showed no AI Overview. Each question counts equally, and each measure is reported separately for Google AI Overviews and OpenAI API (research mode). Two of the measures show how much a counting rule matters:
- Visibility is the share of counted answers that mention your brand.
- Share of voice is your brand's share of the brands named in answers, counting each brand once per answer, among your brand and the competitor set you approve.
Small differences in these rules move the figures. A tool that leaves refusals out of its count can show higher visibility than one that counts them. Share of voice worked out against a different competitor set gives a different figure.
Unavailable, never zero
If a period has no counted answers, the figure shows as unavailable, never as 0.
Refusals, and searches recorded as "no AI Overview", will be counted answers without a mention of your brand, so they lower visibility. Empty answers will be left out of the counted answers, and failed collections will never become answers. Engines are never pooled, so a gap in one can't hide inside another.
Another tool may show such a period differently, and then its figures for that period will differ.
Sample size and completeness
Sample sizes for Google AI Overviews and OpenAI API (research mode) are recorded with their numbers. The sample size, n, is the number of counted answers, not of questions.
Completeness compares what was collected with what was planned:
- Planned answers: questions × scheduled runs × answers per question.
- Completeness: counted answers ÷ planned answers.
For example, 16 questions run daily for 28 days on Google AI Overviews plan 16 × 28 × 3 = 1,344 answers.
When two periods are compared, the change will carry a low-sample warning if either period has fewer than 30 counted answers; this is not a confidence estimate. A figure that rests on a few answers can move a long way when one answer changes.
Why tools differ
Unwoven AI's figures rest on these choices: the questions in each panel, the engine and how it is reached, the country, the answers collected per question and how mentions are counted. A tool that makes different choices can report different figures for the same brand.
The same is true inside Unwoven AI. Its hand-run audit and its software collect answers in different ways, so their figures are not directly comparable.
What to ask when two figures differ
- Were the same questions asked, from the same country?
- Did the answers come from the same engine, reached the same way?
- How many answers sit behind each figure?
- Were refusals and searches with no AI Overview counted the same way?
- Was missing data shown as unavailable or as 0?
- Was share of voice worked out against the same competitors?
The full rules, and their limits at pilot, are in Unwoven AI's methodology.