An AI visibility tool or audit tells you how AI answers talk about your brand. Before you buy one, it helps to know how its figures are made. This guide sets out what to check in any offer, what a good answer looks like, how to check it, and the warning signs. It names no other AI visibility tool or audit provider, and ranks no one.
What these offers measure
An AI visibility audit checks what AI answers say about a brand in response to buyer questions about its category, and which sources those answers cite. A tracking tool asks the same kind of questions on a schedule and reports figures over time.
Every figure in either one rests on choices: which questions are asked, which engine answers them, how often each question is run, and how mentions and citations are counted. A tool that makes different choices can report different figures for the same brand. The checks below are about those choices.
Does it name the engines it collects from?
Ask for the name of every engine the offer collects answers from, and which of them are collecting today.
Why it matters. Each engine gives its own answers, so a figure means little until you know which engine it came from. A number of engines is not the same as a list of them.
A good answer names each engine and says which are live and which are planned.
How to check. Look for the engine's name next to each figure in the product, not only on the sales page.
Does it say how it reaches each engine?
An engine can be reached in more than one way: through an official API, through a third-party data provider, or through the app that people use.
Why it matters. Answers collected through an API can differ from the answers someone sees in the consumer chat app. An offer that labels API answers with an app's name is describing something it did not collect.
A good answer says, for each engine, how its answers were collected, and shows that next to the data.
How to check. Ask whether the answers come from an API, a data provider or the app itself. Ask the same for every engine on the list.
Does it say where, and in which language, it asks?
A question can be asked from a set country, in a set language.
Why it matters. If an offer does not say where its questions are asked from, you can't tell whose buyers the answers stand for.
A good answer names the country and the language, and says whether it targets a city or region.
How to check. Compare the country in the offer with the markets you sell in.
Does it show you the questions?
Every figure rests on the questions asked, so you should be able to read them.
Why it matters. A set of questions that name your brand will tend to mention you more than a set that doesn't. Questions that name your brand are worth reading apart from the ones that don't.
A good answer lets you see the questions and choose them, and keeps their wording fixed over time, so comparisons stay like for like. If a question's wording changes, a good tool treats it as a new version rather than joining the old wording and the new.
How to check. Ask for the full list of questions behind one report. Ask whether the questions are a set list. A tool that works from a set list doesn't see what real people type, so be wary of any figure for how many people ask a question that comes with no source.
Does it run each question more than once?
AI answers change from one run to the next. One run of a question is one snapshot.
Why it matters. If each question runs once, you can't tell a real change from the normal variation between runs.
A good answer says how many times each question is run, whether that number is fixed, and what the number of runs can and can't show.
How to check. Ask how many answers sit behind one figure, and work back to the number of runs.
Does it keep mentions and citations apart?
A mention is a brand named in an answer. A citation is a source the answer gives.
Why it matters. Being named and being cited as a source are different results. A single figure that mixes them can't tell you which one moved.
A good answer reports mentions and citations separately and says what counts as your own source. It also keeps the sources an answer cited apart from pages the engine only retrieved.
How to check. Open one answer in the tool and see whether it shows the mention and the cited sources separately.
Does it say how each figure is worked out?
A visibility figure is usually a share: the answers that mention you, out of the answers counted. What goes into "the answers counted" changes the figure. A refusal, for example, can be counted as an answer without a mention, or left out.
Share of voice depends on which competitors are in the set. Add a competitor to the set, or take one out, and the figure can move.
Why it matters. Two figures with the same name can be worked out in different ways, and then they can't be compared.
A good answer gives a written definition of each figure. It says what is counted, what is left out, and whether each question counts equally.
How to check. Ask for the definitions in writing, then check that a figure in the report matches its definition.
Does it keep engines apart?
A score that blends engines into one number can hide a gap. If one engine names you often and another never does, a blended score shows something in between.
Why it matters. You need to know where you are seen and where you are not, and a blended score can't tell you.
A good answer reports each engine on its own.
How to check. Look for a figure per engine, not only an overall score.
Does it show missing answers as unavailable?
Some collections fail, and some answers come back empty.
Why it matters. A figure of 0 says you were not mentioned. A figure with no answers behind it says nothing at all. Showing it as 0 turns missing data into a bad result.
A good answer shows a period with no counted answers as unavailable, never as 0. It also says how empty answers and failed collections are handled.
How to check. Ask what a figure looks like for a period when collection failed.
Does it show how many answers each figure rests on?
A figure that rests on a few answers can move a long way when one answer changes.
Why it matters. Without the sample size, you can't tell a real change from a small sample.
A good answer shows the sample size next to each figure, as a number of answers counted, not a number of questions. It also shows how complete the collection was: the answers counted against the answers planned. It warns you when a change rests on too few answers, and says what that warning does and doesn't measure.
How to check. Ask for the sample size behind one figure, and for how many answers were planned.
Does it say what is live and what is planned?
Why it matters. A planned feature can't help you yet. An offer that describes planned features as available is overclaiming.
A good answer marks each feature as available, in pilot or planned, in words a buyer can check.
How to check. Ask which features a customer can use in the product today, and which are still to come.
If it is an audit, what does it include?
Why it matters. An audit's value is in what it leaves you with. Audits sold under the same name can cover very different work.
A good answer lists what the audit covers, who does the work and what you get at the end.
How to check. Ask who does the work, what you get at the end, and whether a recheck is included.
Does it promise outcomes?
AI answers change from one run to the next, so no tool or audit can decide what they will say.
Why it matters. A promise of citations, placement, rankings, traffic or revenue is a promise nobody can guarantee.
A good answer says what the offer measures and what it leaves to you.
How to check. Read the sales page and the contract for any promise of a result, and ask how it would be measured.
Warning signs
- A number of engines with no names, or no word on which are collecting today.
- Answers labelled with an app's name when they were collected through an API.
- One blended score across engines, with no figure for each.
- Missing answers shown as 0.
- No sample size next to the figures.
- No written definition of each figure.
- A figure for how many people ask a question, with no source.
- Planned features described as available.
- A promise of citations, placement, rankings, traffic or revenue.
- A single score presented as something search engines rank on.
- A claim that no other offer can do the same thing.
Putting the checks to work
Write the checks down as questions, and send them to each offer you are considering. Compare the written answers side by side. An offer that answers every check in writing is easier to hold to its word.