How to compare AI visibility tracking reports

Why two AI visibility reports disagree

Every AI visibility report is a sample. A person or a tracking tool asks a set of questions in a few AI products and counts how often your business turns up in the answers. Change the questions, the products, where or when they're asked, what the count is divided by or how failed runs are handled, and the score moves. Your business might not have changed at all.

The answers move on their own, too. Microsoft describes AI-generated answers as dynamic and contextual, often built from many sources at once. It says citation patterns can shift with changes in user behaviour, evolving models, freshness signals, partner refresh cycles and broader changes across the web. So a gap between two reports might come from the method or from that drift as much as from anything your business did. You can't tell which until the methods line up.

Reports then print their own labels on top: a visibility score, share of voice, sentiment, average position. Every one of those is built from the same raw runs, so check the settings first and read the labels second. If you haven't started tracking, the AI visibility checklist sets up a dated test log. If you already hold reports, get an answer to each of these questions from every provider before you compare anything:

Prompts: the same questions, word for word

The prompt list decides everything downstream, so ask for the exact wording of every prompt, not a topic label. A topic called "emergency plumber" could be "who can fix a burst pipe in Caloundra tonight" in one tool and "24 hour plumber Sunshine Coast" in another. Different wording can surface different businesses.

Be careful with prompt volumes. Google's announcement of the Search Console generative AI reports lists impressions, pages, countries, devices and dates, with no prompts. Bing Webmaster Tools shows grounding queries, the phrases its AI used to fetch content it cited, and Microsoft says that data is a sample. Neither shows how many people typed a prompt, so any prompt volume in a report is the vendor's estimate. Ask how it was made.

Watch for keyword-level reporting. Some tools start from a keyword, expand it into a set of generated prompts and print one score for the keyword. That score hides the wording, so two keyword-level reports can't be compared prompt by prompt unless both providers export the prompts behind each keyword. Ask for that export before anything else.

Then check these things about the list:

• Branded or unbranded. A prompt naming your business tests what AI says about you. One that doesn't tests whether you're named at all. Keep them in separate columns, since branded prompts flatter a score.
• Where the prompts came from. A list built from real customer enquiries, one generated from keywords and one picked from a vendor's prompt library can't be compared line for line. Ask which it was.
• Runs per prompt. One run is one draw. No official source sets a number, so pick a count, keep it fixed every period and ask whether you can see every run or only a summary. If results swing between runs of the same prompt, add runs before drawing conclusions.
• Changes between periods. If prompts were added, dropped or reworded, compare only the prompts present in both periods. Ask for a dated change log.
• Plan limits. Entry plans on many tools cap the number of tracked prompts, and some include only certain AI products, with others sold as add-ons. A plan change between months can drop prompts or products without anyone rewording a thing, so ask whether the plan changed during the period.

Products and collection: which AI, and which way in

A score labelled "AI visibility" is a blend unless the report splits it. Ask for results by product and mode. Google AI Overviews, AI Mode, ChatGPT, Perplexity and Copilot each pick sources their own way, as how AI visibility differs by product explains. Even within Google, AI Mode and AI Overviews can show different responses and links from each other, since Google says the two may use different models and techniques. First-party reports blend too. Microsoft says its AI Performance report in Bing Webmaster Tools aggregates citations across Copilot, AI-generated summaries in Bing and select partner integrations.

Next, ask how each answer was collected. Reports fall into three broad collection methods, live app or browser capture, the developer API and a stored prompt or crawl index, and each measures something different.

The developer API route needs the closest reading. OpenAI's API documentation lets a developer set an approximate user location, choose how much search context the model receives and limit results to chosen domains. Its Chat Completions search models always search before responding, while in the Responses API web search is a tool the model can call when it chooses. Those are the tool builder's settings, so an API report and a report taken from the ChatGPT app measure different things, and two API reports built with different settings measure different things again.

Products also change underneath a report. OpenAI's API documentation lists its preview search models as shut down on 23 July 2026, so any tracker built on them had to switch models. When a provider changes model, product or access route, mark the date and treat the months either side as separate series.

Geography and timing: where and when the question was asked

Local answers depend on location, so a report run from the wrong place measures the wrong market. Google's Search help says Google always estimates the general area you're searching from, and may use areas you've searched for before. OpenAI's help centre says ChatGPT may use an approximate location based on your IP address, that a VPN or network can change it, and that saved memories can shape the search it runs.

Tracking tools usually run on servers, so location is whatever the tool was told. OpenAI's API accepts a country, city, region and timezone. Perplexity's API accepts a location object that can include a city and coordinates. Ask what each report set for every run. A run with no location, or with the country only, can't tell a Maroochydore electrician how it shows up for someone in Maroochydore.

Match the geography before you compare. If one report ran from Brisbane and the other from Sydney, a lower score in the second may just reflect a different market. A business that wants work in several areas should set each area as its own run location, group results by area and keep the same areas every month. For Google, the Search Console generative AI report breaks impressions down by country, an Australia-only baseline from real searches. Its counting rules are in how to interpret Google AI impressions.

Time matters as much as place. A one-day snapshot, a weekly run and a month of daily runs are three different samples. Line up the windows before comparing: the same start and end dates and the same number of run days. If one provider runs weekly and the other daily, compare monthly totals of raw counts, not an average of daily scores against a single weekly reading.

Google's own report shows how dates can slip. Search Console's generative AI report can show data by hour, day, week or month, displays dates in Pacific Time and marks its newest data as preliminary, because it can still change over the next few hours. A tracker running on Queensland time won't share its day boundaries with Search Console, so leave the latest days out until they settle and compare whole weeks or months. Microsoft's Compare view in Bing Webmaster Tools overlays a previous period on the current one, for example the current 30 days against the prior 30. Use the same pair of periods in every report you set beside it.

The denominator and the metrics: a share of what?

Every percentage divides one count by another. Two reports can hold identical raw results and still print different scores, because they divide by different bases. Ask for the numerator and denominator as plain counts, and work out the rate yourself.

Microsoft's Citation Share in Bing Webmaster Tools shows what a stated base looks like. Microsoft defines it as the percentage of citations attributed to your site out of all citations shown across all sites for the same grounding query, and calls it an observational metric rather than a ranking or a competitive scoreboard.

Share of voice moves whenever the competitor list moves. Add a strong competitor and your share falls with no change in your own results. Drop one and it rises. Ask whether the list was entered by hand or found by the tool, and get a dated record of every change. Treat sentiment as the least comparable column of all: each tool classifies tone under its own rules, often with a model, so the same answer can be labelled neutral in one report and negative in another. Compare the passages behind the labels, not the labels.

Check what each report counts as a hit, too. A mention, a linked citation and a recommendation are different events, as AI recommendations versus mentions explains. There are layers even inside one answer. OpenAI's API documentation says its sources field lists every URL the model consulted, and that the number of sources is often greater than the number of citations. A report counting consulted sources will print a bigger figure than one counting visible citations, from exactly the same runs. Then ask how hits were detected:

• Text match: searching saved answers for your business name. Fast, but it can count a different business with a similar name and miss a misspelt one.
• Link match: counting answers whose citations include your domain. Precise for citations, blind to mentions without a link.
• Model-judged: a second AI reads each answer and decides whether you were mentioned, recommended or criticised. Ask what instructions it follows and how often a person checks its calls.
• Human-checked: a person confirms each hit against the saved answer. The slowest method and the easiest to audit.

Failed runs: the rows that quietly vanish

Some runs never produce a usable answer. OpenAI's API documentation describes rate limits as restrictions on how many times a client can use its services in a set period, and its error guidance covers temporary model overload. ChatGPT searches are subject to plan usage limits. Runs also time out, get blocked or come back empty.

Some apparent failures are real results. Google says AI Overviews show only when its systems judge them additive to classic Search, and often don't trigger. ChatGPT may search the web automatically when a question would benefit from current information, so some answers arrive with no sources at all. Both are findings. Log each as its own outcome instead of deleting it.

Ask for the evidence behind every row, too: the saved answer text or a screenshot for each run, with its date, product, location and outcome. A disputed mention or a sudden drop can only be checked against the answer itself, and a report that keeps no answers can't be audited.

If a report can't tell you how many runs it attempted, you can't compare it with anything. When it can, ask which of these it did with the failures:

• Dropped them from the base, so a month full of errors can look better than it was.
• Counted them as misses, dragging the score down though nothing was measured.
• Retried them. Fine, if the report says how many retries ran and which attempt it kept.
• Reported them separately, with attempted, completed, failed and no-answer counts beside the score. The clearest option, since you can check the arithmetic.

Worked example: two reports for a Noosa pest control business

Say, hypothetically, a pest control business in Noosa receives two reports for the same month. Report A comes from a paid tracking tool and says "AI visibility 20%". Report B is a free one-off check and says "visibility score 48 out of 100". The owner wants to know which one is right. Every number in this example is illustrative, not a benchmark.

The owner gets each method in writing. Report A ran 20 unbranded prompts three times each across the month, in ChatGPT through the developer API with the location set to Noosa Heads, and detected hits by text match. Report B ran 15 prompts once each on a single day in the ChatGPT app from a Brisbane office network, and five of its prompts named the business. Its score blends mentions, position and sentiment with weights the provider doesn't publish.

Filled in, the settings table at the top of this page shows a mismatch on every row. Even the products row differs: both reports used ChatGPT, but Report A reached it through the developer API and Report B through the ChatGPT app. So the headline numbers can't be compared at all. Report B's 48 is set aside, because without its component counts and weights there is nothing to recalculate.

Report A's log shows 63 attempts: three timed out, each was retried once and the retry was kept, leaving 60 completed runs. The business is named in 12 runs, spread across 8 of the 20 prompts. At least one business was named in 40 of the 60 runs. Report A's 20% is the first row of the base table here. The other rows are the same results on different bases.

Ten unbranded prompts appear in both reports. Report B kept its answer screenshots, so the owner counts by hand: named for 3 of those 10 prompts in B's single run, against at least once for 5 of the 10 across A's three runs. Location could not be matched, since Report B ran from Brisbane and Report A from Noosa Heads, so this 10-prompt comparison is indicative only. Several differences could explain a gap that size: the run location, one run against three, the ChatGPT app against the developer API, and a single day against a month. Neither report shows the business losing ground.

If more than one setting differs and a provider can't supply raw rows, don't compare the headline numbers. Show them side by side with the mismatch noted, and raise it in your monthly SEO report review. To test whether a change to your site moves these numbers, set it up as a local visibility experiment. The wider work these reports are meant to track sits under AI SEO. For any pair of reports, the general routine runs in this order:

1. Get each provider's method in writing, covering every setting in the table at the top of this page.
2. Mark every mismatch, and set aside any score that has no raw counts behind it.
3. Cut both reports down to what they share: the same prompts, product and location.
4. Recalculate both on one base from the raw counts, with failed and retried runs on their own line.
5. Compare periods only when the method held steady. Where a model, prompt list or location changed, start a new series from that date.
6. Check the direction against first-party data: Search Console for Google's AI features, Bing Webmaster Tools for Copilot, and referrals plus enquiry records for leads, as how to measure ChatGPT referrals sets out.