Does an AI crawler visiting your website prove you were cited?

What an AI crawler is, and the jobs it does

An AI crawler is an automated program that requests pages from websites on behalf of an AI company. Each request arrives with a user agent, the name the bot gives itself, and your server records it like any other request. ChatGPT, for example, isn't one crawler. OpenAI's crawler documentation lists GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for actions a user starts, plus OAI-AdsBot, which checks the safety of pages submitted as ads on ChatGPT.

The job matters more than the name. A training crawler collects content that may help train a future model. A search crawler builds the index a product draws on when it answers. A user-triggered fetcher opens a page because someone just asked a question or gave an instruction. Only the last two feed the answers people read today, though OpenAI says that if a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both uses.

Google works differently. Its AI features documentation says a page must be indexed and eligible to show a snippet in Google Search to appear as a supporting link in AI Overviews or AI Mode, so the crawler that matters there is ordinary Googlebot. Google-Extended, the control for Gemini training and for grounding in Gemini Apps and Vertex AI, has no user agent string of its own, so it never appears in a log. Apple's Applebot-Extended is similar: Apple says it doesn't crawl pages, and disallowing it only opts content out of training Apple's generative foundation models.

The table lists the agents a small business site is most likely to see, as each operator describes them on its own crawler page. If a name in your logs isn't here, find the operator's documentation before deciding what it does.

A crawl, a citation and a visit leave different records

Each of the three events happens in a different place, so each leaves its evidence somewhere else, and none can stand in for another. A log line can't tell you what an answer said. An answer can't tell you whether anyone clicked. A visit can't tell you which question sent the person.

Mix them up and you make bad calls: blocking a bot you need, or reporting AI wins that never reached a customer. Keep the three in separate columns of whatever record you build.

The same split explains the difference between a search crawler and an AI training crawler. When Googlebot fetches a page, that page can go on to earn a classic search result and a supporting link in an AI Overview, because both come from Google's index. A crawl made only for training, such as a CCBot visit for Common Crawl's archive, produces no link anyone can click.

How to find AI bots in your own logs

Your access logs are the only first-hand record of which bots asked for which pages. GA4 won't show them: Google's Analytics help says traffic from known bots and spiders is automatically excluded, and that you can't see how much was excluded. So the log file is where this part of the evidence lives. A text editor or spreadsheet is enough for a small site:

1. Find where your logs live. Many hosting control panels offer a raw access log download. On managed website platforms, ask the host whether raw logs are available at all. If your site sits behind a CDN, its dashboard may keep a separate record of requests, including ones it blocked before they reached your server.
2. Download a few weeks at a time, so one busy day doesn't distort the picture, and note the time zone the log uses.
3. Filter by user agent. Search for each name in the table above, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
4. Separate robots.txt checks from page fetches. A request for /robots.txt only shows a bot reading your rules. OpenAI says GPTBot and OAI-SearchBot may add a robots.txt marker to their user agent when fetching that file, to help owners tell those requests apart, especially when logs don't include paths.
5. Read the status code. Apache's logging documentation explains that codes beginning with 2 mean success, 3 a redirect, 4 a client error and 5 a server error. A bot that received a 403 or a 503 never got your content, whatever its name says.
6. Group the requests by page rather than counting them. Knowing that user-triggered fetchers keep opening your emergency service page tells you more than a monthly total.
7. Verify the address of each bot you plan to rely on before you treat the name as true. The next section shows how.

Reading one log line, then checking it's genuine

Here's an illustrative line in the combined log format that Apache documents, with a documentation-range address and an invented date, shortened at the end: 203.0.113.10 - - [14/Sep/2026:09:12:44 +1000] "GET /hot-water-repairs/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0".

Read it left to right. First comes the address that made the request, then two dashes for identity fields that weren't available, then the time with its offset from UTC. The quoted request line shows the method, the page and the protocol. The 200 is the status code, meaning the page was served, and 18342 is the number of bytes sent. The dash in quotes is the referrer, empty here, and the last quoted field is the user agent. What it proves: something calling itself ChatGPT-User received the hot water page at 9:12 that morning. What it doesn't prove: who asked, what they asked, or what ChatGPT said.

Then check the name, because user agents are easy to fake. Common Crawl says it's aware of crawlers falsely identifying themselves as CCBot, and Google's crawler verification guide exists for the same reason. Each operator in the earlier table gives you a way to test a request:

• For Googlebot, Google describes four steps: run a reverse DNS lookup on the address from your log, confirm the name ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that name, and check it returns the original address. Google also publishes its crawler IP ranges as JSON files for automated checks.
• For Applebot, Apple says genuine traffic resolves in reverse DNS to the applebot.apple.com domain.
• For OpenAI, Anthropic, Perplexity and Amazon agents, match the address against the IP list each company publishes. Perplexity recommends combining user agent matching with its lists if you let its bots through a firewall.
• For CCBot, Common Crawl says it uses dedicated IP ranges with reverse DNS, except over IPv6, and publishes the ranges as JSON.
• If a request fails the check, treat it as an unknown bot using a borrowed name. Don't count it as AI interest, and handle it like any other unwanted traffic.

Where a citation shows up, and how a visit gets recorded

A citation lives in the rendered answer: the words, links and source cards a person sees on screen. It also moves. Google says AI Overviews and AI Mode may use different models and techniques, so the responses and links they show will vary. OpenAI says ChatGPT may use an approximate location from your IP address for local results, and saved memories, if memory is on, when rewriting a search. Two people asking the same thing can see different sources.

A missing crawl doesn't prove you're absent from answers either. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers but can still appear as navigational links, and that one crawl may serve both GPTBot and OAI-SearchBot. Your CDN may also be turning bots away before your server sees them. Cloudflare's documentation describes a setting that blocks verified bots classified as AI training crawlers, and says that from 15 September 2026 new domains get updated defaults that block bots classified as Training or Agent on pages that display ads, while Search stays allowed. If your logs show no AI bots at all, check your CDN or host's bot settings before concluding none came.

Some platforms report citations from their side. Microsoft's Bing Webmaster Tools launched an AI Performance report as a public preview in February 2026. It shows how often your pages appear as sources in Microsoft Copilot, AI-generated summaries in Bing and some partner integrations, plus the grounding queries used to fetch them. Microsoft says the figures don't indicate ranking, importance or placement within an answer, and it lists no click data. Google's Search Central blog announced a Generative AI performance report for Search Console on 3 June 2026, and a note on the post says it reached all websites worldwide on 31 August 2026. It counts impressions in features such as AI Overviews and AI Mode by page, country, device and date, with no clicks or queries. The guide on how to interpret Google AI impressions covers what those numbers can show.

These reports count appearances. To see what an answer actually said, and for products with no report of their own, the only record is the answer you capture yourself. One screenshot is an anecdote. To make a manual check count:

1. Write down the exact question, the product and mode, the date and time, where you searched from and whether you were signed in.
2. Capture the full answer, including the sources panel, then open each cited link to confirm which URL it points to.
3. Re-run the same questions on a fixed schedule instead of repeating one until you like the result.
4. Record a mention without a link separately from a linked citation. The guide on AI recommendations versus mentions covers why they differ.
5. For visits, check GA4. OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral links, and Google's Analytics help says utm_source fills the Session source dimension, so those sessions show chatgpt.com as their source. For other assistants, check your Traffic acquisition report rather than assuming.
6. Expect some real visits to lose their label. Google's Analytics help says traffic shows as (direct) / (none) when there's no clear referral source, and names missing UTM tags, redirects that strip them and ad blockers among the causes. Someone who reads an answer and then types your business name into Google arrives as an ordinary Google visit. The guide on how to measure ChatGPT referrals walks through the set-up.

Worked example: a Caloundra plumber reads all three records

Say a plumbing business in Caloundra wants to know whether AI assistants are sending it work. The business and every number below are illustrative, not a real case or a benchmark.

Step one is the logs. The owner downloads four weeks of raw access logs from the host's control panel and filters for AI user agents. Three patterns stand out: ChatGPT-User requests to the hot water repairs page on most weekdays, Claude-SearchBot requests across several service pages, and a run of requests calling themselves GPTBot that hit dozens of odd URLs in a few minutes.

Step two is verification. The ChatGPT-User addresses match OpenAI's published list, and the Claude-SearchBot addresses match Anthropic's. The so-called GPTBot requests come from addresses on no OpenAI list, so the owner discards them as an impostor and blocks that address range at the firewall, which is a security decision, not an AI decision.

Step three is the answers. Each Monday, the owner asks ChatGPT and Claude the same five questions, such as which plumbers in Caloundra repair hot water systems, and records each answer with its sources. Over the month the hot water page is linked in some answers and missing from others, and one answer names the business without linking it.

Step four is visits. GA4 shows a handful of sessions with chatgpt.com as the source, two of which reached the contact page.

Each record supports a modest claim, and together they justify one action: tighten the hot water page's opening answer so it states the service, the area and how to book in plain HTML, then watch whether tagged sessions and enquiries follow. What the owner can't honestly report is that AI is now a lead source, because no enquiry has yet been traced to an AI answer. A small claim you can back up beats a big one you can't. This table is the rule of thumb the owner used:

Before you block or allow an AI crawler

Decide by job, not by company. Letting search and user-triggered bots in makes a page eligible for that product's answers, though no one can guarantee it gets chosen. Blocking the wrong bot can take you out of the running.

The table shows an illustrative robots.txt for a business that wants to appear in AI search answers but keep its content out of model training. Under the Robots Exclusion Protocol, RFC 9309, a crawler follows the group of rules that names it and falls back to the group marked with an asterisk only when none does, so each bot below reads only its own lines. Check each line against the operator's current page before you copy it, because names and controls change.

• Put the file at /robots.txt on the top level of the site. RFC 9309 requires that location, and Anthropic asks site owners to add rules for every subdomain they want to opt out.
• Allow time. OpenAI says its systems can take about 24 hours to adjust after a robots.txt update, Perplexity says changes can take up to 24 hours, and RFC 9309 says crawlers shouldn't use a cached copy for more than 24 hours unless the file is unreachable.
• Remember robots.txt is a request. Google's robots.txt guide says the file can't enforce crawler behaviour. OpenAI says its rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them. Anthropic says its bots honour robots.txt, including Claude-User.
• Prefer robots.txt to IP blocking for bots that honour it. Anthropic warns that blocking its addresses may not work reliably, because that stops it reading your robots.txt file.
• If load is the worry, throttle before you block. Anthropic says ClaudeBot supports the non-standard Crawl-delay rule, while Amazon says its agents don't support crawl-delay, so check each operator's page.
• Help the bots you allow find your priority pages. Google says a sitemap helps search engines discover URLs but doesn't guarantee they'll be crawled and indexed. The IndexNow FAQ says a submission is shared with all participating search engines, including Bing, and doesn't guarantee indexing. Clear internal links from your AI SEO and service pages matter too.
• Keep the facts that matter in the page's HTML. Google's JavaScript guide notes that not all bots can run JavaScript, so services, service areas and contact details that only load through scripts may be missed. That's a technical SEO check worth doing before blaming any AI product.