AI crawler permissions: what that robots.txt line actually controls
AI training and AI answers are different jobs. Read the crawler rule before changing it so you know which use of your content you are allowing.
Updated
A blocked training crawler does not mean a website has disappeared from every AI answer. The name above the rule matters as much as the rule itself.
Start with Google's two controls below, then check your page and read the evidence behind its AI Discoverability score.
Google Search and Google-Extended do different jobs
Our original explanation got this wrong: it said blocking Google-Extended meant opting out of AI Overviews. That is not what Google's documentation says.
| Control | What it covers |
|---|---|
| Googlebot | Crawling for Google Search, including its AI features. |
| Google-Extended | Training and grounding in some other Google AI systems; it is not the access control for AI Overviews in Search. |
Grounding means using source material to support an AI response. It is different from using that material to train an AI system.
Google says a supporting page in AI Overviews or AI Mode must be indexed and eligible for a Search snippet. Meeting those requirements still does not guarantee an appearance. See Google's explanation of AI features in Search.
What the heavy signal means in this checker
The thermometer groups several crawler permissions into one AI Discoverability signal. That makes it a useful flag to investigate, but it cannot tell you that every AI service has stopped reading your site.
| AI crawler permission signal | 16 points |
|---|---|
| Available points inside AI Discoverability | 75 points |
Those are the checker's chosen weights. They are not a measured percentage of lost visitors or AI mentions. The signal registry shows the current checks, while how the reading works explains their limits.
A lower result here means the tested permissions deserve a closer look. It does not prove your content was excluded from every answer engine. An allowed crawler can still fail to reach a page, and a successful fetch does not promise a citation.
Read the crawler name before changing the rule
A robots.txt file can contain separate groups for different crawlers. A rule under one named group is not automatically a rule for every bot.
- Open the site's robots.txt file. Find the group that contains the rule you are reviewing.
- Check that crawler's documented job. Separate collection for training from retrieval for search or an AI answer.
- Compare the rule with your actual choice. An intentional permission decision and a forgotten rule need different responses.
You can allow useful discovery without pretending all AI uses are the same. The point is to make the choice knowingly, then use the reading to check what the tool observed.
For a page-level check, take a reading. For help interpreting the wider site's findings, request a diagnostic.
Corrected October 5, 2026. This article clarifies crawler roles; the checker's scoring weights have not changed.
Filed under: robots, ai-crawlers, weighting