Crawler
Also known as: bot, spider, user agent
An automated client that fetches web pages on behalf of a search engine, an answer engine, an archive or a tool.
Crawlers identify themselves with a user agent string, which is how robots.txt rules are targeted at specific ones. Googlebot and Bingbot serve classic search. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and CCBot serve various parts of the AI ecosystem, and they do not all do the same job.
The distinction that trips people up: some AI-related agents collect training data, some fetch a page live to answer a question being asked right now, and blocking them has completely different consequences. Blocking a training crawler is a defensible business decision. Blocking a live retrieval agent removes you from answers as they are being written.
Google-Extended is the clearest example. It governs training and grounding use, not whether Googlebot can index you, and people block it believing it is the other one.