robots.txt
Also known as: robots file, crawler directives
A plain text file at the root of a site declaring which crawlers may request which paths.
robots.txt controls fetching, not indexing. A disallowed URL can still appear in search results if enough other pages link to it, because the rule stopped the crawler reading the page, not knowing about it. To keep something out of an index, use noindex and let the crawler in to see it.
It is also where a sitemap reference belongs, and increasingly where AI crawler permissions are decided.
Two site-wide failures show up constantly. The first is a Disallow: / left behind from a staging environment, which is a switch in the wrong position rather than a decision. The second is a block on AI crawlers added in 2023 during a wave of advice, never revisited, now quietly removing the site from every AI answer while its owner wonders why.
Neither is an argument for or against blocking AI crawlers. It is an argument for knowing you did it.