Why a good site from 2019 reads hot for search and cold for AI
The most common shape in the readings is a site scoring in the eighties for Search Visibility and the forties for AI Discoverability. Nothing it did was wrong when it was built. That is the whole problem.
Updated
A site built carefully in 2019 commonly reads in the eighties for Search Visibility and the forties for AI Discoverability, and there is usually nothing wrong with it. Three specific things changed underneath it: AI crawlers arrived, retrieval started reading raw HTML instead of rendered pages, and answer engines began quoting passages rather than ranking documents.
None of those were mistakes at the time. That is what makes this shape so common and so hard to spot from the inside.
The three causes, in order of what they cost
One line in robots.txt
The heaviest single signal in the AI Discoverability reading is whether AI crawlers are allowed, and it is worth 16 of the 75 points available in that category. One line.
In 2023 there was a wave of advice, much of it reasonable at the time, telling site owners to block GPTBot. A lot of people did. Very few of them have looked at that file since, and in the meantime the same directive has quietly come to cover systems that fetch a page live to answer a question somebody is asking right now.
This is not an argument for or against blocking AI crawlers. It is a defensible business decision either way. It is an argument for knowing that you did it, on purpose, recently, rather than inheriting it from an afternoon two years ago.
The version that catches careful people is Google-Extended. It governs training and grounding use. It does not govern whether Googlebot indexes you. People block it believing it is the other one.
Content assembled in the browser
Worth 14 points, and the one that produces the most disbelief.
The test is blunt: at least 250 words of visible text in the raw HTML response. A page that looks full in a browser and returns 40 words to a fetch is not borderline. It is a page most retrieval systems cannot read, because they read the response and never run the script.
Search engines got better at rendering JavaScript, slowly and expensively, over about a decade. Most AI retrievers have not made that investment and several do not intend to. So a decision that became safe for search around 2018 became unsafe again for a different audience around 2023, without anything changing on the site.
This does not mean abandoning a framework. Every serious one has supported server rendering for years. It means the main content has to arrive in the response.
Nothing quotable at the top
Worth 10 points, and the cheapest of the three to fix.
Answer engines quote the first passage that stands on its own. An introduction that warms up for three paragraphs before reaching the point gives them nothing to lift, so the page loses to a worse article that answered immediately.
The measured version: within 1500 characters after the H1, a paragraph of at least 25 words. That is not a demanding bar. Most pages that fail it fail because the first thing after the heading is a breadcrumb, an image, a byline block and a share widget, and the actual writing starts 2000 characters down.
Why this shape is invisible from the inside
Because everything that would normally warn you is fine.
Search traffic is stable or slowly declining, which every site has been able to explain away for three years. The site loads. It looks good. Google Search Console shows nothing alarming. There is no report anywhere that says "you were not in the candidate set for 400 answers this month," because the systems that would know do not publish that and probably never will.
The absence is the whole problem. You cannot see a citation you did not get.
What it adds up to
Those three signals are 40 of the 75 points in the AI reading. A site failing all three lands around the mid forties there regardless of how good everything else is, which is exactly the shape the readings keep producing.
And because AI Discoverability is 30 percent of the overall temperature, a site at 85 for search and 45 for AI comes out around 70. Warm. Fine. Nothing obviously wrong. Which is precisely how it stays that way for another two years.
The order worth doing them in
- Read your robots.txt. Out loud if necessary. Free, takes a minute, and is the highest-weight signal on the list.
- Fetch your own page without a browser.
curlit and count the words. If the number surprises you, that is the finding. - Move the answer up. Editorial work, no deploy required, and it improves the page for humans at the same rate.
All three are checkable in an afternoon, and none of them need anyone's permission. Take a reading first so you have a number to compare against afterwards, because a fix nobody measured is a story rather than a result.
Filed under: ai-visibility, robots, server-rendering