What Token Budgets Mean for the HTML AI Crawlers Read
AI search crawlers read your HTML inside a token limit. Bloated markup burns that budget before they reach your content. Here is what to trim.
AI search crawlers don't read your page the way Googlebot does. They pull your HTML into a model with a fixed context window, and every tag, inline style, and nested wrapper spends tokens before the model reaches a single sentence of what you actually wrote. Googlebot can render, queue, and come back later. A model answering a live query can't. It works inside a budget, and your markup is what spends it.
That changes which pages get used in AI answers. A page that buries its one useful paragraph under 4,000 lines of layout scaffolding is not "lower quality" to a model. It's more expensive to read, and the cheaper source wins the citation.
Why a token budget exists at all
A language model processes text as tokens, roughly three-quarters of a word each. Context windows have grown, but compute still costs money per token on every request, and a retrieval engine fetching live pages mid-answer is paying that cost in real time. So it economizes. It truncates. It prioritizes the part of the document it can reach early and cheaply.
Your HTML competes for that space against navigation, cookie banners, inline SVG, framework hydration data, and tracking snippets. None of that helps the model answer the question. All of it counts against the budget.
The markup that wastes it
Three patterns burn tokens for no informational return.
Deeply nested wrappers come first. A single visible button sitting inside ten div layers serializes as ten opening tags and ten closing tags. Multiply that across a component-heavy page and the structural noise can outweigh the prose.
Then the inline data blobs. Many React and Next.js pages ship a large JSON payload inside the HTML to hydrate client state. To a model parsing the document, that's a wall of tokens that reads like machine output, because it is.
Boilerplate does the quiet damage. The same mega-menu, footer, and legal block on every URL means every page pays the same tax before its unique content begins.
DOM depth and node count have published thresholds
You don't have to guess what "too much" looks like. Lighthouse, the audit engine inside Chrome DevTools and PageSpeed Insights, flags DOM size against specific numbers.
| Signal | Lighthouse warning threshold | Why it matters to a crawler |
|---|---|---|
| Total DOM nodes | over ~1,500 | More nodes, more tokens to serialize |
| DOM depth | over 32 levels | Deep nesting multiplies structural tags |
| Child elements per parent | over 60 | Wide nodes bloat a single branch |
These thresholds were written for rendering performance and memory on low-end devices. They double as a rough proxy for token cost, because the same bloat that slows a phone also pads what a model has to chew through.
What to trim first
The first place to look is the gap between visible content and source weight. On many component-heavy pages the actual article doesn't begin until line 1,800 of the raw HTML, and everything above it spent tokens a model could have used on your answer.
Flatten the wrappers that exist only for styling. Modern CSS with grid and flexbox removes most of the reason for nesting div inside div inside div. Move large hydration payloads out of the initial HTML where your framework allows it. Put the substance, the answer, the comparison, the definition, high in the source order so an engine that reads top-down hits it early.
The goal isn't a minimalist site. It's a high ratio of meaning to markup, so that when a model reads the first slice of your page, that slice is the part you'd want quoted.
Most teams have never looked at their pages this way, which is exactly why there's room to win. The AI Visibility Check renders a page the way a crawler sees it and surfaces the gap between what humans read and what the model can extract. If you want someone to trim the markup and rebuild the source order with you, that is the kind of work I take on in consultancy.