LLM SEO: What Changes, What Does Not, and How to Test

On 10 July 2026 Google published a page called "Google's Guide to Optimizing for Generative AI Features on Google Search". It is the first document from a party that actually runs a retrieval system that tells you, in a numbered list, which popular LLM SEO tactics do nothing.

None of the pages currently ranking on the first page for llm seo cites it. Several of them recommend, in detail, three of the things it names as unnecessary.

That gap is the reason this article exists. LLM SEO is roughly eighty percent ordinary SEO with a new name attached, ten percent access control that has genuinely changed, and ten percent measurement that nobody had a tool for until this summer. This article separates the three, with the source for every claim and commands you can run against your own domain.

It does not repeat the ChatGPT specifics. Those are in our article on ranking in ChatGPT, which covers OpenAI's four crawlers, server rendering and a prompt sampling method in more depth than fits here. This one is the cross assistant view: Google, Anthropic, Perplexity, OpenAI, and the layer underneath all of them where most failures actually happen.

LLM SEO, defined by the people who run the retrieval

LLM SEO is the practice of making your pages retrievable and quotable by systems that answer a question instead of returning ten links. The mechanism is not a new ranking algorithm. It is retrieval on top of an existing index.

Google states this plainly. Its generative AI features "are rooted in our core Search ranking and quality systems", and the guide names the two techniques involved.

The first is retrieval-augmented generation, which Google also calls grounding: "a technique used to improve the quality, accuracy, and freshness of AI responses by relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index".

The second is query fan-out: "a set of concurrent, related queries generated by the model to request more information and fetch additional relevant search results to address the user's query". Google's own example is a user asking how to fix a lawn full of weeds, which fans out into "best herbicides for lawns", "remove weeds without chemicals" and "how to prevent weeds in lawn".

Two consequences follow from that, and they are the whole difference between LLM SEO and the SEO you already know.

You are competing for a passage, not a page. A grounded answer lifts the part of your page that addresses one fan-out query. The rest of the page is context the model may never surface.

You are competing on queries the user never typed. The fan-out queries are generated by the model. You cannot see them in Search Console, and there is no keyword tool that reports them, because they are not searches anyone performed.

Google's position on the naming is worth quoting because it settles a recurring argument. "From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO." If a vendor tells you AEO and GEO are new disciplines requiring a new subscription, that sentence is the counterargument, published by Google.

The five things Google says you can skip

The guide has a section titled "Mythbusting generative AI search: what you don't need to do". It is short, it is specific, and it contradicts a large fraction of what is currently ranking for this keyword. Here it is, condensed, with the wording that matters kept verbatim.

Read the llms.txt line carefully, because it is more precise than the summaries of it. Google does not say the file is harmful. It says "doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them", and adds that maintaining one for other services is "completely fine".

That is consistent with what we wrote when we published our own llms.txt: it is a curated index that a coding agent pointed at your docs can traverse, not a documented input to any production retrieval pipeline. The section on llms.txt in our ChatGPT article goes through the evidence. Nothing has changed since except that one vendor has now stated its position in writing.

The structured data line deserves the same care. Google says it is not required for generative AI features and immediately adds that "it's a good idea to continue using it as part of your overall SEO strategy, as it helps with being eligible for rich results". Keep your JSON-LD. Stop being sold it as an AI visibility feature. The JSON-LD section of our Next.js SEO article has the implementation, including the escaping most tutorials get wrong.

There is one more line in that guide that reads like it was written about a specific category of tool. Google warns that creating separate content for every variation of how people might search, "for example, by focusing on other queries that people have asked, or fan-out queries", when done primarily to manipulate rankings, "violates Google's scaled content abuse spam policy".

So the fan-out mechanism is real, and building a page per predicted fan-out query is named as spam by the same document that describes the mechanism. Both things are true at once.

Where LLM SEO is genuinely different: access is per crawler now

Here is the part that is not just SEO with a new name. For Google you dealt with one crawler and one directive. Today at least four organisations run distinct crawlers with distinct purposes, and the difference between them decides whether you are readable, trainable, or invisible.

The important distinction inside each vendor is between the bot that builds a search index, the bot that fetches a page live because a user asked something, and the bot that collects training data. They are separate user agents, they obey different rules, and blocking the wrong one removes you from the citation channel while leaving the training channel open, which is usually the exact opposite of what people intend.

Here is the current documented picture, taken from each vendor's own documentation on 15 August 2026.

Two things in that table cost people visibility every week.

OpenAI documents that ChatGPT-User is exempt from your robots.txt: "Because these actions are initiated by a user, robots.txt rules may not apply." Anthropic takes the opposite position and states that its bots, including Claude-User, respect "do not crawl" signals by honoring industry standard directives in robots.txt. Same category of traffic, two different contracts. A single blanket rule cannot express what you want across both.

Google-Extended is not a crawler at all. It is a robots.txt token that controls whether your already crawled content can be used for Gemini training and grounding. Disallowing it does not reduce Googlebot's access, and allowing it does not create a new crawl. People add it to a robots.txt expecting a new bot to appear in their logs, and then conclude the file is broken.

robots.ts when there are nine crawlers instead of one

If you ship Next.js, your robots file is TypeScript, not a text file, which makes this easy to keep correct. Our own robots.ts and sitemap.ts breakdown covers the API shape; here is the multi agent version of it.

// src/app/robots.ts
// Verified against next 16.2.12. The `rules` array accepts multiple entries and
// Next.js emits one User-agent block per entry, in order.
import type { MetadataRoute } from 'next'

const SITE_URL = 'https://example.com'

export default function robots(): MetadataRoute.Robots {
    return {
        rules: [
            // Everyone: the site is open, the private surfaces are not.
            {
                userAgent: '*',
                allow: '/',
                disallow: ['/api/', '/dashboard', '/account'],
            },
            // Retrieval bots. These are the ones that produce citations.
            // Allow them explicitly so a future blanket rule cannot bury them.
            {
                userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'],
                allow: '/',
                disallow: ['/api/', '/dashboard', '/account'],
            },
            // Live user-initiated fetches. Anthropic honours this block.
            // OpenAI documents that ChatGPT-User may ignore robots.txt entirely.
            {
                userAgent: ['Claude-User', 'ChatGPT-User', 'Perplexity-User'],
                allow: '/',
                disallow: ['/api/', '/dashboard', '/account'],
            },
            // Training. This is the line to change if you want out of training
            // corpora while staying eligible for citations. It is a business
            // decision, not a technical one.
            {
                userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended'],
                allow: '/',
            },
        ],
        sitemap: `${SITE_URL}/sitemap.xml`,
        host: SITE_URL,
    }
}

Then confirm the generated file says what you think it says, because a rules array that compiles is not the same as a robots.txt that parses.

curl -s https://example.com/robots.txt

One warning about ordering that costs people their citation channel. Most robots.txt parsers apply the most specific matching user agent group and ignore the others entirely. If you name GPTBot in its own group, that group is the only one GPTBot reads, and any Disallow you wrote under * no longer applies to it. Write each named group as complete, not as a delta on the wildcard group. The code above does that deliberately, which is why the private paths are repeated instead of inherited.

Your CDN can block what your robots.txt allows

This is the failure mode that produces the most confusing symptom in LLM SEO: a robots.txt that allows everything, a page that renders correctly in a browser, and no assistant that has ever seen it.

The reason is that robots.txt is a request, and your edge network is an enforcement point. If the edge returns 403 to a crawler, the crawler never reads your robots.txt, and nothing in your application logs will tell you, because the request never reached your application.

Cloudflare made this concrete on 1 July 2026 by shipping crawler categories and new defaults. It classifies crawler behaviour into three buckets, in its own words: "Search: any behavior that collects or indexes your content" to answer questions later, "Agent: automated behavior that is acting, usually in real time, on a person's behalf", and "Training: a crawler taking your content to train or fine-tune a model".

The defaults change on 15 September 2026. From that date, for new domains onboarding to Cloudflare, "Training and Agent will be blocked by default on the pages that display ads", while "Search will remain allowed by default".

Read what Agent covers before deciding that does not affect you. Agent is the category holding the live fetch a chatbot performs when a user asks about you. Blocking it on ad bearing pages is defensible for a publisher. For a SaaS marketing site it removes exactly the traffic you want.

There is a second trap in the same announcement, and it is the more expensive one. Cloudflare states that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training", because those crawlers combine search with training purposes. A checkbox intended to keep your content out of model training can take out Googlebot. That is a classic SEO catastrophe reached through an AI control panel.

Whatever your CDN is, verify rather than assume. This loop fetches one of your URLs as each documented crawler and reports the status code and how many words came back in the raw HTML, before any JavaScript.

#!/usr/bin/env bash
# crawler-check.sh https://example.com/blog/your-article
URL="$1"
AGENTS=(
  "OAI-SearchBot/1.4; +https://openai.com/searchbot"
  "GPTBot/1.4; +https://openai.com/gptbot"
  "ChatGPT-User/1.0; +https://openai.com/bot"
  "ClaudeBot/1.0; +claudebot@anthropic.com"
  "Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot"
  "Claude-User/1.0; +Claude-User@anthropic.com"
  "PerplexityBot/1.0; +https://perplexity.ai/perplexitybot"
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
)

for ua in "${AGENTS[@]}"; do
  body=$(curl -sS -A "$ua" -w '\n%{http_code}' "$URL")
  code=$(printf '%s' "$body" | tail -n1)
  words=$(printf '%s' "$body" | sed '$d' | sed -e 's/<[^>]*>/ /g' | wc -w)
  printf '%-70s %s  %s words\n' "${ua%%;*}" "$code" "$words"
done

Anything other than 200 is a blocked channel. A 200 with a word count far below your article length is the other failure: the page returned a shell and expects a browser to fill it in, which the rendering section of our ChatGPT article covers, and which the crawlability check in our SaaS SEO guide covers from the application side.

Run it from outside your own network. Your office IP is frequently allowlisted in ways a crawler's is not.

Verifying a crawler is who it says it is

The inverse problem matters as much, and almost nobody checks it. A user agent string is a header. Anyone can send OAI-SearchBot and scrape you, and a log full of forged crawler hits will make you believe your visibility work is landing when nothing is happening.

Every serious vendor publishes IP ranges for exactly this. OpenAI publishes one file per bot, and they are live right now: openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json. Anthropic and Perplexity publish equivalents. Each file carries a creationTime and a list of CIDR prefixes.

curl -s https://openai.com/searchbot.json | head -20

Here is a script that reads your access log, keeps the lines claiming to be an AI crawler, and tells you which ones came from an address OpenAI actually publishes.

// verify-crawlers.js
// Usage: node verify-crawlers.js /var/log/nginx/access.log
// Reads OpenAI's published prefixes and checks the claimed OAI-SearchBot hits.
import { readFileSync } from 'node:fs'
import { isIPv4 } from 'node:net'

const LOG = process.argv[2]
const FEED = 'https://openai.com/searchbot.json'

const toInt = ip => ip.split('.').reduce((acc, part) => acc * 256 + Number(part), 0)

function inPrefix(ip, prefix) {
    const [base, bitsRaw] = prefix.split('/')
    const bits = Number(bitsRaw)
    if (!isIPv4(ip) || !isIPv4(base)) return false
    const mask = bits === 0 ? 0 : (-1 << (32 - bits)) >>> 0
    return (toInt(ip) & mask) === (toInt(base) & mask)
}

const feed = await fetch(FEED).then(r => r.json())
const prefixes = feed.prefixes.map(p => p.ipv4Prefix).filter(Boolean)
console.log(`${prefixes.length} prefixes, published ${feed.creationTime}`)

const counts = { real: 0, forged: 0 }
for (const line of readFileSync(LOG, 'utf8').split('\n')) {
    if (!line.includes('OAI-SearchBot')) continue
    const ip = line.split(' ')[0]
    const ok = prefixes.some(p => inPrefix(ip, p))
    counts[ok ? 'real' : 'forged']++
    if (!ok) console.log(`forged: ${ip}`)
}
console.log(counts)

Two numbers come out of this and both are useful. A high forged count means someone is scraping you behind a borrowed name, which is a rate limiting problem rather than an SEO one. A real count of zero on a page you believe is optimised means the retrieval bot has never fetched it, and no amount of writing will fix that until the access problem is fixed.

Server logs are the leading indicator for all of this work. Citation share moves in weeks. Crawler behaviour moves in days, and it is a fact rather than a sample.

Measuring the Google side: the report that landed in June

Until this summer there was no first party way to see how your pages performed inside AI Overviews or AI Mode. That data was folded into the regular Web search type in Search Console, which meant it existed but could not be isolated.

Google announced Search generative AI performance reports on 3 June 2026, and the rollout has been widening since. The documentation is specific about what you get and what you do not.

What it reports. Impressions, defined as "how many times links to your site were shown to a user in a generative AI feature on Google Search", broken down by page, country, device and date, across two surfaces: AI Overviews and AI Mode.

What it does not report. Clicks, click through rate, and query data. There is no way to see which prompt produced the impression, which is the number everyone wants and nobody has.

Why you might not see it. Google lists three reasons: the rollout has not reached your property, the site has too few impressions in generative AI features, or the site is excluded from Google's generative AI experiences. Search Labs experiments are excluded from the data.

That third reason connects to a requirement in the optimization guide that is easy to miss. To be eligible for display in generative AI features, a page must be indexed and eligible to be shown with a snippet, and "a site must be included in Search generative AI features in Search Console". A nosnippet directive is enough to remove a page from consideration. If you inherited a site with defensive snippet controls from the AI panic of 2024, that is the first thing to check.

Google also attaches a warning to the measurement section that is worth repeating in a market full of dashboards: "Be wary of third-party tools that promise ranking success or claim to use 'internal' Google metrics. No third-party tool has access to our internal ranking or AI systems."

For the assistants, there is no equivalent report and there will not be one, because nobody logs your impressions inside a chat. The defensible method is sampling a frozen prompt panel and recording which domains get cited, which we set out step by step in the measurement section of our ChatGPT article. The short version: freeze the prompts before you start, repeat each one in a clean session, record every domain rather than only your own, and take a baseline before you change anything.

What to write, according to the only party that publishes guidance

Once access and measurement are handled, the remaining work is writing, and here the guidance is less exotic than the category name suggests.

Google's guide puts one instruction above every other in the document: "Creating content that people find unique, compelling, and useful will likely influence your website's presence in generative AI search in the long run more than any of the other suggestions in this guide."

It then defines the failure mode with an example rather than an adjective, which is the most useful paragraph in the whole page. Commodity content is "something like '7 Tips for First-Time Homebuyers'", based on common knowledge that "could originate from anyone". Non-commodity content is "something like 'Why We Waived the Inspection & Saved Money: A Look Inside the Sewer Line'", providing "unique expert or experienced takes that go beyond common knowledge".

Apply that test to your own last ten posts. If the title could have been written by someone who has never used your product, that page is competing on a claim any of a thousand sites can make, and a retrieval system asked to pick a source has no reason to pick yours.

Three practical consequences, in the order they pay off.

Structure so a single section survives being lifted. This is not the chunking Google says you do not need. It is the older discipline of answering a heading's question in its first sentence, so that a passage extracted without its neighbours still makes sense. Google's own advice supports it in plain terms: readers "appreciate it when web pages are organized by paragraphs and sections, along with headings that provide a clear structure".

Publish the thing only you can publish. Your own numbers, your own logs, your own failed experiment. Aggregators cannot restate what has not been stated. This is also the only defence against the argument that AI answers eliminate the need to visit your site: a summary of your data cites your data.

Pick keywords by who ranks, not by volume. A page that ranks for the wrong reader produces traffic and no signups whether the traffic arrives from ten blue links or from an assistant. That check comes before everything above, and it is set out in the keyword section of our SaaS SEO article.

If you are weighing whether to bring in outside help for this, note that Google added a vetting question specifically about it. Its guidance on hiring an SEO now asks whether a provider's advice on AI experiences, "also known as 'AEO' 'GEO' services", is aligned with Google's official guidance on optimizing for generative AI features. We went through the rest of that list, and what agency retainers actually contain, in our breakdown of SaaS SEO agency pricing.

What none of this buys you

No one can guarantee that a specific page appears in a specific AI answer, on Google or in any assistant. The retrieval systems are probabilistic, partially undocumented, and change without notice. Results vary by domain, category and week, and anyone selling you a position in an AI answer is selling something they do not control.

What the work above does buy you is the removal of failure modes you can observe. Either OAI-SearchBot gets a 200 or it does not. Either your CDN blocks the Agent category or it does not. Either the raw HTML contains your article text or it contains a loading state. Either your Search Console property is eligible for generative AI features or a nosnippet is keeping it out.

Those are binary, checkable, and fixable in an afternoon. They are also, in our experience, where most of the losses are. The interesting part of LLM SEO is not a new tactic. It is that a page can be perfect and unreachable, and until this year there was no first party way to find out.

The remaining work, writing something a retrieval system has a reason to prefer over the fifty pages that already say the same thing, is slow and uncertain and has not changed since 2010. The Next.js implementation details are a weekend. The rest is a habit.

FAQ

What is LLM in SEO?

LLM stands for large language model, the class of system behind ChatGPT, Claude, Perplexity and Google's AI Overviews and AI Mode. In an SEO context it refers to optimising for answers those systems generate rather than for a ranked list of links. Mechanically it is retrieval on top of a search index, not a separate ranking algorithm, which is why Google states its generative AI features are "rooted in our core Search ranking and quality systems".

What is the equivalent of SEO for LLM?

There is no separate discipline, according to the only search provider that has published guidance on it. Google's position is that "optimizing for generative AI search is optimizing for the search experience, and thus still SEO". The terms AEO and GEO describe the same work with a narrower focus. The genuinely new parts are per crawler access control and a measurement layer, both covered above, and neither of them is a rewrite of how you produce content.

What is AI SEO called now?

You will see LLM SEO, GEO (generative engine optimization), AEO (answer engine optimization) and LLMO used more or less interchangeably. Google acknowledges AEO and GEO as "terms you may see used" and treats the underlying activity as SEO. Vocabulary choice matters less than checking whether a given piece of advice is supported by something a vendor has published, since most of the disagreement in this field is between things vendors have documented and things consultants have inferred.

Will SEO be replaced by AI?

The retrieval layer has changed, the index has not. Every documented generative feature we could verify draws from a conventional crawled index, which means crawlability, indexation, snippet eligibility and content quality still determine whether you are a candidate. What has plausibly changed is click volume on informational queries, since an answer can satisfy a user without a visit. The rational response is to publish material that cannot be fully restated in a summary, which mostly means original data and first hand experience.

How do I check whether AI crawlers can actually reach my site?

Fetch one of your real URLs with each documented user agent and check both the status code and the word count of the raw HTML, using the shell loop earlier in this article. A non 200 means a blocked channel, usually at your CDN rather than in robots.txt. A 200 with a word count far below the article length means the page needs JavaScript to show its content. Then check your access logs against the vendors' published IP ranges, so you are counting real crawler visits rather than forged user agent strings.

Should I add an llms.txt file to my site?

It will not affect your visibility in Google Search, which states directly that it ignores such files and that maintaining one "will neither harm nor help your site's visibility or rankings". No major vendor has published documentation saying its production retrieval reads third party llms.txt files. It does have one real use: a coding agent pointed at your documentation can use it as an index. Write one if you have docs worth navigating, and keep it out of your visibility plan.