By Mehdi HassanIs llms.txt worth it?
For most websites, no. An llms.txt file won't hurt, but there's no evidence it helps you get cited by AI search engines. Google's AI features documentation and its May 2026 guidance list llms.txt among the things you don't need, and Ahrefs found that 97% of llms.txt files got zero requests. A file nobody fetches can't influence anything.
That's a clear verdict on a topic where a lot of AEO vendors still sell the file as a quick win. If you want to know what does affect AI citations, our hub on how to structure content for AI search covers the habits with evidence behind them. This article covers what llms.txt is, what the data says, and what actually controls AI access to your site.
What is an llms.txt file?
An llms.txt file is a Markdown file placed at the root of a website (yoursite.com/llms.txt) that gives language models a short summary of the site and a curated list of links to its most important pages. It was proposed in 2024 as a way to help models find clean, relevant content without parsing navigation, ads and scripts.
A typical file has a title, a one-paragraph description of the company or project, and sections of links with one-line descriptions. Some sites also publish a longer companion file with full page contents. The idea is reasonable. The problem is adoption: a proposal only matters if the systems you care about read it, and the major AI search crawlers haven't said they do.
File | What it does | Who respects it |
|---|---|---|
robots.txt | Allows or blocks crawlers by user agent and path | Search engines and the major AI crawlers, by their own documentation |
sitemap.xml | Lists URLs you want discovered and their update dates | Search engines |
llms.txt | Suggests a summary and key links for language models | No major AI search engine has committed to it |
What does the data say about llms.txt?
The data says almost nothing requests llms.txt. Ahrefs found that 97% of llms.txt files got zero requests. The practical reading: whatever leads crawlers to your content today, it isn't this file. They rely on robots.txt, sitemaps, links and the pages themselves.
Google is explicit. Its documentation for AI Overviews and AI Mode says you don't need special files or markup to appear, and its May 2026 guide lists llms.txt alongside content chunking and AI-specific schema as unnecessary. Other engines haven't published a commitment to use the file for search either. Until one does, llms.txt is a note left on a door nobody uses.
What actually controls whether AI engines can read your site?
Three things control AI access: your robots.txt rules, whether your content is in the HTML the crawler receives, and whether the page is reachable through links and sitemaps. llms.txt sits outside all three. If your robots.txt blocks a search crawler, no llms.txt will get you cited by that engine.
- robots.txt per user agent: AI companies publish crawler names and what each one is for. OpenAI separates OAI-SearchBot (search) from GPTBot (training), and Google's Google-Extended token controls Gemini training use without affecting Google Search. Check each provider's documentation for current names.
- Server-rendered HTML: many AI crawlers don't run JavaScript. Content that only appears after scripts run may be invisible to them.
- Discoverability: pages linked from your navigation and listed in your sitemap get found. Orphan pages often don't.
- Clear content: once a crawler can read the page, what gets cited is a direct, specific answer near the top.
The training-versus-search distinction is where most mistakes happen. Teams block every AI user agent to stop training, and accidentally remove themselves from AI search citations too. Decide separately: do you want to be used for training, and do you want to be cited in answers? Most startups selling to buyers who research in ChatGPT want the second, whatever they decide about the first.
A sensible robots.txt approach for a startup marketing site
Start from the question of what you want, not from a template.
If you want to be found and cited: allow Googlebot and Bingbot, and allow the search-specific crawlers of the AI engines your buyers use, such as OpenAI's OAI-SearchBot and PerplexityBot.
If you don't want your content used for model training: disallow the training crawlers by name where providers offer a separate one, such as GPTBot, and use Google-Extended to opt out of Gemini training. This doesn't remove you from Google Search or AI Overviews.
Then check the result. Fetch your robots.txt, test a few URLs in Google Search Console's tools, and look at your logs a week later to confirm which bots are visiting. Revisit it every few months, because crawler names and purposes change.
When might an llms.txt file be useful?
An llms.txt file is plausibly useful for developer documentation, where coding assistants and AI tools are pointed at docs directly by users or integrations. In that case, a clean Markdown index of your API reference and guides can save a tool from parsing a heavy docs site. Several documentation platforms generate one automatically.
Pros
- Cheap to create and maintain if your platform generates it
- Can help developer tools that are explicitly given your docs URL
- Forces a useful exercise: deciding which pages matter most
Cons
- No major AI search engine has committed to reading it
- Ahrefs found 97% of files got zero requests
- Easy to let go stale, pointing to old or removed pages
- Distracts teams from fixes that do affect citations
For a startup marketing site of a dozen pages, that exercise of choosing your most important pages is better spent on internal links and the homepage. Our guide to entity SEO for startups covers how to make engines understand who you are, which is the problem llms.txt is often bought to solve.
How do you add an llms.txt if you decide to?
If you decide to add one, keep it small, accurate and generated where possible. It should take less than an hour, and it should never be the main item on an AEO plan.
Fix robots.txt and rendering first
Confirm the crawlers you want are allowed and your key pages are server-rendered.
Write a two-sentence description
What the company does and for whom, matching the wording on your homepage.
List your key pages
Homepage, pricing, main service or product pages, docs and a few cornerstone articles, each with a one-line description.
Generate it if you can
In Next.js, serve it from a route that reads from your CMS, so it updates when pages change. Webflow and Framer users can upload or host a static file.
Put a review date on it
Check it whenever you change pricing, rename a product or restructure the site, so it never points at stale pages.
What should you do instead of llms.txt?
Instead of llms.txt, put the effort into the things AI engines demonstrably use: clear, server-rendered pages that answer buyer questions directly, a well-configured robots.txt, fresh content and mentions of your brand on other sites. Those overlap almost entirely with good SEO, which is Google's point.
If you're sorting real AEO advice from noise, our breakdown of SEO vs AEO vs GEO explains what's actually different. For the same evidence-based treatment of structured data, read does schema help AI citations, and for the question behind all of it, see is SEO dead.
