If you have been keeping an eye on SEO trends lately, you have probably come across the term llms.txt. It is popping up in LinkedIn threads, Reddit SEO debates, and Google's own AI Overviews. If you are trying to figure out what it actually is, and whether it matters for your visibility in tools like ChatGPT, Perplexity, or Google AI Overviews, you are in the right spot.

Here is what you will learn in this post:

  • What an llms.txt file is and where it came from
  • How it compares to robots.txt and sitemap.xml
  • How AI systems actually find and cite content
  • How to create your own llms.txt file, with a real example
  • Where llms.txt sits in your overall AEO priority list
  • What actually moves the needle for AI visibility in 2025 and beyond

Let's dig in.

A quick TL;DR

  • llms.txt is a plain-text Markdown file at your site's root that helps AI systems understand your site and find your most important content.
  • It was proposed by Jeremy Howard of Answer.AI and is not an official web standard, but it has been adopted by 844,000+ websites.
  • It is different from robots.txt (which controls access) and sitemap.xml (which lists all URLs). llms.txt is a curated, AI-friendly content map.
  • llms-full.txt is the companion file that includes full page text, useful for documentation-heavy sites.
  • To create one: list 20 to 50 high-signal URLs in Markdown, keep it under 10KB, and host it at yourdomain.com/llms.txt.
  • For AEO, the priority stack is content quality first, then structured data with JSON-LD, then E-E-A-T, then entity signals, then AI-friendly formatting, then llms.txt.
  • AI-referred traffic is real and growing fast, up 527% year over year in 2025, and converting 42% better in US retail (Adobe, March 2026).
  • llms.txt is low effort, low risk, and worth doing, but it will not move the needle on its own.
  • Companies like Stripe, Vercel, Anthropic, and Cloudflare are already using it, and Chrome Lighthouse now checks for it.
  • The biggest AEO wins still come from genuinely useful, well-structured, well-cited content that AI systems trust enough to quote.

What is an llms.txt file?

An llms.txt file is a plain-text document you place at the root of your website to help AI systems understand what your site is about and where your most important content lives. Think of it as a curated reading list you create for AI models. Instead of forcing a large language model to crawl every page, you hand it a short, organized map and say: here are the pages that actually matter, and here is what this site does.

The file lives at https://yourwebsite.com/llms.txt. It is written in Markdown, a lightweight text format that both humans and machines read easily. A basic file is just a few labeled sections:

  • A top-level # Your Brand Name heading, followed by a one-line > summary of what the site does.
  • A ## Documentation section linking your Getting Started and API Reference pages.
  • A ## Resources section linking Pricing, FAQ, and Case Studies.

Simple, right? It is low-effort, human-readable, and designed to reduce the work AI systems do when figuring out what your site is about.

Where did llms.txt come from?

The concept was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, as an informal community standard, documented at the official llms.txt spec. It is not an official W3C or IETF standard, and no standards body has formally approved it. But it has picked up real-world adoption fast. As of 2025, over 844,000 websites have added an llms.txt file, including Stripe, Cloudflare, Vercel, Mintlify, and even Anthropic, the company that makes Claude.

How is it different from robots.txt and sitemap.xml?

These three files all live in your root directory and help machines understand your site, but they serve very different purposes:

  • robots.txt: tells crawlers which pages they can or cannot access. Used by search engine bots like Googlebot and Bingbot. Plain text, key-value rules.
  • sitemap.xml: lists every URL on your site for indexing. Used by search engines. XML format.
  • llms.txt: provides a curated, high-signal summary for AI systems, agents, and answer engines. Markdown format.
  • ai.txt: controls AI crawler access, similar to robots.txt but for AI bots. Plain text.

The key difference is intent. robots.txt is a gatekeeper. sitemap.xml is a directory that says "here is every page that exists." llms.txt is more like a concierge that says "here is what matters most, and here is enough context to understand us." There is also a related file, llms-full.txt, which we cover below.

Why does this matter? The shift from SEO to AEO

To understand why llms.txt exists, you need to understand a bigger shift happening in search. Traditional SEO was built around one idea: rank higher in Google's blue-link results, get more clicks, drive traffic. But more people are skipping the blue links entirely. Roughly 80% of consumers now rely on zero-click search results in at least 40% of their searches, meaning they get their answer on the results page without clicking through.

And it is not just Google. People ask ChatGPT, Perplexity, Google AI Overviews, and Gemini questions and trust the answers directly. This is where Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) come in. AEO is the practice of optimizing content to be cited and surfaced by answer engines. GEO is about getting your content included in AI-generated responses. For a deeper primer on this shift, Ahrefs has solid ongoing coverage of how AI search is changing organic traffic.

As Carolyn Shelby, Principal SEO at Yoast, put it: "Ranking is no longer the prize. Inclusion is." That is the world llms.txt was designed for. It is not about ranking. It is about making sure AI models know what your site does, trust it enough to reference it, and can find the right pages when they need to.

How AI systems actually find and cite your content

AI systems like ChatGPT or Perplexity do not crawl the web in real time the way Google does. Most of their knowledge comes from training data collected months or years earlier. When they do look at the live web, they use their own AI crawlers:

  • GPTBot, used by OpenAI
  • ClaudeBot, used by Anthropic
  • PerplexityBot, used by Perplexity

These crawlers fetch pages, and that content becomes part of what the AI can reference. Here is what the research shows about what actually gets cited:

  • 86% of AI citations come from brand-controlled or brand-influenced sources (Yext, 2025).
  • Content with direct quotes, original statistics, and structured comparisons can lift source visibility in AI responses by up to 40%.
  • AI-referred traffic converts 42% better than non-AI traffic in US retail (Adobe, March 2026).
  • AI-referred sessions grew 527% year over year in 2025.

So AI visibility is not a theoretical future thing. It is generating real, high-converting traffic right now. The question is how you get into that citation authority pool, and llms.txt is one small piece of a larger puzzle.

What is llms-full.txt?

The llms.txt file is a short, curated index. It points to your key pages but does not include their full text. llms-full.txt is the comprehensive companion: it includes the full Markdown text of all your important pages concatenated into one document. This gives AI systems and agents the full content without crawling each URL.

Think of it this way: llms.txt is a table of contents, and llms-full.txt is the whole book. For most marketing and business sites, llms.txt is enough. For developer tools, APIs, and documentation-heavy platforms where AI agents read deep technical content, llms-full.txt can be more useful, especially for token efficiency within a model's context window.

How to create an llms.txt file, step by step

Good news: this is not complicated. A junior developer or a technically comfortable marketer can set it up in about one hour.

  1. Decide what to include. You are not linking every page. Curate 20 to 50 high-signal links: your homepage and about page, core product or service pages, documentation, pricing, FAQ, your best case studies, API reference if you have one, and key policies.
  2. Write the file in Markdown. Use clear ## sections (About, Products, Documentation, Resources, Policies) with one link per line. Keep the file under 10KB, because AI models have a limited context window and a bloated file might get cut off.
  3. Host it at the right location. It must be reachable at https://yourdomain.com/llms.txt, return an HTTP 200 status, and serve as text/plain. If it returns a 404 or is blocked by robots.txt, crawlers cannot read it.
  4. Verify it is accessible. Visit the URL in your browser. If you see a plain-text Markdown document, you are good. Check your server logs for visits from GPTBot or ClaudeBot to confirm it is being fetched.
  5. Optionally create llms-full.txt. Concatenate the full Markdown of each linked page. This is especially useful for documentation sites and developer tools.

Does llms.txt actually help with AEO?

Here is the honest answer: a little, but it should not be high on your priority list. There is currently no strong public evidence that adding llms.txt significantly improves your citation rate in AI answers. Major LLM providers have not confirmed they use it as a citation-ranking signal, Google has not said it affects AI Overviews, and Perplexity has not made official statements either.

What is true is that several major tools (Anthropic, Mintlify, Zapier, Stripe) have adopted it, and the Chrome Lighthouse "agentic browsing" audit now checks for it, which signals growing infrastructure-level recognition. It is low-risk and low-effort, with no real downside. But if you are choosing between writing a genuinely useful, well-researched blog post and adding llms.txt, write the blog post. As Addy Osmani of Google Cloud AI framed it, the emphasis is on structuring and serving content so that AI agents can actually use it, not just on file infrastructure.

The AEO priority stack: where llms.txt actually fits

If you want to show up in AI answers, here is where to focus, in order of impact.

1. Content quality and topical authority (highest impact)

Answer questions directly. Structure content with the answer first, then the explanation. Build content clusters around your core topics so AI systems recognize you as an authority. AI favors content that gives clear answers near the top, includes original data or firsthand experience, covers a topic comprehensively without bloat, and uses headings that match how people ask questions.

2. Structured data with Schema.org markup

This is one of the highest-leverage technical moves for AEO. JSON-LD Schema.org markup tells machines exactly what your content is by labeling it in a standardized way. The most useful types: FAQPage for FAQ sections, HowTo for step-by-step guides, Article for blog posts, Product for product pages, Organization for brand identity, and BreadcrumbList for navigation. If you are not using JSON-LD yet, start here before you even think about llms.txt.

3. E-E-A-T: experience, expertise, authoritativeness, trustworthiness

Google's E-E-A-T framework was built for human quality raters, but AI systems use similar signals when deciding which sources to trust. In practice: show experience with case studies and test results, attribute technical content to credentialed authors for expertise, build third-party mentions for authoritativeness, and cite every statistic with named sources and years for trustworthiness. As Mike Walrath, CEO of Yext, put it: "When brands control their data, they control their visibility."

4. Entity signals and brand consistency

AI systems organize knowledge around entities: brands, people, products, concepts. Use the same brand name, description, and contact details everywhere, claim your Wikipedia or Wikidata entry if you are a recognized brand, build consistent mentions across Reddit, Quora, and YouTube, and use Organization and Person schema to make your brand machine-readable.

5. AI-friendly content formatting

Use descriptive H2 and H3 headings that mirror real questions, include tables and comparison lists for decisions, keep paragraphs to two to four sentences, use bullet and numbered lists for steps, and write explanatory content in plain language. Always include original statistics with proper source attribution. Google's own guidance on creating helpful content lines up closely with what AI systems reward.

6. AI crawler access (and llms.txt)

This is where llms.txt and crawler permissions live. Make sure your robots.txt does not accidentally block GPTBot, ClaudeBot, or PerplexityBot, your llms.txt is live and under 10KB, your important pages load fast and are not behind login walls or heavy JavaScript, and you review server logs periodically to confirm AI crawlers are fetching your content.

Fan-out queries: the hidden AEO opportunity

Here is something most people overlook. When someone asks an AI a question, the AI often generates multiple sub-queries to gather information. These are called fan-out queries. Ask "What is the best CRM for a bootstrapped SaaS company with a small team?" and the AI might spin up sub-queries like "affordable CRM for startups," "CRM with free tier for small teams," and "CRM reviews for SaaS companies."

Your content strategy should account for these derivative questions, not just the primary keyword. If your content covers the full range of related questions, you are more likely to appear across multiple fan-out queries in the same response. You can discover which ones drive traffic by reviewing Google Search Console and filtering for queries that appear in AI-generated results.

Risks and limitations to know

Let's be honest about what llms.txt cannot do.

  • It does not guarantee anything. AI behavior is non-deterministic. Even if GPTBot reads your file perfectly, the model might still cite a competitor for any given query.
  • It is not an official standard. Unlike robots.txt or sitemap.xml, llms.txt is a community convention with no guarantee that every AI system will look for it.
  • It will not fix weak content. If your pages are thin or poorly sourced, a well-formatted llms.txt will not save them. AI evaluates the pages themselves.
  • Privacy matters. Do not list internal-only pages, staging environments, or admin areas. Everything in the file is publicly visible.
  • Over-optimization risk is low but real. Do not get strategically distracted by a trendy, unproven tactic at the expense of core content.

Who should prioritize llms.txt?

Not every site needs the same level of attention here.

  • SaaS and developer tools: high. AI coding agents and technical users query these constantly.
  • Documentation-heavy sites: high. AI agents need to navigate large doc structures efficiently.
  • APIs and SDKs: high. Agentic browsing tools rely on well-structured API docs.
  • Marketing blogs: medium. Useful but not critical; content quality matters more.
  • E-commerce stores: low to medium. Schema markup and product data matter more.
  • Local businesses: low. Entity consistency and Google Business Profile matter more.
  • Academic publishing: medium to high. AI needs clear paper summaries and structured findings.

Real-world adoption: who is already doing this?

Some of the most developer-respected companies in the world have already shipped llms.txt files, so this is not a fringe experiment. Active files include Stripe (payments), Cloudflare (security and networking), Vercel (frontend deployment), Mintlify (documentation), Zapier (workflow automation), OpenAI, Anthropic, Microsoft Teams SDK, and Mastercard. The common thread is developer-facing products where AI agents and LLM-assisted coding tools regularly reference documentation. For these companies, llms.txt is practical infrastructure, not a marketing experiment.

FAQ

Is llms.txt the same as robots.txt? No. robots.txt tells crawlers which pages they can and cannot access. llms.txt is a curated content guide that helps AI systems understand what your site does and where the most important pages are. They serve different purposes, and you can use both at once.

Does Google use llms.txt for AI Overviews? Google has not officially confirmed that llms.txt influences AI Overviews rankings or citation decisions. Chrome Lighthouse now checks for it in its agentic browsing audit, which suggests growing infrastructure-level recognition, but there is no confirmed ranking signal yet.

How big should an llms.txt file be? Keep it under 10KB. Larger files risk being cut off by AI systems with limited context windows. Aim for 20 to 50 links pointing to your highest-value pages.

Should I use llms.txt or llms-full.txt? For most marketing and business sites, llms.txt is sufficient. If you run a documentation platform, developer tool, or API product where AI agents need full content without crawling individual pages, adding llms-full.txt is worth the effort.

Does llms.txt help with ChatGPT, Perplexity, and Gemini visibility? Possibly, but there is no confirmed evidence from any of these platforms that llms.txt is used as a direct citation signal. Treat it as low-cost infrastructure that may contribute over time, not a guaranteed ranking factor.

How long does it take to create an llms.txt file? A junior developer or technically capable marketer can create and deploy a clean llms.txt file in about one hour.