What Is llms.txt? Why Every AI-Ready Website Should Publish It

The next important visitor to your website may never see your homepage.

It may be an AI coding assistant working inside Windsurf or Cursor. It may be a chatbot researching vendors, an automated compliance assessment, a procurement agent, a custom RAG application, or an AI assistant retrieving information for a user.

These systems do not navigate websites the way humans do. They extract information from JavaScript-heavy pages, incomplete navigation, duplicated marketing content, ambiguous links, or hundreds of URLs competing for limited context.

That is the problem llms.txt is designed to solve.

llms.txt is a lightweight Markdown file that gives AI systems a curated map of a website: what the site represents, which pages are authoritative, and where the most useful machine-readable content can be found.

My position: Websites already publish interfaces for humans, search engines, applications, accessibility tools, and APIs. Publishing an interface for AI agents is the logical next layer.

Published: July 25, 2026 · Last verified: July 25, 2026

What is llms.txt?

llms.txt is a proposed machine-readable publishing convention introduced by Jeremy Howard in September 2024. The file is placed at the root of a domain:

https://example.com/llms.txt

It provides:

• A concise description of the website
• Important context about the company, product, or project
• Curated links to authoritative pages
• Descriptions explaining what each page contains
• Optional links that agents can skip when context is limited

The original proposal was created because language models increasingly rely on website information, while complex HTML, navigation, advertising, scripts, and limited context windows make websites inefficient to process. It uses Markdown because the format is readable by humans, language models, parsers, and traditional tools. [1]

A simple implementation looks like this:

# Example Company

> Example Company provides AI vendor-risk assessment and governance tools.

Important context:

- Assessments are based on publicly available evidence.
- Results support, but do not replace, human security and legal review.
- The methodology page is the authoritative source for scoring logic.

## Core Resources

- [Product overview](https://example.com/product.md): Capabilities, intended users, and product limitations.
- [Assessment methodology](https://example.com/methodology.md): Evidence collection, scoring, and confidence methodology.
- [Security](https://example.com/security.md): Security architecture and data-handling practices.
- [Documentation](https://example.com/docs.md): Implementation and integration guidance.

## Optional

- [Company blog](https://example.com/blog.md): Educational articles and company perspectives.

This is not intended to replace the website. It is a machine-readable map of the website.

The argument for llms.txt starts with how AI systems work

An AI product such as ChatGPT is not simply one model reading the entire internet. A modern AI information pipeline contains several separate components:

1. A crawler or search index discovers content.
2. A retrieval system finds possible sources.
3. A ranking layer selects relevant pages or passages.
4. A context-building layer prepares information for the model.
5. The language model generates an answer based on the supplied context.

The language model is probabilistic. The retrieval path does not have to be.

A website owner cannot control every decision the model makes, but the owner can reduce ambiguity about:

• Which pages exist
• Which pages are important
• Which source is canonical
• What each page contains
• Which content is optional
• Where clean Markdown versions can be found

That is the real value of llms.txt.

OpenAI's own documentation separates its search crawler, OAI-SearchBot, from GPTBot and user-triggered agents. OAI-SearchBot surfaces websites in ChatGPT Search, while GPTBot is associated with content that may be used for model development. This illustrates the distinction between the model and the systems that discover and retrieve website information. [2]

llms.txt belongs primarily in that discovery and retrieval layer.

A deterministic map for a probabilistic consumer

The strongest way to understand llms.txt is not as an SEO file. It is a deterministic content interface.

Traditional websites make humans infer structure through navigation bars, dropdown menus, visual cards, footer links, buttons, page hierarchy, branding, and layout. An AI retrieval system may instead encounter:

• Client-side rendering
• Repeated navigation text
• Multiple versions of similar pages
• Weak internal links
• Orphan pages
• Tracking parameters
• Unclear canonical sources
• Large amounts of marketing text
• Important content hidden behind interactive components

When the system cannot confidently determine what exists, it may search repeatedly, guess URLs, retrieve the wrong page, or miss the information entirely.

llms.txt reduces that search space. It says: here is what this website represents, here are the authoritative resources, here is what each resource contains, and here are the pages you can ignore when context is limited.

That is a practical design pattern whether the consumer is ChatGPT, Claude, a coding agent, an internal copilot, or a custom application.

What happened in my AI Vendor Risk Assessment tool

My support for llms.txt comes from building, not from following an SEO trend.

I built an AI Vendor Risk Assessment tool that analyzes company websites and collects evidence related to security, privacy, AI usage, data retention, subprocessors, compliance frameworks, model providers, governance controls, and contractual limitations.

During testing, the retrieval workflow repeatedly failed to find important pages. This occurred even on some of my own websites, where I already knew the pages existed.

The pages were public and accessible. A human could eventually locate them. But some were buried in navigation, weakly linked, or difficult for the scraper to identify as authoritative.

I added those pages to llms.txt and configured the retrieval process to consult the file. The number of missed pages and unnecessary navigation attempts dropped.

That result does not depend on waiting for a search engine to adopt a new ranking rule. It demonstrates an immediate architectural use case: when an AI application is designed to consult llms.txt, the file makes website discovery more deterministic.

The same principle applies to vendor assessments, compliance scanners, support agents, product-research agents, documentation assistants, procurement systems, IDE assistants, and custom RAG applications. A website that provides an authoritative map is easier to consume than one that requires every system to reconstruct its structure independently.

The evidence: agents perform better when they are given a map

In July 2026, Mintlify published an open-source benchmark comparing four ways of serving documentation to AI agents: HTML, plain Markdown, Markdown linking to llms.txt, and Markdown with the complete llms.txt file embedded.

The benchmark covered 2,400 runs across 20 documentation sites using Claude Code and Codex.

When agents received a link to llms.txt, average failed URL requests dropped from 2.23 per task with HTML to 0.11. That represents roughly 90% fewer dead URLs, alongside fewer wasted fetches and reduced token usage.

Answer accuracy remained broadly similar. The improvement was navigation efficiency: agents reached the correct information faster and with less waste. Mintlify replicated the pattern across additional models, where failed requests again fell close to zero. The benchmark and implementation were published openly. [3]

The practical finding: Markdown alone was not enough. The map was the valuable part.

Chrome is already aligning the web toward agents

The strongest signal is not a prediction from an SEO influencer. It is Chrome incorporating agent-readiness into Lighthouse.

Chrome's Agentic Browsing documentation now describes llms.txt as an emerging convention that provides a machine-readable summary of a website for language models and AI agents. Chrome states that without the file, agents may spend more time crawling a website to understand its structure and primary content. Lighthouse now checks whether the file can be retrieved and whether it follows the proposed format. [4]

This matters because Chrome is not treating agent compatibility as a theoretical marketing topic. It is beginning to operationalize it as a website-quality concern.

The broader Agentic Browsing work also evaluates accessibility for agents, layout stability, machine-readable form descriptions, registered WebMCP tools, and agent interaction reliability. The direction is clear: future websites will not only be read by agents — they will increasingly be operated by them.

Major developer platforms are building AI-readable documentation

The convention is being adopted by platforms whose documentation is heavily consumed by coding agents.

Vercel publishes llms.txt and llms-full.txt, teaches developers how to implement both, and describes llms.txt as the selective navigation layer while llms-full.txt provides complete documentation context. Vercel specifically recommends using its machine-readable resources with ChatGPT, Claude, Gemini, Cursor, and Windsurf. [5]

Cloudflare documentation now exposes site-level llms.txt, product-level llms.txt, llms-full.txt, Markdown versions of individual pages, content negotiation for machine consumers, and agent-specific guidance embedded into documentation pages. Cloudflare's pages explicitly direct agents away from context-heavy HTML and toward Markdown and the relevant llms.txt index. [6]

Mintlify automatically generates both files, hosts them at root and .well-known locations, and advertises them using HTTP response headers so agents can discover them without already knowing their paths. [7]

This is what first-mover adoption looks like. It starts with developer documentation, where machine consumption is obvious, and expands into other information-heavy industries.

llms.txt is not only for developer documentation

Documentation is the first clear use case, not the last. The original proposal explicitly identifies possible applications across software projects, businesses, legislation, personal websites, ecommerce, education, and professional profiles. [1]

SaaS platforms

An AI agent can quickly identify product capabilities, pricing, integrations, security documentation, API references, limitations, and changelogs.

Compliance and vendor-risk platforms

A machine-readable map can identify assessment methodologies, framework mappings, security controls, evidence requirements, privacy policies, data-retention rules, subprocessors, and product limitations.

Ecommerce websites

Agents can locate product categories, return policies, warranty information, shipping policies, product specifications, and compatibility information.

Professional and personal websites

An agent can understand professional background, areas of expertise, projects, publications, speaking topics, services, and contact information.

Universities and public institutions

AI systems can locate programs, admission requirements, courses, policies, research, and public resources.

The underlying need is the same: reduce the cost and ambiguity of machine retrieval.

What about sites whose llms.txt files receive no traffic?

A large Ahrefs server-log study found that most published llms.txt files received no requests during the study period. It also found that AI systems generally did not probe blindly for a missing /llms.txt file. When files were fetched, agents often reached them through a link, an index, or an explicit instruction. [8]

That finding strengthens the implementation case rather than eliminating it. The correct lesson is: publishing the file silently is incomplete. Discovery must also be designed.

A sitemap that is never submitted or linked is less useful. An API that nobody knows exists receives no calls. A security policy hidden in an unlinked PDF will not guide an assessment. The same principle applies to llms.txt.

A complete implementation should:

1. Publish /llms.txt.
2. Publish /llms-full.txt where appropriate.
3. Link the files from documentation and relevant HTML pages.
4. Advertise them through HTTP Link headers.
5. Provide Markdown versions of important pages.
6. Reference the file in agent instructions and integration documentation.
7. Configure custom agents and scrapers to check it first.
8. Monitor server logs and downstream retrieval behavior.

The future advantage will not go to the site that merely owns the file. It will go to the site that builds the entire machine-readable content path.

llms.txt versus llms-full.txt

The easiest way to understand the distinction: llms.txt is the map. llms-full.txt is the compiled reference.

Use llms.txt when:

• The website has many pages.
• The agent should retrieve only what it needs.
• Context efficiency matters.
• Content changes frequently.
• Individual Markdown endpoints are available.

Use llms-full.txt when:

• The content set is reasonably contained.
• A user or agent needs a complete reference.
• The file will be attached to an AI project.
• The content supports documentation, compliance, or research.
• One-fetch ingestion is more useful than selective retrieval.

For larger sites, provide both.

The official llms.txt structure

The original proposal defines a specific order: [1]

1. Optional byte-order mark
2. H1 containing the site or project name
3. Blockquote containing a short summary
4. Optional paragraphs or lists providing context
5. H2 sections containing lists of links
6. Optional ## Optional section for secondary resources

Only the H1 is required, but the descriptions and curated sections create most of the practical value.

Enhanced production template

# Company or Project Name

> A concise, factual explanation of what the company, website, or project does.

Canonical domain: https://example.com
Primary language: en-US
Last reviewed: 2026-07-25
Content scope: Public product, technical, security, and policy information.

Important context:

- State what the product does.
- Identify the intended user.
- Explain which page is authoritative for methodology or product claims.
- State relevant limitations.
- Clarify whether the content is informational, contractual, technical, or legal.
- Avoid promotional claims that cannot be independently supported.

## Company and Product

- [Company overview](https://example.com/about.md): Company identity, leadership, purpose, and operating scope.
- [Product overview](https://example.com/product.md): Product capabilities, intended users, and limitations.
- [Use cases](https://example.com/use-cases.md): Supported applications and implementation scenarios.

## Methodology

- [Methodology](https://example.com/methodology.md): How evidence, analysis, and results are produced.
- [Scoring](https://example.com/scoring.md): Scoring rules, confidence levels, and interpretation guidance.
- [Limitations](https://example.com/limitations.md): Known exclusions and conditions affecting results.

## Security and Privacy

- [Security](https://example.com/security.md): Security architecture and control environment.
- [Privacy](https://example.com/privacy.md): Personal-data collection, use, retention, and rights.
- [Subprocessors](https://example.com/subprocessors.md): Third-party service providers and their roles.
- [Data retention](https://example.com/data-retention.md): Retention and deletion practices.

## Technical Documentation

- [Getting started](https://example.com/docs/getting-started.md): Initial setup and onboarding.
- [API reference](https://example.com/docs/api.md): Authentication, endpoints, requests, and errors.
- [Architecture](https://example.com/docs/architecture.md): System components and data flows.
- [Changelog](https://example.com/changelog.md): Current releases and deprecations.

## Commercial and Support

- [Pricing](https://example.com/pricing.md): Current plans and included capabilities.
- [Support](https://example.com/support.md): Support channels and escalation procedures.
- [Contact](https://example.com/contact.md): Official company contact information.

## Optional

- [Blog](https://example.com/blog.md): Educational content and company perspectives.
- [News](https://example.com/news.md): Announcements and media coverage.
- [Case studies](https://example.com/case-studies.md): Examples of reported customer outcomes.

Recommended llms-full.txt structure

llms-full.txt has become a widely implemented companion convention, although the original proposal more specifically describes generated context files named llms-ctx.txt and llms-ctx-full.txt. [1]

A production-quality llms-full.txt should contain cleaned, authoritative content rather than an uncontrolled dump of every page.

# Company Name — Full AI-Readable Reference

> Consolidated public documentation for AI assistants, agents, retrieval systems, and human readers.

Canonical domain: https://example.com
Generated: 2026-07-25T12:00:00-05:00
Content version: 1.0
Language: en-US
Source index: https://example.com/llms.txt
Scope: Public authoritative content only

## Table of Contents

1. Company Overview
2. Product Overview
3. Methodology
4. Security
5. Privacy
6. Compliance
7. Technical Documentation
8. Pricing and Support
9. Limitations
10. Frequently Asked Questions

---

## Company Overview

Source: https://example.com/about
Last updated: 2026-07-20

[Clean Markdown content from the authoritative company page.]

---

## Product Overview

Source: https://example.com/product
Last updated: 2026-07-22

[Complete product description, capabilities, intended users, and exclusions.]

---

## Methodology

Source: https://example.com/methodology
Last updated: 2026-07-24

[Complete methodology, evidence hierarchy, scoring rules, confidence levels,
human-review requirements, and known limitations.]

Exclude from llms-full.txt

Do not include: cookie banners, navigation, repeated footers, tracking parameters, search-results pages, duplicate landing pages, outdated announcements, empty category pages, private customer content, security secrets, unreviewed user-generated content, or unsupported promotional claims.

The goal is not maximum volume. The goal is maximum useful signal per token.

llms.txt should be part of a broader AI SEO and GEO stack

Generative Engine Optimization should not be reduced to one file. A strong AI-readable website includes several coordinated layers:

1. Crawlability

Important pages must return clean server responses, use canonical URLs, and remain accessible to the crawlers and agents you intend to support.

2. Clear entities

The company name, product names, founder identity, categories, locations, and capabilities should remain consistent across the site and external references.

3. Extractable answers

Pages should contain direct definitions, factual statements, tables, FAQs, limitations, and concise summaries that can be accurately quoted.

4. Original evidence

Research, methodologies, benchmarks, frameworks, case studies, and firsthand experience give AI systems a reason to use the site as a source rather than merely summarize it.

5. Third-party authority

Independent references, media coverage, directories, GitHub repositories, reviews, community discussion, and citations strengthen the broader entity record.

6. Structured data

Use appropriate Schema.org markup for articles, organizations, people, software applications, products, FAQs, breadcrumbs, and services.

7. Machine-readable resources

Publish llms.txt, llms-full.txt, Markdown page versions, OpenAPI specifications, RSS feeds, XML sitemaps, structured data, and MCP resources where applicable.

8. Measurement

Monitor AI crawler requests, user-triggered agent requests, access to .md pages, access to llms.txt and llms-full.txt, brand mentions, AI citations, retrieval accuracy, missed-page rates, failed URL requests, and downstream conversions.

SEO makes content discoverable in search. GEO makes content understandable and usable inside generated answers. Agent readiness makes the website navigable and actionable by machines. llms.txt sits at the intersection of all three.

Traffic is no longer limited to search-result clicks

The traditional journey was: search query → search result → website visit.

The emerging journeys look different:

Question in an IDE → documentation retrieval → product adoption

Chatbot request → source retrieval → brand recommendation

Vendor-risk assessment → policy discovery → procurement review

AI agent → product documentation → API implementation

Internal copilot → knowledge retrieval → employee action

Some of these interactions produce website visits. Others produce brand exposure, product inclusion, technical adoption, vendor shortlisting, evidence collection, API usage, agent actions, or later human visits.

Vercel already positions llms-full.txt as context that users can provide to ChatGPT, Claude, Gemini, Cursor, and Windsurf. Cloudflare actively routes machine consumers toward Markdown and site-level documentation indexes. [9]

The website is no longer only a destination. It is becoming a knowledge source inside other interfaces.

The first-mover advantage

The first-mover advantage is not simply having a file before competitors. It is gaining experience with the machine-facing layer of your website.

Early adopters can begin learning which pages agents fail to discover, which descriptions improve retrieval, which file structures reduce errors, which content is frequently requested, whether agents prefer individual pages or complete context, how quickly machine-facing information becomes stale, how AI systems interpret company and product entities, how agent traffic differs from search traffic, how to secure machine-facing instructions, and how to connect static content with APIs and MCP tools.

The implementation cost is low when the files are generated from an existing content source. The strategic upside is asymmetric. If adoption develops slowly, the organization still gains a structured, curated content map usable by its own applications. If adoption accelerates, the organization already understands how to publish, monitor, secure, and optimize information for agents.

Chrome, Vercel, Cloudflare, Mintlify, coding agents, and custom AI applications are already moving in that direction. [4]

How to implement llms.txt correctly

Publish both files

/llms.txt
/llms-full.txt

Provide Markdown pages

/about.md
/product.md
/security.md
/docs/getting-started.md

Advertise the files through HTTP headers

Link: </llms.txt>; rel="llms-txt", </llms-full.txt>; rel="llms-full-txt"
X-Llms-Txt: /llms.txt

Mintlify currently uses this approach to improve file discovery. [7]

Link from visible pages

Add an "AI and Agent Resources" section to documentation footers, API pages, developer resources, security pages, GitHub READMEs, and integration instructions.

Configure your own applications to use it

A custom retrieval workflow should:

1. Check /llms.txt.
2. Parse its sections and descriptions.
3. Identify relevant authoritative pages.
4. Retrieve the required Markdown content.
5. Fall back to a sitemap or conventional crawl when needed.
6. Record failed and successful retrieval paths.

Version-control it

Treat llms.txt as production content. Monitor for unauthorized changes, broken links, stale claims, redirects, unexpected external links, prompt-injection language, and conflicts with human-facing pages.

How to test whether llms.txt is working

Run a controlled comparison.

Baseline

Ask an agent questions requiring information from deeply nested pages without providing llms.txt.

Intervention

Give the same agent access to llms.txt.

Measure

Track relevant-page discovery, failed URLs, number of pages fetched, retrieval time, input tokens, answer completeness, source accuracy, stale information, and human correction required.

Repeat

Use multiple models, multiple question phrasings, and multiple runs. Generative answers vary. Your evaluation method should not depend on one prompt or one successful response.

Frequently asked questions

What is llms.txt?

`llms.txt` is a Markdown file that gives AI agents and retrieval systems a curated overview of a website and links to its most important resources.

Is it llm.txt or llms.txt?

The proposed filename is `llms.txt`, with an "s." The singular `llm.txt` is a common typo and search variation.

What is llms-full.txt?

`llms-full.txt` consolidates the substantive content of multiple pages into one machine-readable file that can be supplied directly to an AI assistant or retrieval system.

Does llms.txt replace sitemap.xml?

No. A sitemap broadly lists indexable URLs. `llms.txt` curates important resources, explains what they contain, and can direct agents toward Markdown versions designed for efficient retrieval.

Does llms.txt replace robots.txt?

No. `robots.txt` communicates access preferences. `llms.txt` communicates context and resource structure.

Is llms.txt useful for AI SEO and GEO?

Yes, as part of a broader machine-readable publishing strategy. It helps agents and retrieval systems understand which content exists and where authoritative information can be found.

Will llms.txt help AI agents navigate a website?

Evidence from a 2,400-run Mintlify benchmark showed that providing agents with a link to `llms.txt` substantially reduced failed URL requests and retrieval waste. [3]

Should a small website create llms.txt?

Yes, when the website contains important product, professional, policy, compliance, or documentation content. A small file can be maintained with minimal effort.

Should every website create llms-full.txt?

Not necessarily. It is most valuable when the authoritative content can be consolidated without creating an excessively large or repetitive file.

How often should the files be updated?

Update them whenever important pages, product capabilities, policies, methodologies, pricing, or canonical URLs change. Automatic generation from the site's source content is preferable.

How do agents discover llms.txt?

Agents may find it through direct links, HTTP headers, documentation instructions, a user-provided URL, custom application logic, or platform integrations. Publishing the file and actively exposing it is more effective than leaving it unlinked.

Can llms.txt contain instructions?

It can contain factual interpretation guidance and context. Avoid manipulative language, unsupported claims, or instructions attempting to override an agent's policies.

Is llms.txt only for developers?

No. It can support ecommerce, education, professional profiles, SaaS products, compliance systems, legislation, public institutions, and any website whose content may be consumed by AI applications.

Conclusion

The web was originally built for humans. Search engines added another audience, and websites responded with robots.txt, XML sitemaps, canonical tags, metadata, and structured data.

AI agents are now becoming another audience. They operate through chatbots, IDEs, custom applications, retrieval systems, compliance tools, procurement workflows, and agentic browsers. They need clean content, clear authority, efficient discovery, and less ambiguity.

llms.txt gives them a map. llms-full.txt gives them a consolidated reference.

The value is not that one file magically changes a probabilistic model. The value is that it makes the information supplied to that model more deliberate.

My own vendor-risk assessment tool found pages more reliably after I introduced a curated llms.txt path. Mintlify's benchmark found that agents reached documentation with far fewer failed requests when given the same kind of map. Chrome now recognizes the convention as part of its Agentic Browsing work. Major developer platforms are actively publishing and teaching machine-readable documentation.

The signals are moving in one direction. Websites are becoming interfaces for machines as well as people. The companies that begin designing that interface now will understand it before everyone else is forced to catch up.

A deterministic map for probabilistic systems is not a gimmick. It is good infrastructure.

Author

Subodh KC is an AI Systems Architect and Governance Expert. Former Fortune 50 AI Strategy CTL. Founder of HAIEC — Holistic AI Ethics & Compliance. 16+ years building production AI systems from startups to global enterprise. His work includes LLMVerify, the HAIEC compliance platform, and an AI Vendor Risk Assessment system designed to discover, analyze, and validate public evidence about AI vendors.

Sources

1. The /llms.txt file — llms-txt
2. Overview of OpenAI Crawlers
3. Docs URL Benchmark: Markdown & llms.txt > HTML — Mintlify
4. llms.txt — Lighthouse — Chrome for Developers
5. Add llms.txt — Vercel Academy
6. Cloudflare Documentation — llms.txt
7. llms.txt — Mintlify
8. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Ahrefs
9. Agent Resources — Vercel
10. Article Schema Markup — Google Search Central