What is llms.txt and how do you write one?

What is llms.txt?

llms.txt is a plain-text map that tells a machine, in one file, what a site is and what each of its pages covers. Jeremy Howard published the proposal at llmstxt.org in September 2024; the specification reached its second version in August 2026. The file can sit at the root or under a subpath — /docs/llms.txt, for instance.

The name recalls robots.txt, which is why the two get confused, but the function is entirely different. robots.txt is an access file: it says which crawler may fetch which path. llms.txt is a description file; it grants nobody anything, it only describes content. Neither replaces the other — they sit side by side.

If robots.txt blocks GPTBot, publishing an llms.txt changes nothing. The crawler cannot fetch the file; the map stays behind the locked door.

What problem does llms.txt solve?

When a language model tries to read a site, most of the HTML in front of it is not content: navigation, cookie notices, scripts and stylesheets. The model has to strip all of that out and spends part of its context window doing it. On a typical corporate home page the readable text is a few per cent of the bytes downloaded.

llms.txt proposes that the site's authors do that stripping in advance. A single plain-text file at the root states what the site is in one paragraph, then lists the pages that matter under headings. The model's job drops from "what is in this pile of HTML?" to "which page on this list is the one I need?".

Where the file clearly earns its place today is documentation sites, not marketing sites. If a developer tool can pull one file from a known address instead of searching for a library's install page, the work genuinely gets faster. On a corporate site the gain is more indirect, and the section below on whether anyone actually reads it says so plainly.

How do you write an llms.txt file?

The format is Markdown, and the specification fixes the order of the sections. That order is not a matter of taste — it is the structure whatever parses the file expects to find.

1. 01

   One H1: the name of the project or site

   This is the only section the specification requires. One line, one heading, the company's full name — not a tagline.

2. 02

   A blockquote summary

   The line starts with >. It must carry whatever is needed to make sense of the rest of the file: what is sold, to whom, where. Most circulating examples set this line in italics; the specification says blockquote.

3. 03

   Free-form sections without headings (optional)

   After the summary you may add paragraphs or lists carrying no headings: if you do not publish prices, if measurements are in a particular unit, if orders only go through dealers — this is where that is stated.

4. 04

   Link lists under H2 headings

   Each line is a Markdown link, then a colon and a short note on what that page covers. The 'Optional' heading is reserved by convention for secondary material: links an agent can skip when it needs a shorter context.

# Kuzey Furniture

> An Istanbul-based manufacturer of office and home furniture. The catalogue
> and technical dimensions are published on this site; sales go through
> dealers only.

No price list is published. All product dimensions are in millimetres.

## Products

- [Office chairs](https://example.com/products/office-chairs): Dimensions, fabrics and warranty for 14 models
- [Meeting tables](https://example.com/products/meeting-tables): Modular table systems, size table for 6–24 people
- [Storage](https://example.com/products/storage): Cabinet and pedestal ranges, locking options

## Company

- [About](https://example.com/about): Founding year, production facility and capacity
- [Dealership](https://example.com/dealership): Application terms and territory list
- [Contact](https://example.com/contact): Address, phone and opening hours

## Optional

- [Press archive](https://example.com/press): Published releases
- [Careers](https://example.com/careers): Open positions

Not every page belongs on the list. The measure is: the pages a model needs in order to understand the site. Writing three hundred lines for a three-hundred-product catalogue turns the map into a sitemap and removes the file's reason to exist; list the category pages and let the products sit beneath them.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a map: it hands over links, not text. llms-full.txt concatenates the site's entire text into one file so a model can read it without following a single link. The distinction that matters: llms-full.txt does not appear in the specification. It is a practice that spread, not a standard, and it should be understood as such.

In practice size decides. On a twenty-page corporate site concatenating the full text is sensible and the file stays a few hundred kilobytes. On an e-commerce site with thousands of products the same file runs to megabytes; nobody fetches it, and fetching it would not help. There, writing llms.txt and skipping llms-full.txt is the right call.

The specification also recommends a less familiar route: publishing a clean Markdown version of each page — page.html.md, or the extension swapped for .md. It is cheaper than generating one enormous file and considerably easier to keep current.

How should the file be served, and how do I verify it?

Writing the file is the easy half. Getting the server to hand it over with the right headers is the step most often skipped, and a misconfigured llms.txt looks, to a crawler, exactly like no file at all. One command verifies it:

$ curl -sIL https://example.com/llms.txt

HTTP/2 200
content-type: text/plain; charset=utf-8
content-length: 1483

# Responses you do not want to see:
#   301/302  -> a redirect; the file is not at the root
#   404      -> not published, or the path is wrong
#   text/html-> the server is handing it over as a page
#   content-disposition: attachment -> crawlers cannot read the contents

• The response code must be 200; a 30x redirect chain makes the file invisible to some clients.
• The content type must be text/plain with charset=utf-8, or non-ASCII characters break.
• There must be no content-disposition: attachment header — it marks the file as something to download rather than read.
• The file must be UTF-8. The specification permits a byte-order mark, but on files produced under Windows the BOM commonly attaches to the first heading and corrupts the '# ' line.
• It must not sit behind authentication or bot protection: if curl cannot read it without logging in, no crawler can either.

How do you add llms.txt on Next.js, WordPress and Shopify?

Writing the file by hand and dropping it on the server is the shortest route and the most fragile one: when the site changes the file stays where it was and starts lying within a few months. On all three platforms the right move is to generate it from the content.

export const dynamic = "force-static";

export function GET() {
  const pages = [
    { url: "/products", name: "Products", note: "Catalogue and dimension tables" },
    { url: "/dealership", name: "Dealership", note: "Application terms and territories" },
  ];

  const body = [
    "# Kuzey Furniture",
    "",
    "> An Istanbul-based maker of office and home furniture.",
    "",
    "## Pages",
    "",
    ...pages.map((p) => `- [${p.name}](https://example.com${p.url}): ${p.note}`),
    "",
  ].join("\n");

  return new Response(body, {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
}

WordPress gives you two routes. Dropping a physical file in the root works but disappears at the next theme or server migration; the durable answer is a plugin or a functions.php hook that catches the /llms.txt path and builds the response from published content. If you do use a physical file, remember that multisite installs need one per domain.

Shopify is different, and it is worth knowing before you promise anything. Shopify's theme templates let you edit robots.txt.liquid only; placing an arbitrary file at the root is not part of theme templating. In practice llms.txt gets served either through an app or proxy, or from an edge layer in front of the store such as a Cloudflare Worker. Publishing it as a page and redirecting does not work: the response comes back as HTML, not text/plain.

Does anything actually read llms.txt?

This is the most-sold and least honestly answered question about the file. As of today no major engine has publicly stated that it reads llms.txt. Google writes the opposite outright: to appear in AI Overviews and AI Mode you 'don't need to create new machine readable files, AI text files, or markup', and 'there's also no special schema.org structured data that you need to add'.

OpenAI and Anthropic document their crawlers in detail — which user agent does which job, from which IP ranges — and neither documentation carries a line about llms.txt support. Publishing the file, then, does not open a channel the engines read; it means being ready if they decide to.

Publishing it is still reasonable, because it costs an afternoon and its readers are not only the large engines: agents making a single request to a known address, documentation tools and internal search systems use the file today. The honest framing is this — llms.txt is a cheap, harmless move that belongs at the bottom of the priority list. The moment it moves ahead of crawler access and page structure, it becomes a waste of time.

The order is not negotiable: crawler access first, then the answer structure on the page, then markup, and llms.txt last. Break that order and you get the familiar picture — a flawless llms.txt above a search crawler blocked in robots.txt.

How should llms.txt work on a multilingual site?

Because the specification allows the file at subpaths, two patterns are valid. First: one file at the root with an H2 section per language. Second: a separate file under each language's path — /tr/llms.txt, /en/llms.txt — with the root file pointing at them.

On a mid-sized four-language site the first works better: one request, one file, everything together. Once the file passes a few hundred lines, switch to the second. In both cases one mistake is fatal: if the language of the links and the language of the descriptions come apart, the file sends the model to the page in the wrong language.

# Kuzey Mobilya

> İstanbul merkezli ofis ve ev mobilyası üreticisi. Site Türkçe, İngilizce ve
> Almanca yayımlanır; her sayfanın diğer dillerdeki karşılığı vardır.

## Türkçe

- [Ürünler](https://example.com/tr/urunler): Katalog ve ölçü tabloları
- [Bayilik](https://example.com/tr/bayilik): Başvuru koşulları

## English

- [Products](https://example.com/en/products): Catalogue and dimension tables
- [Dealership](https://example.com/en/dealership): Application terms

## Deutsch

- [Produkte](https://example.com/de/produkte): Katalog und Maßtabellen
- [Händler](https://example.com/de/haendler): Bewerbungsbedingungen

How do llms.txt, robots.txt and sitemap.xml differ?

What are the five most common mistakes?

• Copying sitemap.xml and publishing it as llms.txt. A sitemap hands a machine addresses; llms.txt exists to hand it meaning, and a link list without descriptions adds nothing.
• Setting the summary line in italics. The specification says blockquote (>); an italic line is not recognised as the summary by a client that parses strictly.
• Mistaking the file for a permission. llms.txt grants no crawler anything; if the block is in robots.txt or the firewall, that is where it gets solved.
• Writing the file and leaving the pages as they were. The map takes a model to the page; if there is no quotable passage there, the visit comes to nothing.
• Writing it once and forgetting it. Six months on, an llms.txt with three links returning 404 does not describe the site — it misdescribes it, and drags the credibility of the whole file down with it.