llms.txt: What It Is — and Does It Actually Work?
Every few weeks someone declares llms.txt the future of the web, and someone else declares it dead. We build a validator and a generator for the thing, so people ask us which it is. Here's the honest answer, current as of October 2026.
The spec in 60 seconds
llms.txt is a file at your site root — https://yoursite.com/llms.txt — written in ordinary Markdown. It has a heading with your site's name, a blockquote with a one-line summary, and a list of your key pages:
# Example.com > Invoicing software for freelancers. ## Pages - [Pricing](/pricing): Plans and limits - [Docs](/docs): API reference - [Changelog](/changelog): Weekly releases
The idea: when a model or crawler wants to know what a site is, instead of guessing from navigation menus and meta tags, it reads one curated file. Like robots.txt, but describing content instead of permissions. It is a community-proposed specification, not an official web standard.
The controversy in one paragraph
Search engines never needed sites to describe themselves — that's what ranking algorithms are for. Skeptics argue the same applies to AI engines: models already learn what sites are from training data and retrieval, and a self-declared file can say anything. Supporters counter that a machine-readable summary lowers the cost of understanding small sites that algorithms under-serve. Both are right, which is why adoption is uneven.
Which engines use it today
Honestly: it varies, and it changes. Some AI crawlers fetch it opportunistically; some index it if present; several ignore it entirely, and no major engine has made it a formal, documented dependency. [We re-verify this quarterly — last review Oct 2026.] Anyone who tells you llms.txt definitively gets you cited is overselling. Anyone who tells you it's pointless is overselling too.
When llms.txt is worth it
- Your site is small or new. Less training data exists about you; a curated self-description helps retrieval systems that actually read it.
- Your best pages aren't your homepage. Docs, comparisons, calculators — llms.txt points engines at the pages worth quoting.
- You want a forcing function. Writing "what this site is, in one line, plus the five pages that matter" is a genuinely useful exercise, and this is where it lands.
When it isn't
- Your robots.txt blocks the AI crawlers anyway — fix that first; a description nobody fetches is decoration.
- Your homepage renders nothing without JavaScript — engines reading raw HTML see an empty shell with or without llms.txt.
- You expect it to substitute for being mentionable. llms.txt describes you; it doesn't vouch for you.
Try it on your own site
Our generator builds a valid file in about a minute, and the checker validates any site's file — structure, links, and how it fits with everything else. Or run the full AI visibility check to see all four factors in one report.
Whether or not engines adopt it, a valid llms.txt costs you ten minutes and forces you to write down what your site is about — that alone is worth it.