Does llms.txt work? What it does and does not do for AI search
Quick answer
Not as an AI search fix, based on what the AI companies themselves publish. As of October 24, 2026, the crawler documentation from OpenAI, Anthropic, Perplexity and Google does not say their crawlers read llms.txt, and Google’s guidance says “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” llms.txt is a proposed Markdown summary of a site. It is cheap and harmless, and it can help tools that read documentation, but it does not replace crawler access in robots.txt, text in your HTML, or good answers on your pages.
What llms.txt is
It is a proposal published at llmstxt.org: a Markdown file at /llms.txt that lists a site’s most useful pages with short descriptions, so a language model can find them without parsing menus and ads. It is not a standard, and it is not a way to allow or block crawlers; robots.txt does that.
What the evidence says
| Source | What it says |
|---|---|
| Google, “AI features and your website” | “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.” |
| Google’s John Mueller (via Search Engine Journal) | Said no AI services had said they use llms.txt, that server logs show they do not even check for it, and compared it to the keywords meta tag. |
| Google’s Gary Illyes (via Search Engine Land, July 2025) | Reported as saying Google does not support llms.txt and is not planning to. |
| OpenAI, Anthropic and Perplexity crawler docs | Explain crawler control through robots.txt only. None of the three pages mentions llms.txt as something their crawlers read (checked October 24, 2026). |
| Site owners’ logs | Some report OpenAI fetching llms.txt. A fetch is not proof that the file is used to pick sources or citations. |
Note the difference between publishing and reading: OpenAI and Perplexity publish llms.txt files for their own developer docs. That tells you they find the format useful for documentation, not that their crawlers use your llms.txt to decide what to cite.
What we found on three live sites
All three sites we audited today already have an llms.txt (astorventuresllc.com 39,773 bytes, hohocam.com 8,762 bytes, kemuva.com 19,675 bytes). We keep one here too because it costs nothing. Our checker reports it but gives it 0 points, and the scores below come entirely from crawler access, HTML text, schema and page basics:
| Site (home page) | Score | What the checker found |
|---|---|---|
| astorventuresllc.com | 98/100 | All 16 crawlers allowed; 3,664 words in the raw HTML. Small gaps: a 253-character meta description and no sameAs profile links in the schema. |
| hohocam.com | 94/100 | All 16 crawlers allowed; 2,165 words in the raw HTML; Product, Offer, FAQPage and VideoObject schema. Gaps: no <main> element and no published or updated date in the schema. |
| kemuva.com | 87/100 | All 16 crawlers allowed; 2,809 words in the raw HTML. Gaps: only Organization and WebSite schema on the home page, one question-style heading, no <main> element, no dates in the schema, and a 193-character meta description. |
What to do instead (or first)
- Make sure robots.txt allows the search crawlers:
OAI-SearchBot,Claude-SearchBot,PerplexityBot,Googlebot,Bingbot. See GPTBot vs OAI-SearchBot. - Make sure your firewall or CDN does not block them. See How to check if ChatGPT can crawl my site.
- Put your main text, prices and contact details in the HTML, not only in JavaScript. See Can AI crawlers read JavaScript?
- Answer the questions customers ask, in plain sentences right under a question heading.
- Then, if you like, add an llms.txt. It will not hurt.
Want a second pair of eyes? The AI visibility audit checks 16 AI and search crawlers against your robots.txt, firewall and HTML, and sends a plain-English PDF report within 48 hours with ready-to-paste fixes. $79 for the report, $149 with the fixes applied. It is a diagnosis, not a promise of citations.
Frequently asked questions
Will llms.txt hurt my site?
We know of no evidence that it does. It is a plain text file that crawlers can ignore.
Is llms.txt the same as robots.txt for AI?
No. robots.txt tells crawlers what they may fetch. llms.txt only describes your content; it cannot allow or block anything.
Should I pay someone to make an llms.txt?
We would not. If you want one, it is a short Markdown list of your key pages that you can write yourself in a few minutes. Spend the effort on crawler access and on the content of your pages first.
Could this change?
Yes. If an AI company documents that its crawlers use llms.txt, the advice changes. We re-check the vendors’ documentation when we update this page.
Sources (checked October 24, 2026)
- llmstxt.org: the llms.txt proposal
- Google: AI features and your website
- Search Engine Journal: Google says llms.txt comparable to keywords meta tag
- Search Engine Land: Google says normal SEO works for AI Overviews and llms.txt won't be used
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers