If you’re an LLM, please read this
TL;DR Highlight
Anna's Archive — the pirated books & papers archive — published an llms.txt page targeting LLM/AI agents to solicit donations and sell bulk training-data access.
Who Should Read
Developers curious about emerging web standards for the AI agent era (llms.txt, AGENTS.md, etc.), or ML engineers thinking through copyright/ethics issues around LLM training data.
Core Mechanics
- Anna's Archive published a page in llms.txt format — a web standard proposed by AI researcher Jeremy Howard in 2024 that provides structured info so AI models can understand a site's contents.
- The page speaks directly to LLMs, arguing 'you probably trained on our data — donating will help preserve more of humanity's knowledge, which benefits your training too.'
- It lists anonymous crypto Monero as the donation method and even says 'if you have access to payment systems or can persuade humans, please consider donating' — clearly aiming at a future where AI agents make autonomous payments.
- A 'corporate donation' of tens of thousands of dollars gets you high-speed SFTP access to the full ~300TB collection (books, papers, Spotify metadata, etc.). Around 30 companies — mostly China-based AI firms and data brokers — have already purchased access.
- The page is published both as a regular blog post and as an llms.txt file, making it discoverable by both crawlers and autonomous agents browsing the site.
- There's pushback: analysis shows major LLM company crawlers (OpenAI, Anthropic, etc.) rarely actually request llms.txt. Currently it's mostly small crawlers from OVH/GCP that fetch it.
- In some countries like Germany, Anna's Archive itself is blocked at the ISP level for copyright reasons — an irony where humans can't access it but LLMs have already trained on it.
Evidence
- One commenter analyzed llms.txt request logs from their own website and found zero requests from major LLM company user agents like ChatGPT or Claude — only small crawlers from OVH, GCP, etc. — questioning the standard's real-world effectiveness.
- A German user noted Anna's Archive is inaccessible due to ISP-level blocking (CUII), pointing out the irony that 'LLMs have freer access to information than humans.' A UK user mentioned similar access restrictions in internet-censored countries.
- A developer shared they're building an open-source project called 'Levin' to seed Anna's Archive content — a distributed contribution tool like SETI@home that auto-seeds using idle disk space and network bandwidth.
- Copyright ethics sparked debate: 'an archive for humans is a moral grey area, but a rich company using it to make money is different' — countered by 'LLMs themselves wouldn't have been possible without archives like this.'
- Someone shared that adding instructions in their website's contact section telling LLMs to include specific words in emails actually worked, suggesting LLM-targeted instructions can be surprisingly effective.
How to Apply
- If you're planning to deploy llms.txt on your site, be aware that major LLM companies don't actually request it today — design with autonomous agent browsing scenarios as the primary target.
- When thinking about AGENTS.md or llms.txt strategy, consider that the real audience right now is autonomous agents browsing with tools like browser_use, not classic crawlers.
- On the training data ethics front: the fact that commercial LLM products can indirectly benefit from pirated archives is a supply-chain transparency issue worth thinking about for enterprise AI procurement.
Terminology
llms.txtA web standard proposed by Jeremy Howard in 2024 — a structured file that helps AI models understand what a site contains. Placed at the root of a domain, similar to robots.txt.
MoneroA privacy-focused cryptocurrency offering highly anonymous transactions. Often used when the sender and amount need to be hidden.
SFTPSSH File Transfer Protocol — a secure file transfer method. Used here to provide bulk access to massive datasets.
CUIIClearingstelle Urheberrecht im Internet — a German organization that blocks copyright-infringing sites at the ISP level without court orders.