Serve Markdown to AI Agents with Accept Headers
TL;DR Highlight
Serve Markdown from the same URL when clients send Accept: text/markdown.
Who Should Read
Backend or web developers whose docs are read by agents or RAG pipelines and who want fewer wasted tokens and less noise
Core Mechanics
- One URL, two responses
- Browsers get HTML, and clients sending Accept: text/markdown get Markdown.
- It uses HTTP content negotiation, where a request header picks the response format.
- Claimed token savings
- Markdown drops nav, styles, scripts and layout wrappers.
- Agents spend context on your prose instead of your DOM.
- Less noise for RAG
- Ads, related-content rails and modal overlays disappear.
- The text a RAG pipeline embeds gets a better signal-to-noise ratio.
- Claimed faster first token
- There is less to fetch, parse and stuff into the context window.
- The model starts thinking sooner.
- A correctness checklist
- Check that Accept: text/markdown returns Markdown and that Vary: Accept is set.
Community Reactions
- Caching concerns raised
- Commenters quoted Roy Fielding and Simon Willison saying Accept-based negotiation is bad for caching.
- They noted Cloudflare CDN may serve cached JSON to an HTML client or vice versa.
- No agents use it yet
- Some said they won't adopt it until a top AI chatbot sends this header.
- Others asked what sites get in return.
- Harnesses should convert
- Some argued the agent harness should convert HTML to Markdown itself.
- They said the whole web shouldn't be rewritten for that.
- Trust and security worries
- Some said bots can't trust that the Markdown matches the HTML.
- Others warned about prompt injection and feeding LLMs bad content.
How to Apply
- Check your site's headers with curl
- Today, run curl -sI -H "Accept: text/markdown" <URL> and check Content-Type and Vary.
- Fetch the body with curl -s and see whether HTML still comes back.
- Add Vary: Accept and 406 handling
- When adding a Markdown variant, always set Vary: Accept on responses.
- Return 406 for unsupported types.
- Test CDN caching first
- Before shipping, verify the CDN caches responses separately per Accept header.
- If it doesn't, serve Markdown from a separate URL such as .md.
Try It
# Paste into a terminal (replace with your site URL)
curl -sI -H "Accept: text/markdown" {{URL}}
# Fetch the Markdown body
curl -s -H "Accept: text/markdown" {{URL}} | head -50
# Compare with a normal HTML request
curl -sI -H "Accept: text/html" {{URL}}Terminology
Related Papers
Mercury 3: Diffusion at scale
Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality
Timestral: Mistral's time-series foundation models
A 209M time-series model ranks first on three benchmarks and runs on CPU.
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.
If you don't run your own LLM proxy, how do you know whats going on?
Running your own LLM proxy shows which keys, models, and cache misses drive your usage.
DeepSeek: DeepSeek v4.1 Flash 1017
DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents
DeepSeek-v4.1-Flash-1017
DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length
Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware