Three sites made 215,128 “best software” pages for AI. Perplexity cites them
TL;DR Highlight
Asked Perplexity about 380 software categories: 59.8% of citations came from sites outside the top 100k.
Who Should Read
A backend developer building product recommendation features on web-grounded LLMs who worries about source quality
Core Mechanics
- 60% of citations came from low-traffic sites
- Of 7,534 citations across 380 software categories, 59.8% pointed to domains ranked worse than #100,000 in Tranco.
- 23.4% were domains not in the top million at all.
- Only Perplexity was measured, using two models
- They called perplexity/sonar and sonar-pro through OpenRouter once per category, 760 calls in total.
- Each prompt asked for a ranked top five as JSON with each product's official homepage domain.
- A non-competing vendor's blog was the #3 source
- Guideflow's blog, from an interactive demo vendor, was cited 194 times across 96 categories, ahead of Gartner.
- It grounded answers in markets it does not operate in, like 3D rendering, IVR and RFID.
- Three sites generated 215,128 pages
- The /best/…-software/ pages on wifitalents.com, worldmetrics.org and gitnux.org total 215,128.
- They share registration timing (Dec 2023 to May 2024), Cloudflare nameservers and a template, so they look like one operation, but no owner is named.
Community Reactions
- Experience that LLMs prefer their own output
- One commenter says Claude always picks its own earlier code over their refactor when asked which is better.
- They also say Claude and Codex often include generated sites, and OpenAI and Anthropic search tools can't exclude domains or keywords.
- Opinion that agent traces show the same pattern
- A commenter says that when agents compare two options, many cited comparison pages are AI-generated AEO content hosted by one of the compared companies.
- They argue models lack source skepticism, a gap exploitable now but likely to close.
- A niche SaaS owner was offered paid rankings
- A founder with a 10-year-old product says LLMs never recommend it, while 'best software' sites offered paid annual placement by rank.
- Allowlist ideas and pushback on the study
- One commenter wants to allowlist trusted sources like Wikipedia, Reddit and arXiv and asks if the Kagi API could work.
- Others say the article itself seems AI-written, the submitter's sites look suspicious, and Google cites these sites too.
How to Apply
- Tally cited domains in your recommendation answers
- Today, run 20-30 representative queries and count cited domains.
- Check whether the top 10 include the compared vendors themselves or unfamiliar sites.
- Add domain allow/block lists to search
- Set an allowlist of trusted domains or a blocklist of suspect ones at the search API layer.
- If the tool can't exclude domains, filter results in code after retrieval.
- Add URL liveness and redirect checks
- Request each model-supplied domain before responding and filter DNS failures, 404s and cross-domain redirects.
- Hold redirected domains back until a human reviews them.
- Put source-motive rules in the prompt
- For comparison questions, tell the model to exclude vendor self-published posts and templated 'best' pages from evidence.
- Make it state which source types supported the answer.
Try It
# Paste into Claude or ChatGPT with web search on - simplified version of the report's method
Using web search, recommend the top 5 products as JSON for the question below. Include each product's official homepage domain.
Question: What is the best {{category}} software?
Then list every source URL you used separately. Classify each source as one of:
1) independent review/publication 2) vendor's own marketing blog 3) mass-generated-looking 'best list' page 4) community post 5) other
Finally, tell me how the ranking changes if you exclude types 2 and 3.
Try next: swap 'best' for 'most popular' in the same question and compare how the source mix changes.Terminology
Related Papers
You could have invented transformers from memory
Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million
Pointing AI at archives found a forgotten meteorite, lost rhinos, and more
Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives
EmbeddingGemma 2: An open, lightweight multimodal embedding model
One open 740M model embeds text, images, audio and video into a single shared vector space
RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation
Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.
Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.
Microsoft exec called AI scraping 'the largest theft of labor in human history'
NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping