A third of Perplexity's citations don't contain the number they're cited for
TL;DR Highlight
Of 1,826 Perplexity citations, 34.7% didn't open or lacked the cited figure
Who Should Read
A backend developer building a RAG or search feature that shows LLM search API citations directly to users
Core Mechanics
- One in three citations failed as evidence
- Of 1,826 citations on sentences stating a figure, 34.7% pointed to a page that wouldn't open or contained none of the sentence's numbers.
- Per claim, 14.4% of 872 claims failed, passing if any cited page had a figure.
- Dead links aren't the main problem
- Of sonar's 2,915 unique cited URLs, 16.1% were behind a login, paywall, 403 or bot wall, while 2.5% were dead (404 etc.).
- Per answer, 84.2% cited at least one URL an ordinary reader couldn't open.
- Pages that open often don't say it
- Of readable pages, 16.1% contained none of the claim's figures.
- Vercel's $20/month was cited to vercel.com/docs/plans, which doesn't contain $20.
- Headquarters addresses were attributed to Wikipedia, but none of four articles, including Docker's, contained the address.
- CEO and HQ address fail the most
- Figure pass rates were 44.3% for current CEO, 53.0% for headquarters address, and 82.0% for headcount.
- The report reads this as facts written down in one canonical place being cited more accurately.
Community Reactions
- Opinion questioning the report's origin
- Commenters noted a similar report on trellner.com posted the same day with the same layout and About page and suspected coordination.
- Another found it odd that the same person submitted two anti-Perplexity reports.
- Experience that other services do the same
- Commenters said Google search summaries and Gemini also often link to pages that don't support the statement.
- One said a citation preview showed a reliable source but the real source was Reddit and other weak sites.
- Question about deterministic citation tools
- One asked whether, in a custom harness, the model should get a deterministic tool to copy passages with links rather than reproduce them stochastically.
- Opinion citing legal cases
- A commenter said lawyers have gotten in trouble using AI for case law because of plausible but fabricated citations.
- They said a California bill requiring review of AI-generated documents awaits the governor's signature.
How to Apply
- Grep-check your answers' citations
- Today, take 10 answers your service shows and extract sentence-URL pairs where the sentence has a number.
- Fetch each URL and string-check whether any amount, year or number from the sentence appears.
- Collect the failing pairs and compute the rate.
- Add a URL status check before showing citations
- In post-processing, open each cited URL and flag 404, 410, 403 and login walls.
- Mark unopenable sources as unverifiable or drop them.
- Require per-sentence markers for numeric claims
- Use an API that returns inline [n] markers rather than a list of sources at the end.
- Sentence-level markers are what make the check above possible.
- Snapshot cited pages yourself
- For important answers, store the cited page text at response time.
- This keeps the evidence checkable after the link dies.
Try It
# Paste into Claude or ChatGPT (simplified version of the report's method: manual citation check)
Below are a sentence from an AI search answer and the text of the page it cites.
[Answer sentence]
{{sentence}}
[Cited page text]
{{page text}}
Do the following.
1. Extract every money amount, percentage, year and number with 3+ digits from the sentence.
2. Show in a table whether each appears in the page text (including format differences like $185M = 185 million = 185000000).
3. Mark PASS if at least one is present, FAIL if none is.
4. If only part of the sentence is supported, say which part has no support.
Next to try: run 10 answers in the same format and count the FAIL rate.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.