Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

TL;DR Highlight

Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

Who Should Read

Backend developer building RAG or search pipelines who must find content that keyword search misses across millions of documents

Core Mechanics

Embeddings beat keyword search
- 17th-century Dutch spells rhinoceros about 15 ways and OCR is noisy, so keyword search misses most hits.
- The author embedded 5.7M passages and searched by meaning instead.
The meteorite report never said meteorite
- An 1812 Java Government Gazette letter called it a stone precipitated from a thunder-cloud.
- Semantic search found it, and it is absent from six catalogues, 26 years earlier than Maharashtra's current earliest record.
A cheap model filters, Haiku reads
- A tiny decision model, Jev, answers narrow questions like is this a real animal or a meteorite fall.
- Reading 59,000 elephant mentions cost about $3.
- The author says reading everything with Haiku would cost much more money and time.
Test on known events first
- Before searching for new things, the pipeline had to find Breen's dodo, Laki 1783 and Tambora 1815.
- The author compares it to burying your own wristwatch to test a metal detector.

Community Reactions

Commenter says knee-jerk anti-AI sentiment hides the achievement
- They ask whether the same finds via traditional NLP or OCR 3-5 years ago would be dismissed.
- They argue the orchestration work makes it equally impressive and valid.
Commenter objects that it is shallow without expertise
- They note the dodo and Newton cases came from domain experts, while this author started by asking AI what field and questions to pursue.
- They say shallow work feels significant only when you don't know the field.
Commenter doubts the author learned much
- One commenter wonders how much the author learned about the Dutch East India Company and likens it to junk food.
Commenters report trying similar approaches
- One is using the open-sourced antiquity toolkit to investigate the Berlin Goldhat mystery.
- Another asked ChatGPT to spot health signs in old paintings and said the first suggestion was weak but a retry gave a more compelling one.

How to Apply

Test your search pipeline with 3 known answers
- Pick 3 documents in your current RAG or search index whose location you know.
- Check whether they appear in the top results.
- If not, don't trust a no-results answer.
Build a cheap filter then big reader
- When candidates number in the tens of thousands, ask a small model one yes/no question each.
- Pass only survivors to a larger model for summary and extraction.
- Log counts and cost at each stage.
Add semantic search next to keyword search
- For logs or scans full of spelling variants and OCR errors, add embedding search.
- Chunk documents, embed them, and query with a natural-language sentence.
Force source links on every AI claim
- Make an original quote and document ID required fields in your output schema.
- Only promote items a human has checked against the source.

Try It

copy & run

# Paste into Claude or ChatGPT — simplified version of the article's cheap filter + source check

You are a first-pass filter. Judge only whether the passage below directly describes {{topic, e.g. a meteorite fall}}.

Passage:
{{passage}}

Rules:
1. On the first line write only YES / NO / MAYBE.
2. On the second line quote the exact supporting phrase from the passage, or write 'none'.
3. Do not guess beyond the passage.

Next step: first test it with 3 passages you know are positive and 3 unrelated ones to confirm the verdicts are right.

Terminology

EmbeddingA list of numbers that acts as a fingerprint of a text's meaning. Different words with similar meaning land close together.

System One decision modelA small, cheap model that answers narrow questions fast without deep reasoning. It works as a gatekeeper before a bigger model.

OCRTechnology that turns scanned characters into text. Handwriting and old print produce many errors.

GLOBALISEA project that turned millions of handwritten Dutch East India Company pages into searchable text.