History LLMs: Models trained exclusively on pre-1913 texts

Who Should Read

ML researchers interested in domain-specific pretraining and historical NLP, and digital humanities scholars exploring AI for historical text analysis.

Core Mechanics

• The model family was trained exclusively on historical texts from before 1913 — newspapers, books, letters, government documents — containing no modern vocabulary, concepts, or references.
• The result is a model that genuinely 'thinks' in the idiom and worldview of the late 19th/early 20th century: it doesn't know about WWI, modern physics, or computing.
• This makes it useful for period-accurate text generation, historical document analysis, and studying how language and reasoning patterns have changed over time.
• The 80B token pretraining corpus is notable — assembling high-quality historical text at this scale required significant digitization and cleaning effort.
• The model family (4B parameters) is small enough to run locally, making it accessible for humanities research without GPU cluster requirements.
• Evaluation showed the model excels at historical text completion and period-appropriate prose generation, but obviously fails at any task requiring modern knowledge.

Evidence

• Demo outputs shared in the HN thread showed convincingly period-accurate prose — no anachronisms, appropriate vocabulary register, and correct historical references.
• Digital humanities researchers in the comments expressed genuine excitement, noting this fills a gap in current NLP tools for pre-modern text analysis.
• Debate about the value of 'isolated' period models vs fine-tuning a modern model on historical data — the argument for isolation is that modern training data introduces anachronistic reasoning patterns.
• Historians noted potential use in transcribing and extending damaged historical documents where period-accurate language modeling is crucial.

How to Apply

• For digital humanities: use the model for historical document completion, transcription assistance, and period-accurate text generation without worrying about modern contamination.
• For NLP researchers: this is a useful probe model for studying how language and conceptual structure have changed — compare outputs on the same prompts to a modern model.
• If you're building historical education tools or games, this model provides a unique source of period-appropriate generated content.
• The corpus assembly methodology (80B tokens of pre-1913 text) is itself worth studying for researchers building other domain-specific or temporal pretraining datasets.

Terminology

Related Papers

• Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

  LLM의 RL 후처리 학습(post-training)에서 성능 향상의 대부분이 중간 레이어 소수에 집중되며, 단 하나의 레이어만 학습해도 전체 파라미터 학습과 비슷하거나 더 나은 결과를 낼 수 있다는 연구 결과. 이는 RL 학습 비용을 대폭 줄일 수 있는 가능성을 시사한다.

• Knowledge Distillation of Black-Box Large Language Models (2024)

  GPT-4 같은 내부 구조에 접근할 수 없는 독점 LLM에서 작은 모델로 지식을 효과적으로 전달하는 Proxy-KD 기법을 소개하는 논문으로, 전통적인 White-Box 방식보다 성능이 높다는 점에서 주목할 만하다.

• Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

  PyTorch나 autograd 없이 C와 CUDA만으로 GPT-2 수준의 LLM을 처음부터 구현한 교육용 프로젝트로, 역전파·BPE 토크나이저·FlashAttention까지 직접 손으로 작성했다.

• Show HN: Neural Particle Automata

  고정된 격자 대신 움직이는 파티클 위에서 동작하는 Neural Cellular Automata의 확장 버전으로, 형태 생성·포인트 클라우드 분류·텍스처 합성 등 다양한 작업에서 자기조직화 동작을 학습할 수 있다.

• The annotated PyTorch training loop

  PyTorch 학습 루프의 각 코드 줄이 왜 그 위치에 있어야 하는지, 순서를 바꾸거나 빠뜨렸을 때 어떤 문제가 생기는지를 단계별로 설명한 심층 가이드다.

• When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks

  VLM 자가학습 루프에서 verifier가 특정 태스크에 맞지 않으면 학습할수록 오히려 성능이 떨어지는데, DPO 손실값은 멀쩡히 내려가서 눈치채기도 어렵다.

Related Resources

• https://github.com/DGoettlich/history-llms