OpenArch – PyTorch implementations of modern LLM architectures

Who Should Read

A backend developer who uses LLM APIs and wants to compare how model internals like attention and MoE differ in code

Core Mechanics

• One readable file per model
  • Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi and more are hand-written in PyTorch.
  • Each model folder has a model.py and a short README.md.
  • The goal is readability, not production speed.
• Architecture differences at a glance
  • The README table lists normalization, positional encoding, attention type and MoE for each model.
  • For example, Llama 3 uses GQA with no MoE, while DeepSeek R1 uses Multihead Latent Attention with MoE.
  • Gemma 3 and GPT-OSS 20B are listed as GQA with sliding window.
• Only some models are done
  • GPT-2 XL, Llama 2/3/4 Maverick, Qwen 3, Kimi K2 and GLM 4.5 are marked ✅ for forward passes.
  • Qwen3 Next, Qwen3-VL and Dall-e are marked 🚧 as under construction.
  • The target list follows the 103 architectures in Sebastian Raschka's Architecture Gallery.
• Explicitly not for production
  • The author says it is not meant to compete with transformers or other production libraries.

Community Reactions

• Question about how correctness is verified
  • A commenter pointed out the README has no details on how correctness is confirmed.
  • The README itself lists output-matching tests against official weights as a contribution task.
• Suspicious Kimi K2 router code
  • A comment noted that kimi-K2 model.py calls torch.topk with k=self.num_experts.
  • Other MoE models use k=self.top_k, and the constructor's top_k=8 is never stored.
  • They asked whether dense routing was intentional or a bug.
• Surprise that open-weight models need custom code
  • One commenter said they only just learned that weights alone need architecture-specific code to run.
  • They asked whether hosting providers differ in inference engine implementation, not just quantization and configuration.
• Concern about solo maintenance
  • A commenter asked whether one individual maintainer can keep up with future model releases.
  • Another was struck by how short and simple the implementations are.

How to Apply

• Open two models' model.py files side by side
  • Clone the Llama 3 and DeepSeek folders today and compare their attention code.
  • Ask an AI to explain how GQA and Multihead Latent Attention differ in the code.
• Tabulate the architecture of models you use
  • Find the open models your team uses in the README table and note attention and MoE.
  • Add one line on architecture differences to your model selection doc.
• Cross-check what you read against official code
  • When using it for learning, compare the same parts with the official transformers implementation.
  • For services, follow the author's advice and use official implementations or transformers.

Try It

# Paste into Claude or ChatGPT (simple study prompt using OpenArch)

Below is the model.py code for {{Model A name}} and {{Model B name}}.

[Model A code]
{{paste Model A model.py}}

[Model B code]
{{paste Model B model.py}}

Compare the two implementations in a table by: normalization, positional encoding, attention type, MoE presence and routing. Explain with analogies a backend developer would understand.

Next: if any router or attention code looks suspicious, ask how it differs from the official implementation.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://github.com/anuj0456/OpenArch
• https://github.com/lucidrains