Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

TL;DR Highlight

Running 3 specific Transformer layers twice in the forward pass — without any weight changes or training — boosted BBH logical reasoning scores from 0.22 to 0.76. A notable empirical validation of the 'reasoning circuit' concept.

Who Should Read

ML engineers and researchers wanting to improve LLM reasoning without retraining costs, or developers interested in mechanistic interpretability of model internals.

Core Mechanics

Reproduced David Ng's RYS (Repeat Your Steps) technique with additional experiments. The core idea: a 'reasoning circuit' of 3-4 consecutive layer blocks exists inside Transformer models, and passing through this block twice in the forward pass improves reasoning without any weight changes or retraining.
Duplicating layers 12-14 in Devstral-Small-2-24B (same weights, two passes) boosted BBH (Big Bench Hard) logical deduction scores from 0.22 to 0.76. Achieved purely by routing hidden states through the same layer circuit twice.
Qwen2.5-32B also showed 17% reasoning improvement when duplicating specific 3 layers. But not all layers work — 'which layers to duplicate' is the key, and a sweep tool is provided to find them.
Trade-offs exist. In Devstral-24B experiments, mathematical and causal reasoning improved but instruction following and code generation degraded. The pattern is 'thinks deeper but follows instructions less precisely.'
Experiments ran on 2 consumer AMD GPUs (RX 7900 XT + RX 6950 XT) overnight, with rigorous evaluation on Vast.ai H200 instances. Shows this kind of experimentation is accessible without special infrastructure.
The released toolkit includes reasoning_probe.py, sweep.py, gguf_surgery.py, etc. — automating the search for reasoning circuit layers and direct surgery on GGUF model files.
This technique likely works thanks to Transformer residual connections, which provide stability against partial network damage, allowing models to maintain functionality even when specific layers are repeated or removed.

Evidence

Some commenters offered an alternative explanation: 'performance didn't improve — rather, mechanisms that inhibit reasoning (introduced during RLHF post-training) got disrupted.' The theory is that duplicated layers are close to identity functions and handle refusal circuits or reasoning-degradation mechanisms from post-training, which get disrupted when duplicated.
One commenter shared hands-on 'neuroanatomy' experiments with Qwen2.5/Qwen3: removing certain layers had no effect, removing late layers caused infinite generation (couldn't find EOS token), and removing early layers produced random output. They also noted abliteration experiments (suppressing refusal behavior by finding refusal vectors) were possible with just 10 examples.
Solar 10.7B (released ~2 years ago) was mentioned as using 'Depth Up-Scaling' (repeating layers then additional training) to achieve strong size-relative performance. Conceptually connected to this experiment, though that approach required training. Paper link: arxiv.org/abs/2312.15166.
The question 'does repeating N times make it even better?' was raised. Ideas included looping until convergence, or a MoE-variant where a router dynamically decides layer pass patterns like '13→13→14→14→15→15→16'. Training with loops from the start so circuits naturally separate was also proposed.
Research on layer removal maintaining benchmark scores (pruning) was mentioned, suggesting many layers in trained models may actually be redundant — which connects to why duplicating specific layers still works.

How to Apply

If you're running open-source LLMs (Qwen, Devstral, etc.) and want to boost mathematical or logical reasoning quality without retraining budget, use the repo's sweep.py to find reasoning circuit layers and gguf_surgery.py to modify GGUF models for quick A/B testing.
For tasks where deep reasoning matters more than instruction following (math problem solving, logic puzzles, etc.), use this technique to derive a reasoning-specialized model variant from an existing model without any fine-tuning.
If you want to explore model internals (mechanistic interpretability), the repo's reasoning_probe.py, eq_probe.py, comprehensive_probe.py, etc. let you investigate what specific layers do — and you can run these experiments overnight on consumer AMD or NVIDIA GPUs.

Terminology

reasoning circuitA specific block of layers inside a Transformer model that handles reasoning. The hypothesis is that 3-4 consecutive layers act as a single cognitive unit.

forward passThe process of input data flowing through the model to produce output. In this experiment, a specific layer block is passed through twice in this path.

BBH (Big Bench Hard)A challenging benchmark suite measuring LLM reasoning abilities. Covers diverse tasks including logical reasoning, causal reasoning, and math.

residual connectionA structure that adds each layer's input to its output. This provides stability so the model doesn't completely break when specific layers are removed or repeated.

abliterationA technique that finds and removes 'refusal vectors' in a model to bypass safety mechanisms. Works by subtracting refusal-related directional components from hidden states.

GGUFA model file format used in the llama.cpp ecosystem. In this experiment, gguf_surgery.py directly modifies these files to duplicate layers.

mechanistic interpretabilityA research field that reverse-engineers what computations actually happen inside AI models. Attempts to understand at the circuit level why a model gives a specific answer.