A Survey of Post-Training Scaling in Large Language Models

Who Should Read

Researchers and engineers following LLM scaling trends who want to understand the landscape of post-training techniques as pre-training data becomes scarce.

Core Mechanics

• Pre-training data scaling is hitting practical limits — high-quality internet text is largely exhausted for LLM training
• Three post-training scaling approaches are emerging as alternatives: (1) Inference-time scaling (more compute at test time), (2) RL-based reasoning (training models to reason better), (3) Synthetic data generation (models teaching themselves)
• Inference-time scaling (chain-of-thought, self-consistency, tree search) can double effective model capability without any training
• RL-based reasoning training (RLHF, RLAIF, process reward models) improves reasoning ability proportionally to training compute invested
• Synthetic data generation (models generating their own training data) enables continued scaling beyond human-labeled data limits
• The three approaches are complementary — combining them produces superadditive benefits

Evidence

• Inference-time scaling: spending 10x more compute at inference matches training a 3x larger model on reasoning tasks
• RL reasoning training: consistent log-linear improvement in reasoning ability with training compute invested
• Synthetic data: models trained on self-generated + curated data outperform those trained on human-labeled data alone for reasoning tasks

How to Apply

• For immediate capability improvements without training: invest in inference-time scaling — chain-of-thought, self-consistency, and best-of-N sampling are available today for any model.
• For sustained capability improvements: combine RL training (for reasoning) with synthetic data generation (for continued post-training scaling) — this is the trajectory of frontier model development.
• Prioritize based on your constraints: inference-time scaling requires no training but costs more per query; RL training requires significant upfront compute but reduces per-query cost afterward.

Code Example

# TTC Style: Improving Inference Quality with Best-of-N Sampling
import anthropic

client = anthropic.Anthropic()

def best_of_n_inference(prompt: str, n: int = 8) -> str:
    """Generate N responses and select the most confident answer (simple TTC implementation)"""
    responses = []
    for _ in range(n):
        msg = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}]
        )
        responses.append(msg.content[0].text)
    
    # Select the best answer via majority vote or a separate judge model
    judge_prompt = f"""
From the following {n} responses, select the most accurate and logical one and output only its content.

Responses:
" + "\n---\n".join(f"{i+1}. {r}" for i, r in enumerate(responses))
    
    judge = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    return judge.content[0].text

# Usage example
result = best_of_n_inference("Write and explain a Python code to find the 10th term of the Fibonacci sequence.", n=4)
print(result)

Terminology

Related Papers

• Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

  LLM의 RL 후처리 학습(post-training)에서 성능 향상의 대부분이 중간 레이어 소수에 집중되며, 단 하나의 레이어만 학습해도 전체 파라미터 학습과 비슷하거나 더 나은 결과를 낼 수 있다는 연구 결과. 이는 RL 학습 비용을 대폭 줄일 수 있는 가능성을 시사한다.

• Knowledge Distillation of Black-Box Large Language Models (2024)

  GPT-4 같은 내부 구조에 접근할 수 없는 독점 LLM에서 작은 모델로 지식을 효과적으로 전달하는 Proxy-KD 기법을 소개하는 논문으로, 전통적인 White-Box 방식보다 성능이 높다는 점에서 주목할 만하다.

• Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

  PyTorch나 autograd 없이 C와 CUDA만으로 GPT-2 수준의 LLM을 처음부터 구현한 교육용 프로젝트로, 역전파·BPE 토크나이저·FlashAttention까지 직접 손으로 작성했다.

• Show HN: Neural Particle Automata

  고정된 격자 대신 움직이는 파티클 위에서 동작하는 Neural Cellular Automata의 확장 버전으로, 형태 생성·포인트 클라우드 분류·텍스처 합성 등 다양한 작업에서 자기조직화 동작을 학습할 수 있다.

• The annotated PyTorch training loop

  PyTorch 학습 루프의 각 코드 줄이 왜 그 위치에 있어야 하는지, 순서를 바꾸거나 빠뜨렸을 때 어떤 문제가 생기는지를 단계별로 설명한 심층 가이드다.

• When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks

  VLM 자가학습 루프에서 verifier가 특정 태스크에 맞지 않으면 학습할수록 오히려 성능이 떨어지는데, DPO 손실값은 멀쩡히 내려가서 눈치채기도 어렵다.