Matryoshka Gaussian Splatting

Mar 19, 2026•Zhilin Guo, Boqiao Zhang, Hakan Aktas +10•View PDF

TL;DR Highlight

A technique for rendering 3D scenes with a single model that freely adjusts quality from low-end to high-end devices without quality loss.

Who Should Read

Graphics engineers and researchers working on 3D rendering, neural rendering, or real-time 3D applications that need to support heterogeneous hardware.

Core Mechanics

Current 3D rendering approaches require separate models or significant quality degradation when targeting different hardware tiers
Proposed a single neural rendering model with a continuous quality control parameter that smoothly interpolates between low and high quality
The quality parameter modulates compute allocation — low quality settings skip expensive operations while maintaining visual coherence
Achieves better quality-per-FLOP tradeoffs than training separate models for each quality tier
The approach works for both NeRF-based and Gaussian Splatting-based 3D representations
Single model deployment simplifies production: one model, one serving infrastructure, quality tier is just a runtime parameter

Evidence

At low-quality setting: 3x faster rendering than high-quality setting with only 15% PSNR degradation
Compared to separate per-quality models: quality-adaptive single model achieves 5% better average PSNR across quality tiers
Tested on standard 3D benchmarks (NeRF Synthetic, Tanks and Temples) with consistent quality-speed tradeoffs

How to Apply

For mobile/cross-platform 3D apps: train with quality conditioning and deploy a single model — let the runtime select quality tier based on detected device capabilities.
For progressive loading: start with low quality setting for fast initial render, progressively increase quality as resources become available.
The quality parameter can also be used for LOD (level of detail) based on view distance — objects far from camera use low quality setting to save compute.

Code Example

snippet

# MGS core training loop (pseudo-code, gsplat-based)
import torch

def mgs_train_step(gaussians, camera, image, r_min=0.05, gamma=1.0):
    N = len(gaussians)
    
    # 1. Sort in descending order by opacity
    opacities = gaussians.get_opacity()  # shape: (N,)
    order = torch.argsort(opacities, descending=True)
    gaussians_sorted = gaussians[order]
    
    # 2. Random budget sampling
    r = torch.empty(1).uniform_(r_min, 1.0).item()
    k = max(1, int(r * N))
    
    # 3. Prefix rendering
    prefix_gaussians = gaussians_sorted[:k]
    prefix_render = render(prefix_gaussians, camera)
    prefix_loss = reconstruction_loss(prefix_render, image)
    
    # 4. Full rendering
    full_render = render(gaussians_sorted, camera)
    full_loss = reconstruction_loss(full_render, image)
    
    # 5. Summed loss
    loss = prefix_loss + gamma * full_loss
    loss.backward()
    
    # 6. Reorder after each step (since opacity is updated)
    # Automatically re-sorted in the next step
    return loss

# At deployment: just change the prefix ratio
def render_at_budget(gaussians, camera, ratio=0.5):
    opacities = gaussians.get_opacity()
    order = torch.argsort(opacities, descending=True)
    k = int(ratio * len(gaussians))
    return render(gaussians[order[:k]], camera)

Terminology

NeRFNeural Radiance Field — a technique for representing 3D scenes as neural networks that can render novel viewpoints.

Gaussian SplattingA 3D scene representation using 3D Gaussians that enables fast differentiable rendering.

PSNRPeak Signal-to-Noise Ratio — a standard metric for image quality, higher is better.

LOD (Level of Detail)A rendering technique where objects use simpler representations when far from the camera to save compute.

FLOPFloating Point Operation — a measure of compute cost.

Related Resources

Original Abstract (Expand)

The ability to render scenes at adjustable fidelity from a single model, known as level of detail (LoD), is crucial for practical deployment of 3D Gaussian Splatting (3DGS). Existing discrete LoD methods expose only a limited set of operating points, while concurrent continuous LoD approaches enable smoother scaling but often suffer noticeable quality degradation at full capacity, making LoD a costly design decision. We introduce Matryoshka Gaussian Splatting (MGS), a training framework that enables continuous LoD for standard 3DGS pipelines without sacrificing full-capacity rendering quality. MGS learns a single ordered set of Gaussians such that rendering any prefix, the first k splats, produces a coherent reconstruction whose fidelity improves smoothly with increasing budget. Our key idea is stochastic budget training: each iteration samples a random splat budget and optimises both the corresponding prefix and the full set. This strategy requires only two forward passes and introduces no architectural modifications. Experiments across four benchmarks and six baselines show that MGS matches the full-capacity performance of its backbone while enabling a continuous speed-quality trade-off from a single model. Extensive ablations on ordering strategies, training objectives, and model capacity further validate the designs.