NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput

NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput">

Large hybrid MoE models like Nemotron-3-Super are accurate but expensive to serve. Their active parameters, KV cache, and Mamba state cap how many users a node can hold at a given per-user token rate. NVIDIA AI team has released Nemotron-Labs-3-Puzzle-75B-A9B , a compressed variant of Nemotron-3-Super. The parent model has 120.7B total and 12.8B active parameters. The compressed model has 75.3B total and 9.3B active parameters.

The deployment target was fixed before the architecture search began. Target one was 2x server throughput at 100 tokens per second per user. Target two was 8 concurrent 1M-token requests on a single H100. Three checkpoints on Hugging Face: BF16, FP8, and NVFP4.

Nemotron-3-Super is a hybrid Mamba-Transformer MoE model. Puzzle-75B-A9B preserves the parent’s block layout exactly. It has 88 blocks: 40 Mamba, 40 MoE, and 8 attention blocks.

What changed is capacity inside those blocks:

The number of routed experts, the shared expert size, and the MoE latent size are unchanged. Attention layers were left untouched. The proposed research’s stated reason is that Nemotron-3-Super is already very KV-cache efficient. Mamba layers were pruned uniformly, because inference frameworks do not support a different SSM state size per layer.

The result is not a uniformly scaled-down teacher. The above figure shows the allocation across depth. Puzzle preserved capacity in selected middle and late layers, and cut hard elsewhere.

The below table reports Pareto-optimal total throughput on a single 8xB200 node, with single-step decoding.

Both models were served at matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The gap therefore reflects compression, not a change in numeric format. The prefill-heavy 50K/2K regime gains least. The decode-heavy 8K/64K regime gains most.

On a single 8xH100 node at UT = 100, the gains are smaller. They are 1.91x on 50K/2K and 1.82x on 8K/64K. Both models there use FP8 weights, FP8 KV cache, and FP32 Mamba state.

On a single H100 at 1M context, the binding constraint flips from compute to memory. Super’s NVFP4 weights occupy about 70 GB of the 80 GB HBM budget. Each 1M-token request adds about 4 GB of KV cache. Effective concurrency is therefore 1.

Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. Attention layout is unchanged, so per-request KV cost is unchanged. Concurrency at 1M rises to 8. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefill of a 990K-token prompt is about 1.2x faster.