NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule
AI News Desk
·
MarkTechPost
··
6 min read
Linear attention replaces the unbounded KV cache of softmax attention with a fixed-size recurrent state.
Linear attention replaces the unbounded KV cache of softmax attention with a fixed-size recurrent state. This cuts sequence mixing to linear time and decoding to constant memory. The hard part is not what to forget. It is how to edit a compressed memory without scrambling existing associations.
NVIDIA has released Gated DeltaNet-2 , a linear attention layer that targets that bottleneck. The model decouples the active memory edit into two channel-wise gates. It is trained at 1.3B parameters on 100B FineWeb-Edu tokens. It outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across the researchs benchmark suite.
A recurrent linear attention layer stores a matrix state S t and reads it with the query. DeltaNet adds an active edit by subtracting the value currently associated with the current key. It uses a scalar step size β t to control how much to overwrite. Mamba-2 adds a data-dependent scalar decay α t for global forgetting. Gated DeltaNet combined both operations, but both gates remained scalar per head.
Kimi Delta Attention (KDA) refines the decay side. It replaces the scalar α t with a channel-wise vector. KDA still keeps a single scalar β t for the active edit. That scalar controls two different things at once. It decides how much old content to erase on the key side. It also decides how much new content to commit on the value side. These two decisions act on different axes of the state. Tying them together is a modeling restriction, not a property of the delta rule.
Gated DeltaNet-2 separates the two decisions through Gated Delta Rule-2. It introduces a channel-wise erase gate b t ∈ [0,1] d k on the key axis. It also introduces a channel-wise write gate w t ∈ [0,1] d v on the value axis. Both gates are produced by sigmoid projections of the token representation. The update applies decay before the active edit.
S t = (I − k t (b t ⊙ k t ) ⊤ ) D t S t−1 + k t (w t ⊙ v t ) ⊤
Here D t = Diag(α t ) is the channel-wise decay carried over from KDA. The left factor of the erase matrix stays k t , preserving the delta-rule write direction. The right factor becomes b t ⊙ k t , making the read direction channel-selective. The write term k t z t ⊤ uses z t = w t ⊙ v t , making the value update channel-selective.
When both gates collapse to the same scalar β t , the update recovers KDA exactly. When the decay α t also collapses to a scalar, it recovers Gated DeltaNet. Both prior models are preserved as tied subspaces of the new update.
In the fast-weight view, Gated Delta Rule-2 is one online gradient step on a local regression loss. The decayed state stays close to memory, while the residual edit uses gated read and gated write targets.
Source: MarkTechPost
The recurrence admits a chunkwise WY form that matches the structure used by KDA. Cumulative channel-wise decay is absorbed into the two factors of each rank-one erase. The per-chunk update becomes a product of asymmetric matrices of the form I − k̄ r ē r ⊤ . The implementation uses chunk size C = 64 with fused Triton kernels.
For the backward pass, the scalar shortcut used by KDA no longer applies. The write side contains a different diagonal gate over value channels. The erase side contains a different diagonal gate over key channels. So the gate factors must appear inside the dot products that accumulate gradients. The paper derives this gate-aware vector-Jacobian product explicitly. On Hopper GPUs, the fused WY backward kernel is restricted to two and four warps to avoid a Triton WGMMA layout assertion.
Gated DeltaNet-2 is used as the recurrent token mixer in a standard Transformer-style block. Query and key paths use linear projection, short causal convolution, SiLU, and L2 normalization. The value path uses linear projection, short convolution, and SiLU. The decay α t , erase gate b t , and write gate w t come from separate linear branches. The recurrent output is RMS-normalized, multiplied by a SiLU output gate, and projected back.
A hybrid variant inserts Sliding-Window Attention (SWA) after the recurrent mixer. A repeated cell contains Gated DeltaNet-2, an MLP, SWA, and another MLP. SWA handles exact local interactions, while the recurrent mixer compresses long histories. The hybrid retains linear sequence scaling with a bounded attention cache.
All models are 1.3B parameters trained on 100B FineWeb-Edu tokens. Parameter count and recurrent state size are matched across models. The recurrent state holds 262,144 floats per layer per batch element. Training length is 4K tokens, and hybrid models use a 2K SWA window. The Mamba-3 MIMO baseline uses rank R = 4 .
On language modeling and commonsense reasoning, Gated DeltaNet-2 has the best average in both settings. The recurrent model averages 53.11 across LAMBADA and the reasoning suite. That sits above Mamba-3 MIMO at 52.39 and KDA at 52.28. In the hybrid setting, Gated DeltaNet-2 averages 53.97 against Mamba-3 MIMO at 52.72. Since recurrent state size is matched, the gain points to the update rule, not more memory.
The clearest gains appear on RULER long-context retrieval. In the recurrent setting, S-NIAH-2 at 4K rises from 89.0 (KDA) to 93.0. S-NIAH-3 at 2K jumps from 63.2 (KDA) to 89.8. MK-NIAH-1 at 4K climbs from 28.0 (KDA) to 37.8.
On real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP), Gated DeltaNet-2 also leads both settings. The recurrent average is 29.88 and the hybrid average is 42.28.
Decoupling Erase and Write in Linear Attention. A delta-rule recurrent attention layer with channel-wise erase and write gates.
Linear attention compresses an unbounded KV cache into a fixed-size recurrent state. Editing this memory without scrambling existing associations is the hard part.
Prior delta-rule models (Gated DeltaNet, KDA) tie erasing old content and writing new content to one scalar gate β_t .
Split it: a channel-wise erase gate b_t on the key axis, and a channel-wise write gate w_t on the value axis.
With erase gate b_t ∈ [0,1]^{d_k} , write gate w_t ∈ [0,1]^{d_v} , and channel-wise decay D_t = Diag(α_t) , the recurrent state evolves as:
The official PyTorch implementation ships with a Dockerfile, training scripts, and the lit_gpt model definitions.
The streamlined command from the official README. Replace placeholders with your dataset paths and config name.
Matched against Mamba-2, Gated DeltaNet, KDA, and Mamba-3 baselines under identical optimizer settings and recurrent state size.
Global batch 0.5M tokens · sequence length 4K · hybrid models use a 2K sliding-window attention size.
16 heads · d_k = d_v = 128 · per-layer recurrent state 262,144 floats, matched against Mamba-2/3.
Repeated cell: Gated DeltaNet-2 → MLP → SWA → MLP. The recurrent mixer compresses long histories; SWA handles local interactions.
Best average across language modeling and commonsense reasoning, with the largest gains on long-context retrieval.
Everything you need to read, run, and cite Gated DeltaNet-2 in one place.
Check out the Paper and Repo . Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule appeared first on MarkTechPost .