GDN-2-3B leak: a Nemotron-3 Nano hybrid with Gated DeltaNet-2 in place of Mamba-2

A single sentence posted on X on September 26, 2026 is the entire public record of GDN-2-3B so far. The account @teortaxesTex wrote that "one near-term release candidate is the hybrid GDN-2-3B latent MoE, which follows the Nemotron-3 Nano architecture but replaces the Mamba-2 layers with GDN-2." That is it — no Hugging Face repository, no config file, no parameter table, no named builder, no date. GDN-2-3B has not been released, and nothing below should be read as though it had. What makes the line worth unpacking anyway is that both halves of it name something real and checkable: Gated DeltaNet-2 is an NVIDIA-published linear-attention architecture with a public PyTorch implementation and a busy porting trail through TPU, Triton and ONNX tooling, and Nemotron-3 Nano is NVIDIA's shipped open-weights Mamba-2 hybrid that the rumoured model is being grafted onto. This is a what-we-know-so-far piece: the architecture is documented, the model is a rumour, and the difference between those two things is the whole story.

The short version: Gated DeltaNet-2 changes how a linear-attention model edits its compressed memory, and its measured gains land hardest on exactly the long-context retrieval work a small hybrid is bought for. Nemotron-3 Nano is the smallest tier in NVIDIA's current hybrid family and the one most likely to be re-cut with a cheaper recurrence. Put those together and you have a plausible release candidate — and precisely one person saying so.

What the tweet does and does not say

Read the sentence closely, because it is unusually dense and unusually thin at the same time. It says the candidate is hybrid (so, more than one layer type), 3B (a parameter count small enough to run on one consumer GPU), and a latent MoE (an NVIDIA expert-routing design in which tokens are projected into a smaller latent dimension before they reach the experts, which is what the Super 120B and Ultra 550B tiers use and what the shipped Nano tier does not). It says the layer pattern follows Nemotron-3 Nano. And it says the Mamba-2 layers are replaced with GDN-2.

What it does not say: who is building it. "One near-term release candidate" has no subject. It is tempting to read NVIDIA into it — GDN-2 is NVIDIA research and Nemotron-3 Nano is an NVIDIA model, so the pairing looks like a house experiment — but the tweet does not say that, and a third party can legally combine the two today, because the Nemotron-3 Nano weights are open and the Gated DeltaNet-2 reference code is public. Treat the builder as unknown.

Nor does it say when. "Near-term" is the only timing signal, and it is not a date. There is no checkpoint to download, no inference engine that claims to serve it, and no benchmark of the model itself anywhere. Every figure in this article belongs to Gated DeltaNet-2 or to Nemotron-3 Nano as separately published. None of them belongs to GDN-2-3B.

The two halves of the name, and why they fit

Gated DeltaNet-2 in one minute

Linear attention replaces the unbounded KV cache of softmax attention with a fixed-size recurrent state. The hard part is not deciding what to forget — it is editing that state without scrambling associations that are already stored in it. Delta-rule models subtract the current read before writing a new value; Kimi Delta Attention sharpened the forgetting with channel-wise decay. Both, though, still drive two different decisions off one scalar gate: how much old content to erase on the key side, and how much new content to commit on the value side.

Gated DeltaNet-2, published by NVIDIA researchers Ali Hatamizadeh, Yejin Choi and Jan Kautz (arXiv 2605.22791, reference code in the NVlabs repository), splits that scalar into two channel-wise gates — an erase gate over key-side coordinates and a write gate over value-side coordinates — and keeps the channel-wise decay. The paper's framing is that the design strictly generalises what came before: collapse both gates to the same scalar and you recover Kimi Delta Attention; collapse the decay as well and you recover the earlier Gated DeltaNet. In ablation, NVIDIA reports that the erase gate accounts for most of the gain, which fits its role in protecting key-side associations from being overwritten.

The evidence is a matched-parameter study, not a product: every model trained at 1.3B parameters on 100B FineWeb-Edu tokens, with recurrent state size held equal across Mamba-2, Gated DeltaNet, KDA and Mamba-3. In the recurrent setting Gated DeltaNet-2 posts the best WikiText perplexity of the group at 15.90 against Mamba-2's 16.79, and the best average commonsense accuracy at 53.11 against Mamba-3's 52.39. In the hybrid setting — the one that actually resembles Nemotron-3 Nano's interleaved pattern — it is 15.62 WikiText perplexity and 53.97 average accuracy, ahead of Mamba-3 (15.81 / 52.72) and well ahead of a plain Transformer baseline (19.22 / 50.86).

Where the advantage concentrates is the part that matters for a small long-context model. On RULER's recurrent multi-key needle-in-a-haystack test at 4K, Gated DeltaNet-2 scores 37.8 against 27.8 for the older Gated DeltaNet and 28.0 for KDA; on single-needle-at-2K it scores 89.8 against 54.2. Averaged across six real retrieval sets (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP) it leads the recurrent frontier at 29.88 against Mamba-2's 26.84, and the hybrid frontier at 42.28 against 40.14 for KDA. NVIDIA also reports near-flat throughput scaling with sequence length on a single H100, with only a small constant overhead over KDA for the extra gates. These are vendor figures from the paper's own tables, unreproduced by us.

The Nemotron-3 Nano blueprint

Nemotron-3 Nano is a Mixture-of-Experts hybrid Mamba-Transformer, and it is the architecture being borrowed rather than the model. The 30B-A3B release is 52 layers: 23 Mamba-2 layers, 23 MoE layers, and six grouped-query attention layers, with 128 routed experts plus one shared expert per MoE block and six experts active per token. It carries 3.5B active parameters against 30B total, was pretrained on 25 trillion tokens, and supports context windows up to 1M tokens; NVIDIA's own technical report claims up to 3.3x higher inference throughput than similarly sized open models such as GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507. It is released under the NVIDIA Nemotron Open Model License and is explicitly cleared for commercial use.

There is a smaller sibling worth knowing about, because "3B" in the rumoured name lands close to it: Nemotron-3 Nano 4B, compressed from NVIDIA-Nemotron-Nano-9B-v2 using NVIDIA's Elastic framework, is "primarily Mamba-2 and MLP layers combined with just four Attention layers." So the Nano tier is already the part of the family that gets squeezed and re-cut. That is the single most plausible reason a 3B experimental variant would exist at all.

Two mismatches are worth flagging rather than smoothing over. First, in the shipped family it is the large tiers — Nemotron-3 Super 120B-A12B and Nemotron-3 Ultra 550B-A55B — that use latent MoE; the Nano tier does not. A "Nano-shaped latent MoE" is therefore not a straight port of an existing NVIDIA config but a recombination of two tiers' ideas, which is exactly the kind of thing that shows up in research checkpoints before it shows up in a product. Second, the Nano family's MoE blocks are dense-expert routed; moving to latent routing changes the FLOP profile of every expert layer, so a 3B label would tell you very little about what the model actually costs to serve until a config file appears.

The porting trail is real — the model is not

What can be verified is that Gated DeltaNet-2 is being fitted to production runtimes at speed. On September 23, 2026, a pull request to OpenXLA's Tokamax repository added a Gated Delta Net 2 Pallas kernel and operator for TPU, including an autograd backward pass across six inputs and benchmark numbers on Cloud TPU v7x. Flash Linear Attention has carried fused GDN-2 prefill kernels since late June — the pull request there reports an average 2.48x prefill speedup and a 5.13x best case on an H100 — with follow-ups in September fixing a gate-ordering bug and optimising the chunk-training path. ONNX has an open specification gap filed against it, because opset 27's LinearAttention operator cannot express GDN-2's channel-wise erase gate. And NVIDIA's own Megatron-Bridge added FLOPs accounting for both hybrid GDN layers and latent-MoE compression in March.

None of that is a model release, and it would be a mistake to read it as one. Kernel work is infrastructure; it says an architecture is being taken seriously, not that a checkpoint exists. What it does give you is a concrete thing to monitor: if a GDN-2 hybrid does reach the world, it will arrive as support landing in a serving engine first and as an announcement second, and the engine support is already partly in place. The Tokamax and Flash Linear Attention work is also the strongest argument that the pairing in the tweet is technically coherent rather than invented — the pieces needed to serve a GDN-2 hybrid are being built by people who are not the person who wrote the tweet.

Why a 3B GDN-2 hybrid would matter if it ships

The case for the combination is economic rather than benchmark-driven. A 3B-class hybrid with a fixed-size recurrent state is a model you can run on one consumer GPU without a KV cache that grows with your prompt, which is the same bargain Nemotron-3 Nano 4B already makes. Swapping the Mamba-2 layers for GDN-2 buys, on the paper's numbers, better multi-key retrieval and better real-world retrieval averages at matched state size — the two places the older delta-rule family was weakest. That is a targeted upgrade, not a generational one, and it is the sort of change that would be worth 3B of parameters.

It is also a reminder that the interesting competition in small models has moved from parameter counts to memory design. A 3B model whose recurrent state is a fixed tensor behaves differently under long prompts than a 3B transformer with a quantised KV cache: memory is flat, but the model has a fixed budget for what it can remember, and how gracefully it forgets is now the spec. Gated DeltaNet-2's whole contribution is a better forgetting rule. That makes it a design worth watching on its own terms even if GDN-2-3B never appears under that name.

How you would actually test something like this

When an unproven architecture reaches a serving runtime, the blocker for most teams is not the benchmark table — it is that adopting it means a new integration, a new contract and a new failure mode, all against a model with no production history. That is a routing problem before it is a model problem. An aggregator that puts a new endpoint behind the same key you already use means you can send a slice of real traffic at it, keep your existing model as the fallback, and let automatic failover catch the errors while the new route is still unsteady — one API across 200+ models at provider list price with 0% markup, so the pricing you were quoted is the pricing you pay and a vendor cut shows up the same day rather than at the next invoice. GDN-2-3B itself is not hosted anywhere, by us or anyone else; that is what "unreleased" means. The point is the pattern you would use on the day it is.

If you want to see which hybrid and long-context models are routable today, the live model catalogue is the honest list — it marks what is actually servable rather than what has been announced.

What to watch, and what would falsify this

The leak resolves in one of three ways, and each has a tell:

• A config file — the single strongest confirmation would be a Nemotron-H-style configuration listing GDN-2 layer types in a hybrid override pattern. Until a config.json like that exists, everything is inference from a sentence.

• Engine support for a specific checkpoint — GDN-2 kernels already exist; what does not exist is any engine claiming to serve a named GDN-2 hybrid. That gap closing is the real signal.

• A licence change — this one is easy to miss. The Gated DeltaNet-2 reference code is released under the NVIDIA Source Code License-NC, which is non-commercial, while Nemotron model weights ship under the NVIDIA Nemotron Open Model License, which is not. A hybrid that combined the two would have to resolve that tension, and how it resolves tells you whether the result is a research artefact or something you could deploy.

Two things would falsify the framing here. If the "3B" in the name turns out to mean something other than total parameters — active parameters, or a layer count, or simply a codename — the economics change completely. And if the "latent MoE" phrase turns out to describe a plain MoE block, then this is a much smaller idea than it looks: a Nano re-cut with a different linear layer, not a new shape.

Questions this raises

How is Gated DeltaNet-2 different from the Gated DeltaNet already inside Qwen3.8?

The earlier Gated DeltaNet uses a single scalar gate for both erasing and writing; Gated DeltaNet-2 splits it into two channel-wise gates and adds channel-wise decay borrowed from Kimi Delta Attention. So it is a strict generalisation rather than a replacement, and the practical difference shows up in retrieval, not in perplexity — the gap on multi-key needle-in-a-haystack is far wider than the gap on WikiText.

Why would anyone swap Mamba-2 for GDN-2 rather than just shipping more Mamba-2 layers?

Because at matched recurrent state size the paper measures Gated DeltaNet-2 ahead on the retrieval tasks that long-context agentic workloads actually exercise, at the cost of a small constant overhead in throughput. If your workload is retrieval-heavy, that trade is favourable; if it is throughput-bound at short context, the Mamba-2 baseline is the cheaper answer. A 3B testbed is a sensible way to find out which one your traffic looks like.

Does the open-source status of the pieces mean the model would be open too?

No. Nemotron-3 Nano's weights are open and Gated DeltaNet-2's reference code is public, but neither fact obliges anyone to publish a checkpoint built from them — and the non-commercial licence on the GDN-2 code means a commercial release would need a different arrangement. The plausible outcomes range from an NVIDIA research release under the Nemotron Open Model License to something that never leaves an internal cluster.

Is there anything to try today?

Yes, but not GDN-2-3B. The Gated DeltaNet-2 paper's checkpoints are reproducible from the NVlabs repository at 1.3B on FineWeb-Edu, and Nemotron-3 Nano 30B-A3B and 4B run on vLLM, SGLang, TensorRT-LLM and llama.cpp today. Testing either half tells you what the other half would probably do — which is more than the tweet gives you.

None of this makes GDN-2-3B real. It makes it falsifiable, which is the most a single sentence can offer: one config file, one engine claim, or one repository away from being either a genuine near-term release or a plausible-sounding recombination that never gets built.