Soofi Consortium Releases Soofi S 30B-A3B: An Open Hybrid Mamba-Transformer MoE Foundation Model For German And English

A German research consortium has published the pretraining report for Soofi S 30B-A3B . It is an open base model for German and English. Training ran end to end on Deutsche Telekom’s Industrial AI Cloud in Munich. Preview weights are on Hugging Face. It is worth noting that among some of the fully open base models tested, Soofi S records the highest English and German aggregate scores.

Soofi S is a Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model. It totals ~31.6B parameters and activates ~3.2B per token. As a base model, it has no instruction tuning, alignment, or safety tuning. The KI Bundesverband coordinates the consortium, funded by the German Federal Ministry for Economic Affairs and Energy. Participants include Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum.

The efficiency claim starts with the layer stack. The network holds 52 layers. That is 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention (GQA) layers. Only those 6 GQA layers maintain a KV cache. Each MoE layer holds 128 routed experts, activates 6 per token, and adds 2 shared experts. Other details: model dimension 2688, squared ReLU, RMSNorm, and no positional embeddings.

Soofi S adopts the Nemotron 3 Nano reference design without modification. The research team gives three reasons for that choice. Those are deployability on stacks such as vLLM, serving efficiency, and scientific control. Because the backbone is fixed, Nemotron 3 Nano becomes an architecture-identical baseline. The data recipe is the only moving part.

That recipe follows a Warmup–Stable–Decay (WSD) schedule with a minus_sqrt decay segment. Phase 1 consumed ~20T tokens on a diverse, quality-tiered mixture at a 1e-3 plateau. Phase 2 consumed ~6.58T tokens of high-quality annealing data. It decays 1e-3 to 1e-5, then continues at a constant 1e-5. Phase 3 consumed ~0.10T tokens at a 1,048,576-token sequence length. It extends the usable context window up to 1M tokens.

German is the deliberate variable. It rises from 7.2% of Phase 1 effective tokens to 15.32% in Phase 2. The reference Nemotron 3 Nano mixture allocates about 5% to all non-English languages combined. German sources include HPLT v3 and v4, German Commons, German FinePDFs, and FineWiki. Genios adds 193M articles from 916 newspaper and trade-press archives, commercially licensed.

Infrastructure follows the same sovereignty logic. The run used up to 512 NVIDIA B200 GPUs, from 24 March to 13 May 2026. It consumed ~253,000 B200 GPU-hours.

Those choices show up in the evaluation. Soofi S ran against 16 other open base models. All used the same lm-evaluation-harness pipeline, prompts, and few-shot settings.