Alibaba’s Qwen Team Releases Qwen3.8-Flash-Next: A 125B Multimodal MoE With 6B Active Parameters Previewing the Qwen4 Architecture

Multimodal MoE With 6B Active Parameters Previewing the Qwen4 Architecture">

Alibaba’s Qwen team has released Qwen3.8-Flash-Next , an open-weight multimodal Mixture-of-Experts model built for cost per token. The checkpoint pairs a 125B backbone with a 51B N-gram embedding table and a 4B multi-token prediction module. Only 6B parameters activate per token. The team positions it as an early preview of the architecture that will underpin Qwen4 , the same role Qwen3-Next played for Qwen3.5. Four changes carry the release: a Gated DeltaNet and Qwen Sparse Attention hybrid, Gated Residual, N-gram Embedding, and the Muon optimizer. Qwen team reports training cost at roughly one-ninth that of Qwen3.7-Plus.

Yes but not on a workstation. The FP8 checkpoint is 172.78 GiB and the BF16 checkpoint is 335.28 GiB. Per vLLM recipes , TP2 is the minimum validated FP8 configuration on GB300 and TP4 is recommended. On an 8×H200 node, use TEP8; plain TP8 is incompatible with the checkpoint’s 128-wide quantization blocks. Sparse activation cuts compute, not storage.

Qwen3.8-Flash-Next pairs a 125B main model with 51B N-gram embedding parameters and a 4B multi-token prediction module, totaling 180B on disk. Only 6B parameters activate per token. Four changes drive this:

The MoE layer carries 512 experts, activating 10 routed plus 1 shared, at expert intermediate dimension 640.

Qwen reports 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, and 91.9 on LiveCodeBench v6. On agentic tasks it posts 73.9 on CoWorkBench, 55.7 on JobBench, and 73.5 on Toolathlon Verified. Multimodal results include 84.5 on AndroidWorld, 76.6 on LVBench, 88.5 on RealWorldQA, and 95.7 on MathVision with code interpreter.

The model does not lead everywhere. Claude Opus 4.6 (Max) takes HLE at 40.0 against Qwen’s 35.9 , and DeepSeek-V4-Flash-0731 leads NL2Repo-Bench at 54.2 versus 48.1 . Frontier reasoning remains the gap.

Qwen states training cost roughly 1/9 that of Qwen3.7-Plus. On serving, the announcement cites QSA kernel speedups of up to 7.6× prefill and 4.9× decode at 1M tokens, while the SGLang cookbook and vLLM recipes cite 10.2× and 6.6× . Treat the range as vendor-reported until independently measured. Qwen also reports 8.6× the prefill throughput of Qwen3.7-Plus at a 90% prefix-cache hit rate.

Context is 262,144 tokens natively, extensible to 1,000,000 with YaRN.

The model serves through vLLM , SGLang , TokenSpeed, transformers serve , and llama.cpp for GGUF quants. Fine-tuning is supported via Unsloth , Swift , and LLaMA-Factory . It already powers the “Standard” mode on QwenWork and works with Qwen Code .