AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM With 2.8B Active Parameters Trained On Instinct GPUs

AMD released Instella-MoE-16B-A3B , a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. The model holds 16B total parameters but activates only 2.8B per token. AMD is publishing weights from every training stage, along with data mixtures, training configs, and inference code. Two systems-level choices carry the release: Gated Multi-head Latent Attention and FarSkip-Collective connectivity.

Partly. The weights ship under a ResearchRAIL license for academic and research purposes only , so this is not a drop-in commercial model. The training codebase is MIT licensed, and that is the more reusable asset here.

Instella-MoE is a decoder-only MoE with 27 layers, hidden size 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer uses 2 shared experts plus 6 routed experts selected from 64. That yields 2.8B active parameters against 16B total. A Multi-Token Prediction objective is used during pre-training and mid-training.

There are two structural choices that are important to know. Gated MLA adds a lightweight learned output gate to Multi-head Latent Attention. A dedicated linear projection derives an input-conditioned gate, applied multiplicatively before the output projection. FarSkip-Collective passes outdated and partial activations into the MoE and attention layers, overlapping expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token when serving with expert parallelism.

Pre-training covers 7.1T tokens from open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. Mid-training uses Dolma3 Dolmino 100B across three data variants, merged by weight averaging. A long-context stage extends the window from 4K to 64K using YaRN, an increased RoPE theta, and document masking.

Post-training runs SFT on Dolci-Think-SFT-7B plus Nemotron mixtures, ending on a feedback-driven 512K-example set targeting measured weaknesses. DPO follows, with router bias updates and the auxiliary load-balancing loss disabled to prevent degradation. RL runs in the Miles framework: 1,400 steps of instruction-following RLVR, then Multi-Teacher On-Policy Distillation to fold that gain back without losing math or code.

The base checkpoint averages 76.7, the strongest among fully open models, ahead of Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9). It trails Qwen3.5-4B-Base (79.5). It leads on WinoGrande (86.5) and scores 65.7 on HumanEval+. Long-context averages are 41.5 on HELMET and 79.4 on RULER.

Post-training climbs from SFT (71.58) to DPO (72.67) to Think (73.22), above Olmo3-7B-Think (71.97), Gemma-4-E4B think (70.47), and Qwen3.5-4B (69.73). IFEval rises from 77.08 to 83.70.