NVIDIA Releases Audex: A Unified Audio-Text LLM That Preserves Text Intelligence

NVIDIA has released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model. It understands and generates both audio and speech. It also keeps the text intelligence of its backbone.

The checkpoints, along with a smaller Audex-2B, are released under a noncommercial license. Most multimodal models pay a text tax. When labs add audio or vision output, text benchmarks often drop.

NVIDIA research team reports this even for speech-only output models. Audex is designed to avoid that regression. Audex is a single Mixture-of-Experts (MoE) Transformer decoder.

It has 30B total parameters and 3B activated per token. The backbone is Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM. That backbone is a hybrid Mamba-Transformer with 52 layers.

It uses 128 routable experts and 6 activated experts. The design is deliberately simple. Audio inputs are encoded and projected into the text embedding space.

Text tokens and quantized audio tokens are then treated uniformly during generation. There is no thinker-talker split and no stacked cascade of models. Because the design stays simple, Audex runs on standard LLM stacks.

These include Megatron-LM for training and vLLM for inference. It supports both an instruct mode and a thinking mode. Context length reaches 1M tokens.

Three components sit around the LLM backbone: Audex uses two codecs for output. Speech uses X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization (FSQ) with a 65,536 codebook.

Non-speech sound uses X-Codec at 200 tokens per second. It uses four flattened residual vector quantization (RVQ) layers. Complex sound gets a larger token budget than speech.

The interactive demo below computes these token counts for any duration. Audex needs no audio pretraining. It starts from the text-only SFT checkpoint.

Training then adds capabilities stage by stage. The multi-stage SFT curriculum runs in order: text SFT, audio warmup, audio generation, then audio understanding. During audio warmup, text token embeddings stay frozen.

Unfreezing them degraded text quality in ablations. NVIDIA research team also tested a single-stage recipe that mixes all data at once. That recipe broke long-context retrieval on NIAH.

Multi-stage training avoided this, so it became the default. After SFT, the research team applies text-only Cascade RL and multi-domain on-policy distillation (MOPD). Audio tasks show marginal or no regression after this text-only RL.