NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework

Agentic reinforcement learning research is constant algorithm modification. New estimators, new pipeline stages, new rollout schemes. In mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue. That cost lands on the researcher at every iteration.

Molt , from NVIDIA’s NeMo team, targets that cost directly. Its a PyTorch-native agentic RL framework with an unusual design target. The codebase should be compact enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety. The stated footprint is roughly 8.6K lines of RL code, measured by tracing the import graph from each framework’s RL entry point. The same method counts about 62K lines for verl, 25K for slime, and 7.2K for OpenRLHF.

Yes. Molt ships under Apache 2.0 with launch codes, Slurm scripts, and a prebuilt container. But the research paper positions it as research infrastructure, not a production training service, and hardware is the real gate. The shipped recipes assume 2 nodes of 8 H100 GPUs, split 8 for training and 8 for rollout.

That puts Molt in reach of frontier and frontier-adjacent labs, well-funded AI startups doing post-training, enterprise AI research groups in finance, healthcare, and robotics that train agents against proprietary environments, and academic labs with multi-node H100/H200 access. Applications include multi-turn tool-use agents, code-execution agents, vision-language environments (the shipped geo3k recipe), LLM-as-judge reward loops, and on-policy distillation onto a smaller student.

Molt composes Ray for placement and asynchronous queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. None of the three is forked, so upstream improvements arrive as a container pin rather than a rebase.

The runtime is an agent pool, a set of vLLM engines behind a request router, and a single trainable policy actor. A streaming pool keeps prompt groups in flight so engines never drain while the actor trains. Partial rollout pauses the engines, broadcasts actor shards over NCCL directly to each engine, and resumes retained requests instead of discarding them.

An RL run names one Python module that exports an AgentRunner . Everything else is ordinary code, including the reward.

Two forms are supported. With Env , the framework owns the LLM loop in a Gymnasium-aligned step() . With ChatAgent , the user owns the loop through a stock OpenAI or Anthropic SDK. Molt launches a loopback server that speaks both wire protocols, and every request decodes server-side into one token-exact accumulation. When a long-horizon agent compacts its context and rewrites the prefix, the server seals the current segment and opens a fresh one automatically.