Three Components, One Loop: Inside NVIDIA's New Agentic RL Framework

Agentic-first RL training framework from NVIDIA! Molt is a PyTorch-native RL training framework built for agentic research. Three components make up the entire stack: Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. That's it. The agentic-first design means reward is any Python you write. A mathematical grader, an LLM-as-judge, a multi-turn tool evaluator, a VLM environment. You define what "good" means inside an 'Env' or 'ChatAgent' class. No pretrained reward model required. Two agent interfaces cover different workflows. The 'Env' interface is Gymnasium-style - the framework owns the LLM loop and your `step()` returns a reward. The 'ChatAgent' interface gives you the loop via the OpenAI or Anthropic SDK, so any external harness that already speaks those APIs works without modification. The stack is intentionally minimal. One trainable actor, optional KL reference workers, fully async rollout and training overlap. Small enough to read end-to-end and understand every gradient. The same script that trains an 8B model scales to 1T-class MoE. DeepSeek-V3-class actors run at `ep_size 256` with TP, EP, and CP parallelism plus Adam CPU offload. No rewrite between scales. Supported algorithms: REINFORCE, REINFORCE baseline, RLOO, GRPO, DR-GRPO, GAE (PPO), and on-policy distillation. Key capabilities: • Reward is any Python function - graders, LLM-as-judge, multi-turn tools, VLM environments • Fully async rollout, training, and weight sync overlap • Scales from 8B to 1T-class MoE with the same script • Two agent interfaces: Env (Gymnasium-style) and ChatAgent (OpenAI/Anthropic SDK) • REINFORCE, RLOO, GRPO, DR-GRPO, GAE, on-policy distillation 100% open source I've shared the link in the replies!

Excellent — this checks out thoroughly against primary sources.

Fact-Check

This post is highly accurate and matches NVIDIA's own technical report and GitHub repo closely.

Confirmed accurate:

• Molt is real — an open-source, PyTorch-native RL training framework for agentic research, released by NVIDIA (via NVIDIA-NeMo) around July 22, 2026. Confirmed via the official GitHub repo (NVIDIA-NeMo/labs-molt), the accompanying technical paper on arXiv, and independent tech coverage.
• The three-component stack — Ray (placement/async queues), vLLM (rollout), NVIDIA AutoModel with FSDP2 (training) — is exactly how NVIDIA's own paper and repo describe it, down to the "that's the whole stack" framing.
• "Reward is any Python you write" — confirmed; the framework explicitly avoids requiring a pretrained reward model, letting users define reward logic inside Env or ChatAgent classes.
• Two agent interfaces — the Gymnasium-style Env interface and the ChatAgent interface (compatible with OpenAI/Anthropic SDKs via a loopback capture server) — both confirmed in the technical report and repo.
• Single trainable actor + optional KL reference design, emphasizing readability and minimalism — confirmed directly from both the paper and repo descriptions.
• Scaling claim — the same script scaling from 8B to 1T-class MoE, with DeepSeek-V3-class configurations using --fsdp.ep_size 256, TP/EP/CP parallelism, and Adam CPU offload — confirmed verbatim in NVIDIA's own materials.
• Supported algorithms — REINFORCE, REINFORCE with baseline, RLOO, GRPO, DR-GRPO, GAE (PPO), and on-policy distillation — this exact list of seven estimators is directly confirmed in the GitHub repo and technical report.
• Open source — confirmed (public GitHub repo, publicly available technical report).

Minor precision notes:

• Codebase size is described in coverage as "~8,600 lines" in some reporting and "~9.2K lines" in the GitHub repo's own description — a small discrepancy between sources, likely reflecting slightly different snapshots/counting methods, not a meaningful inaccuracy in the post (which doesn't cite a specific line count itself).
• Independent benchmarking (via a third-party comparison against the "slime" framework) found Molt's training step times statistically comparable to a Megatron-based stack — supporting the "performance parity" framing implied by the post, though the post itself doesn't make a specific speed claim, so this is just supporting context.

Verdict: This is an unusually well-corroborated post — nearly every specific technical claim (components, interfaces, scaling numbers, algorithm list) matches NVIDIA's own primary documentation almost word-for-word.