NVIDIA Research has integrated EAGLE-3 speculative decoding directly into NeMo RL with a vLLM backend, delivering measured 1.8× rollout-generation speedup at 8B and a simulator-projected 2.5× end-to-end speedup at 235B. The work uses GRPO (Generative Reward Policy Optimization) and runs on 32 GB200 GPUs across 8 GB200 NVL72 nodes. The interesting bit isn't the speedup number — it's that they treat the RL rollout phase as a generation-bound problem and bring inference-stack optimizations to bear.
Losslessness is the load-bearing claim. The team argues mathematical equivalence: rejection sampling against the target model's distribution is provably equivalent to autoregressive generation from that model. They validate empirically by tracking AIME-2024 validation accuracy throughout training under both autoregressive and speculative regimes — the curves overlay. Reported acceptance lengths (tokens verified per draft) are 2.47 and 2.05 across two workloads (RL-Think for continued reasoning training, RL-Zero for from-base). The 235B 2.5× number is extrapolated through a proprietary GPU performance simulator calibrated to GB200-class compute, memory, and interconnect — not measured. Paper reference: arXiv:2604.26779.
For RL training infra, this is a real efficiency move. Rollout generation is the dominant time sink in modern RL pipelines — a reasoning rollout can be tens of thousands of tokens, repeated across thousands of trajectories per gradient step. Cutting that by 1.8× with a lossless guarantee means more samples per dollar, and the framing as "inference stack inside the RL trainer" is the architectural shift worth tracking. Expect this pattern (specDec, MTP heads, vLLM-style batching inside training) to land in TRL, OpenRLHF, and other open RL stacks within months. The ones that don't ship it become the slow ones.
If you train with NeMo RL, the speedup is in your hands; the integration is in the trainer. If you're on TRL or a custom RL stack, the EAGLE-3 plus native MTP path is documented well enough to port — the harder part is wiring the vLLM backend into your rollout phase without breaking gradient flow. The 235B projection is a simulator number, so don't budget capacity assuming it. The 8B measured number is real, and at the 8-32B scale where most fine-tuning happens, the speedup is take-home.