Venue: ICLR 2026 · arXiv: 2509.09674 · OpenReview: TQhSodCM4r Authors: Haozhan Li, Yuxin Zuo, … Bowen Zhou, Ning Ding et al. (PRIME-RL) — Tsinghua University, Shanghai AI Lab, Shanghai Jiao Tong University, Peking University, The University of Hong Kong Code: github.com/PRIME-RL/SimpleVLA-RL Category: RL for VLA (scaling) Trend tag: Trend 3

flowchart LR
  veRL[veRL infrastructure<br/>scalable RL framework] --> Adapt[VLA-specific adaptations]
  Adapt --> S1[VLA-specific trajectory sampling]
  Adapt --> S2[Scalable parallelization<br/>many envs in parallel]
  Adapt --> S3[Multi-environment rendering]
  Adapt --> S4[Optimized loss computation<br/>VLA-aware]
  S1 --> Train[Train RL on top of OpenVLA-OFT]
  S2 --> Train
  S3 --> Train
  S4 --> Train
  Train --> R[Results:<br/>SOTA LIBERO<br/>beats π0 on RoboTwin 1 & 2<br/>discovers 'pushcut' patterns]

See the SimpleVLA-RL GitHub README and the arXiv preprint (links below) for the authors' own diagrams of the framework and the pushcut analysis figures.

Vision-Language-Action models trained by supervised fine-tuning (SFT) face two fundamental limits:

1. Data scarcity — large-scale human-operated robot trajectories are expensive and slow to collect, so SFT scaling is rate-limited by teleop hours.
2. Generalization — SFT-only VLAs overfit the demonstration distribution and degrade on tasks involving distribution shift.

RL is the natural answer, but as of mid-2025 there was no general, efficient, scalable RL framework purpose-built for VLA models. Existing RL stacks (designed for game-playing or LLM RLHF) don't handle VLA's heavy multi-modal observations, action chunking, or the parallel-rollout requirements of robot envs.

SimpleVLA-RL is an efficient RL framework tailored for VLA models. It builds on veRL (the open-source scalable RL infrastructure used for LLM RL) and adds VLA-specific machinery:

• VLA-specific trajectory sampling — handles action chunks and multi-modal observations natively
• Scalable parallelization — many environments rolled out in parallel for sample throughput
• Multi-environment rendering — efficient batched rendering for sim envs
• Optimized loss computation — VLA-aware advantage estimation and gradient updates

The online RL optimizer is GRPO (Group Relative Policy Optimization, Shao et al. 2024 — normalizes advantages within trajectory groups with PPO-style clipping), with VLA-specific modifications and a sparse 0/1 task-success reward. The framework is applied on top of OpenVLA-OFT (it also supports vanilla OpenVLA) and trained with carefully tuned exploration-enhancing strategies that are necessary to avoid mode-collapse on robot tasks.

• State-of-the-art on LIBERO when applied to OpenVLA-OFT: ~91.6% → 99.1% average, with the largest gain on LIBERO-Long (86.5% → 98.5%, +12 pts).
• Extreme data efficiency. With only a single demonstration per task (one-trajectory SFT cold start), LIBERO-Long jumps from 17.3% → 91.7% after RL — surpassing the full-data SFT baseline, the headline evidence that RL substitutes for teleop data.
• Outperforms π0 on RoboTwin 1.0 and 2.0 with the exploration-enhancing strategies (e.g. RoboTwin 1.0 ≈70.4% vs π0 ≈58.1%; RoboTwin 2.0 ≈68.8% vs π0 ≈49.2%) — a notable comparison, since π0's flow-matching head is the production reference point.
• Surpasses SFT in real-world tasks — not just sim — on AgileX Piper dual-arm hardware (≈17.5% → 38.5% averaged over four tasks), reducing dependence on large-scale teleop data and improving distribution-shift generalization.
• Discovers the "pushcut" phenomenon — during RL training the policy independently discovers strategies absent from the SFT demonstrations, e.g. directly pushing an object to its goal pose instead of the demonstrated grasp-move-place routine, showing genuine exploration rather than mere refinement of the demonstrated behavior. This is strong evidence that RL adds new capabilities to VLAs rather than just polishing existing ones.

SimpleVLA-RL is the scaling-focused counterpart to the residual-RL story told by PLD and the production-engineering story told by RL Tokens. Where PLD asks "how do we get the most out of residual RL on a frozen base?" and RLT asks "how do we ship online RL on a real robot in hours?", SimpleVLA-RL asks "what is the right open-source infrastructure for VLA RL at scale?"

Three things make the paper consequential for the field:

1. Open infrastructure. veRL → SimpleVLA-RL is among the first credible open-source paths to do industrial-scale VLA RL outside Physical Intelligence and a few similar labs (the paper itself frames it as "one of the earliest systematic explorations of VLA online RL"). This will democratize RL fine-tuning of VLAs through 2026–2027.
2. The pushcut finding. "Pushcut" — the policy discovering manipulation patterns beyond the SFT distribution — is the empirical evidence the field needed that VLA RL is not just polishing but genuine capability discovery. This re-frames the question from "does RL help?" (yes, settled) to "what does RL discover?" (now an active research question).
3. Beats π0 on RoboTwin. The strongest open-data result against the production reference point. Combined with PLD's 99% LIBERO and 100% real-world numbers, the "SFT is enough for VLAs" position no longer holds.

• How do the gains scale with backbone size, demonstration count, environment diversity?
• Does pushcut show up on other tasks / other VLA backbones, or is it OpenVLA-OFT-specific?
• Can SimpleVLA-RL's framework host the residual-RL pattern (PLD / RLT) as well as full-policy RL?

• arXiv: https://arxiv.org/abs/2509.09674
• HTML version: https://arxiv.org/html/2509.09674v1
• GitHub: https://github.com/PRIME-RL/SimpleVLA-RL
• OpenReview: https://openreview.net/forum?id=TQhSodCM4r
• HuggingFace paper page: https://huggingface.co/papers/2509.09674

• PLD (residual-RL counterpart)
• RL Tokens (production-engineering counterpart)
• VLA-RFT (world-model alternative)
• π0.6 (the production reference π0 family)
• RL for VLA — section landing

← Back to ICLR-2026 · Topic: RL