Paper: VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models · ICLR 2026 (poster) · arXiv: 2601.03309 (v1 Jan 6, 2026 · v2 May 30, 2026) Affiliation: Tsinghua IIIS (Jianke Zhang, Xiaoyu Chen, Yanjiang Guo, Yucheng Hu, Jianyu Chen) × Qwen Team, Alibaba (Qiuyue Wang, Mingsheng Li, Jiajun Zhang, Shuai Bai, Junyang Lin) OpenReview: https://openreview.net/forum?id=tc2UsBeODW Related summary: VLM4VLA

This is the long-form review page complementing the VLM4VLA summary. All numbers and claims below are verified against the latest revision (arXiv 2601.03309v2, May 2026); the only substantive v1→v2 change is a corrected π0 Calvin Task-3 value (0.786 → 0.686, fixing an internal sum inconsistency — the 3.509 total was always correct).

Everyone assumed the recipe for a better VLA was "use the strongest available VLM." VLM4VLA runs the controlled experiment and finds: general VLM benchmark scores are a poor predictor of downstream manipulation performance. Kosmos-2 (1.7B) matches or beats Qwen3VL-30B-A3B (31B) on SimplerEnv. Fine-tuning VLMs on supposedly-helpful embodied auxiliary tasks (Robopoint, Vica, BridgeVQA, RoboBrain2) consistently hurts downstream VLA performance. Ablations locate the bottleneck: it's the vision encoder, not the language component. Freezing the vision encoder during VLA training is catastrophic (−42 points on SimplerEnv for Paligemma-1), while freezing word embeddings barely matters (±0.2 points). Fine-tuning the VLM's vision encoder on action-labeled real-world data closes the gap by +18 points. Bottom line: VLM competence is necessary but not sufficient for VLA competence, and the visual representations optimized for VQA are not the ones needed for low-level control.

"The majority of existing VLA methods have focused on developing more advanced network architectures … Limited attention has been given to a fundamental question at the core of VLA: how do the choice and specific capabilities of the underlying VLM affect the performance of VLA policies?"

Practitioners have been picking VLM backbones heuristically — "use the newest/largest/best-MMMU-score model." This paper is the first systematic controlled study asking whether that heuristic is even valid.

Figure 1 of the VLM4VLA paper (Zhang et al., ICLR 2026, CC-BY-4.0). Left: the evaluation pipeline, which fine-tunes different VLM backbones on robot data and evaluates on downstream tasks, plus an optional fine-tuning stage on auxiliary embodied tasks. Top right: systematic investigation of three axes influencing VLM-to-VLA transfer — the choice of VLM backbone, the impact of fine-tuning on auxiliary embodied tasks, and the influence of different training strategies (Frozen vs. fine-tuned different VLM modules). Bottom right: visualization of inconsistent performance of various VLM backbones across downstream tasks.

Figure 2 of the VLM4VLA paper. The minimal adaptation harness: a learnable action query token is appended after vision and text tokens; the VLM backbone processes the sequence; the action query output is decoded by a small MLP into an action chunk. All VLM parameters are tuned (vision encoder, LLM, word embeddings), and only <1% new parameters are introduced — deliberately minimal so comparisons isolate the effect of the VLM backbone itself.

flowchart TB
  subgraph Setup[Controlled matched-harness study]
    V1[Qwen2.5VL-3B]
    V2[Qwen2.5VL-7B]
    V3[Qwen3VL-2B / 4B / 8B / 30B-A3B]
    V4[Paligemma-1 / Paligemma-2]
    V5[Kosmos-2]
  end
  V1 & V2 & V3 & V4 & V5 --> H[VLM4VLA harness<br/>< 1% new params<br/>action query token + MLP head]
  H --> T{Train on robot data<br/>identical hyperparameters}
  T --> B1[Calvin ABC-D<br/>long-horizon task chains]
  T --> B2[SimplerEnv Bridge<br/>real-to-sim]
  T --> B3[Libero-Long<br/>10 long-horizon tasks]
  B1 & B2 & B3 --> R[Compare VLM's VQA score<br/>vs. VLA success rate]
  R --> F[Finding: r≈0.84 on Calvin,<br/>r≈-0.36 SimplerEnv,<br/>r≈-0.19 Libero]

  classDef result fill:#ffe8c2,stroke:#b47820,color:#000
  class F result

To compare VLMs fairly, the authors deliberately avoid architectural bells & whistles. The harness introduces:

• One learnable action query token, appended after the image and language tokens.
• A small MLP policy head that decodes the VLM's output at the action-query position into actions.
• No proprioceptive state — visual + language inputs only (fair, reproducible, and isolates VLM effects).
• No diffusion / flow matching — they deliberately avoid this because stochastic sampling inflates evaluation variance.

Action decoding: $$a = \text{MLP}\left(\text{VLM}([\langle\text{img}\rangle \ldots \langle\text{text}\rangle \langle\text{ActionQuery}\rangle])\right)$$

$$\mathcal{L} = \frac{1}{|\mathcal B|} \sum_{\mathcal B} \left(|a^{\text{pos}} - \hat a^{\text{pos}}|_2^2 + \text{BCE}(a^{\text{end}}, \hat a^{\text{end}})\right)$$

• $a^{\text{pos}}$: relative end-effector position (regression).
• $a^{\text{end}}$: discrete gripper open/close (BCE).
• Pure maximum-likelihood imitation learning.

• All VLM parameters fine-tuned (vision encoder, LLM, word embeddings).
• New learnable parameters added by the harness: <1% of the VLM's parameter count.
• Single-view images at 224×224 (resized to model's native resolution if needed).
• One unified hyperparameter set across all VLMs to guarantee fair comparison.

The abstract frames the study as evaluating "24 different VLMs that are either zero-shot or fine-tuned" — i.e. the 9 distinct base backbones below plus their auxiliary-task fine-tuned variants (§6.2). The main Calvin/SimplerEnv/Libero comparison uses these 9 open-source base backbones spanning 1.7B–31B parameters:

• OpenVLA (Llama2-7B, discrete action tokens)
• π0 (Paligemma-1 + flow-matching action expert; modified to remove proprioception, ~3.1B)
• ThinkAct (Qwen2.5VL-7B, RL-enhanced; uses proprioception — a confound in direct comparisons)

Headline: Qwen3VL-2B (smallest Qwen3) is the overall winner at 4.142, beating the 31B MoE (4.075), 8B (4.035), and 7B (4.057). Paligemma-1 + VLM4VLA harness (3.506) matches π0's full flow-matching pipeline (3.509) — strong sign that architectural sophistication matters less than backbone choice.

Headline: Kosmos-2 (the smallest model at 1.7B) wins SimplerEnv at 60.4%, tied with π0. Qwen3VL-30B-A3B (31B, MoE) lags at 44.8%. Libero-Long is won by ThinkAct (70.9%), but ThinkAct cheats by using proprioceptive state; without state, Qwen3VL-2B leads at 55.8%.

Linear regression across all 9 VLMs:

This is the core finding distilled into three numbers: strong VQA scores do not predict strong VLA performance on the benchmarks that measure actual manipulation.

Pretraining is necessary — training from scratch collapses performance. So VLM pretraining is necessary; it's just not sufficient, and stronger pretraining isn't strictly better.

Freeze the vision encoder (or word embeddings) during VLA training; retrain; compare:

Interpretation: Freezing the vision encoder is catastrophic (21–42 points lost on SimplerEnv). Freezing word embeddings is essentially free (±0.2 points). The vision encoder is the primary bottleneck — and fine-tuning the vision encoder matters more than just increasing the LLM's trainable parameters (frozen-vision 7.6B-tunable Qwen2.5VL-7B loses to fully-tuned 3.8B Qwen2.5VL-3B).

Fine-tune each VLM on a supposedly-helpful embodied auxiliary task before running the VLM4VLA harness; measure Calvin delta:

Every auxiliary task hurts. Best case (VQA-Mix) is −0.079. Worst (Omni-Generation) is −0.181. Most show increased variance on top of the degradation. The intuitive "embodied pretraining helps VLA" assumption fails empirically.

Fine-tune Qwen3VL-4B's vision encoder on real-world BridgeV2 images with action information encoded as FAST tokens, then run VLM4VLA:

Interpretation: Fine-tuning VLM without touching the vision encoder doesn't help. Fine-tuning that does touch the vision encoder gains +18.1 points when VLA-time vision is frozen and +3.1 points when VLA-time vision is unfrozen. The gap between "VLM" and "VLA" is not about visual domain (sim vs. real) — it's about which visual features are aligned with low-level control.

Raising input resolution from 224 → 512 → 768 doesn't improve Calvin scores when vision is unfrozen, and widens the frozen-vision gap (−1.00 → −1.12 → −1.31 for Qwen2.5VL-3B). Suggests frozen high-resolution vision leaks more spurious correlations.

1. No real-world robot experiments. "A limitation of our work is the absence of experiments on physical robots." They argue fairness+reproducibility on shared sim benchmarks outweighs this, and note their Table-4 experiments with real-world Bridge images show that sim-vs-real visual gap is not the primary bottleneck.
2. Simulation-only. This prevents other researchers from directly comparing physical-robot VLAs to VLM4VLA findings.
3. VLM coverage is limited to 1B–10B parameter open-source models plus one MoE (Qwen3VL-30B-A3B). No proprietary frontier models, no larger dense models.
4. Auxiliary task sampling is not exhaustive. 7 tasks tested, findings may be dataset-specific.
5. Unified hyperparameters. Required for fairness, but may not be individually optimal for each architecture.

• Regression loss + no diffusion simplifies evaluation variance but also handicaps every VLM equally against π0 (which has a flow-matching expert). The takeaway "Kosmos-2 matches π0" is harness-conditional — with diffusion/flow heads the ranking may shift.
• Deterministic MSE + BCE loss is probably underfit for multimodal action distributions. The paper implicitly argues the ranking is still informative, which is plausible but not proven.
• Calvin benefits QwenVL, SimplerEnv benefits Paligemma/Kosmos-2 — implicit evidence that benchmarks themselves select for different VLM capabilities. The paper frames this as a negative finding; a more positive reading is that different VLMs are specialists for different embodied skill sets, and the real research question is how to identify which VLM suits which benchmark.
• No vision-encoder architecture comparison. Kosmos-2 uses CLIP; Paligemma uses SigLIP; Qwen uses a custom ViT. The paper ablates tuning but doesn't compare vision encoder families directly.
• No study of how many action-conditioned vision-encoder tokens are needed to close the gap (the +18.1 result uses FAST tokens but the amount is not quantified cleanly in the body).
• The paper's diagnostic is strong; the prescription is weak. It correctly identifies the bottleneck but only gestures at solutions (action-aware VLM pretraining). Nobody has delivered a backbone trained end-to-end for VLA use that outperforms repurposed VQA VLMs.

1. The VLM-selection heuristic is broken. "Pick the biggest/best-MMMU model" does not optimize VLA performance. Actually benchmark each candidate on your manipulation task.
2. Vision encoder, not language, is the lever. Freezing word embeddings is nearly free; freezing vision is catastrophic. If you're cost-cutting, freeze the LLM, not the ViT.
3. Embodied VQA pretraining is a red herring. Robopoint, Vica, BridgeVQA, Robo2VLM, RoboBrain2, Omni-Generation — every auxiliary task tested hurts downstream VLA performance. Training a VLM on "robot-adjacent" data without touching the vision encoder with action signal is counterproductive.
4. Action-supervised vision-encoder fine-tuning is the missing recipe. +18.1 points is the largest positive effect in the paper, and the only intervention that consistently helps.
5. Counterintuitive best-in-class: Kosmos-2 (1.7B, 2023-vintage) matches the π0 production system on SimplerEnv. Something about its visual representation is better-aligned with low-level control than strong modern VLMs. The paper doesn't explain why; this is an open research question.

• What does a VLM trained for VLA look like? The paper argues current VLM pretraining objectives are misaligned with action grounding, but doesn't propose a concrete alternative objective.
• Is there a cheap proxy benchmark that actually predicts downstream VLA performance? VQA scores don't; the paper doesn't suggest a replacement.
• Does the result hold under diffusion / flow-matching heads? The MSE harness isolates VLM effects but may not translate directly to production flow-matching VLAs.
• Does the result hold at frontier scale (100B+ VLMs)? Only sub-31B tested.
• Why does Kosmos-2 punch above its weight? Possible hypotheses: it was explicitly trained for grounding (bounding-box tokens), it has denser visual tokens, its architecture is smaller and less distractible. Needs follow-up work.

• Qwen Team's VLA Program — cross-paper review reading this paper as the diagnostic foundation of the Qwen VLA line (Qiuyue Wang / Mingsheng Li / Shuai Bai later authored Qwen-VLA)
• VLM4VLA summary — short version of this page
• ManipBench — the earlier CoRL 2025 paper arguing the same thing via a VLM benchmark on low-level manipulation reasoning
• UniVLA — action-as-first-class-modality direction, a possible answer
• π0.6 — uses Gemma3-4B (not in VLM4VLA's comparison set); informative reference point for production VLAs
• Survey: VLA & Manipulation (ICLR 2026) — where VLM4VLA is flagged as the "surprising finding" (Trend 7)

• arXiv: https://arxiv.org/abs/2601.03309
• arXiv HTML: https://arxiv.org/html/2601.03309v1
• OpenReview: https://openreview.net/forum?id=tc2UsBeODW
• ICLR 2026 poster: https://iclr.cc/virtual/2026/poster/10006964

← Back to ICLR-2026-VLM4VLA · ICLR-2026 · Home