Venue: ICLR 2026 Authors: Chen Li, Zhantao Yang, Han Zhang, Fangyi Chen, Anudeepsekhar Bolimera, Marios Savvides (Carnegie Mellon University); Chenchen Zhu (Meta Reality Labs, USA) Category: VLA Training — Post-training / adaptation Trend tag: Efficient adaptation · meta-learning · context-aware co-training

flowchart LR
  Pre[OpenVLA-7B<br/>or NORA-Long 3B Qwen2.5-VL] --> ANP[Action-ANP module<br/>self-attention over context<br/>+ cross-attention to target]
  CB[Context Bank<br/>4 LIBERO suites + GR00T aux<br/>refreshed every K=200 steps<br/>b_C=32 examples/task] --> ANP
  TB[Target Bank<br/>4 LIBERO suites unified] --> ANP
  ANP --> Hide[Hidden states<br/>concat with Llama2 hidden]
  Hide --> LM[LM head]
  LM --> A[Discrete action tokens]

Mainstream VLA post-training fine-tunes one model per task (OpenVLA: 4 separate 240K-step runs across LIBERO-Goal/Spatial/Object/Long). Naively switching to multi-task SFT helps a little; naively adding diverse auxiliary tasks (different camera views, different DoF, bimanual) degrades performance due to optimization instability from heterogeneous distributions. MetaVLA's question: can meta-learning extract benefit from auxiliary tasks without destabilizing in-domain training?

Built on Attentive Neural Processes (ANP, Kim '19), which model conditional p(y_T | x_T, x_C, y_C) as a meta-learner over functions parameterized by both global and target-specific latents.

For target feature x_T and context pairs (x_C_i, y_C_i):

• Self-attention over context produces per-context representations r_C_i and s_C_i.
• Cross-attention with target query x_T → r_T (deterministic, target-aware).
• Mean over s_C_i → s̄_C; stochastic latent z ~ q(z | s̄_C) (Gaussian via reparameterization).
• ELBO: log p(y_T | x_T, x_C, y_C) ≥ E_{q(z|s̄_T)}[log p(y_T | x_T, r_T, z)] − D_KL(q(z|s̄_T) ∥ q(z|s̄_C)).
• r_T and z are concatenated with Llama2 hidden states before the LM head; rest of the OpenVLA pipeline is unchanged.

The KL term regularizes target distribution toward context distribution. Two variants:

• Deterministic (default): drops the stochastic / KL term → reconstruction loss only. Better on harder tasks (esp. LIBERO-Long).
• +Stochastic: full ELBO. Slightly better on Spatial, but underperforms on Long where domain shift > KL assumption.

• Context bank = in-domain (4 LIBERO suites, split as non-overlapping context vs target sets) + out-of-domain auxiliary (subset of partially open-sourced NVIDIA GR00T data).
• Target bank = target sets of all four LIBERO suites jointly.

Auxiliary task selection deliberately includes structural mismatch:

• Front-view 7-DoF single-arm (similar to LIBERO).
• Side-view 7-DoF single-arm (camera mismatch).
• Bimanual 14-DoF two-arm (action-space mismatch). This is intentionally less curated than CoT-VLA's tightly-aligned auxiliary picks — argues for robustness to context diversity.

• Context refresh interval K = 200 steps.
• Context batch size b_C = 32 examples per context task.
• Per-task SFT cross-entropy loss on action tokens.
• Trained for 75K steps total vs OpenVLA's 240K.
• Hardware: 8× A100 80GB GPUs, ~24 hours total vs OpenVLA's ~100 hours total across the four suites — a ~76% GPU-time reduction (paper's headline figure).
• Eval: one 24GB RTX-4090 GPU.
• Backbone: OpenVLA-7B (Llama2-7B + ViT) primary; ablation on NORA-Long 3B (Qwen2.5-VL-based) for backbone-agnostic claim.

Headline: MetaVLA+6 aux beats OpenVLA by +4.4% avg (and +8.0% on LIBERO-Long) using 75K steps vs 240K (-68.75% steps). vs SFT-4LIBERO: +3.1% avg, +5.1% on Long.

The contrast within Table 1 is the real story: vanilla SFT collapses catastrophically as auxiliary tasks are added (76.2 → 8.6 avg with 6 aux tasks). MetaVLA gains from the same 6 aux tasks (77.8 → 79.3).

+6.4% over NORA-Long avg with aux tasks; +25.4% over native SFT counterpart with aux tasks.

Action-ANP alone (no co-training) already beats OpenVLA at every checkpoint; per-suite Long task continues to improve through 240K.

Context batch size b_C (Fig. 4): success monotonically increases with b_C ∈ {4, 8, 16, 32}; b_C = 32 is the sweet spot — gain saturates and memory remains modest.

Auxiliary task selection (Table 1, three rows): MetaVLA+1single+1bi (78.5), MetaVLA+3single (78.2), MetaVLA+5single+1bi (79.3) — all beat their SFT counterparts by 22-70 pts, demonstrating robustness to context diversity (different views, different action spaces).

Parameter-size confound (MetaVLA-Pretrained-Context-ONLY): replace context bank with bridge_orig + fractal20220817 (already in OpenVLA pretrain) → average drops to 74.4 (vs 77.8 with LIBERO context), showing gain is from out-of-distribution informative auxiliary signals, not just added params.

Stochastic vs deterministic ELBO: stochastic helps Spatial (+0.4), comparable on Goal/Object, hurts Long (-2.3 → 53.0). KL term assumption breaks under bigger domain shift.

Multi-task co-training mechanism (MetaVLA-EACH): Action-ANP alone — single-suite training without context co-training — already beats OpenVLA at half the steps. Co-training collapses 4 models to 1 and saves another 45K steps overall.

Inference-cost overhead: only +0.3 ms/token latency (Fig. 9 in appendix) due to lightweight ANP module.

1. Backbone breadth. Only OpenVLA-7B and NORA-Long-3B tested; broader VLM/VLA spectrum (e.g. PaliGemma-3B π₀-style, Qwen2-VL action experts) remains future work.
2. Real-robot deployment. All experiments are LIBERO simulation; real-world validation deferred.
3. No formal proof of "scaling more robustly". Authors acknowledge their stability claim relative to vanilla multi-task SFT is empirical; a rigorous proof / theoretical bound is "left to future work due to computational constraints".
4. Web-scale context bank not yet tried. They speculate that web-scale data augmentation of the context bank (analogous to π₀.5's pretraining co-training) could help further but didn't run it.
5. Auxiliary task selection. Explore-exhaust of combinations not done due to memory/compute.

Vs OpenVLA / OpenVLA-OFT. OpenVLA fine-tunes each LIBERO suite as a separate Hugging Face model, totaling ~240K steps across the four suites (Table 1; per-suite breakdown 60K/50K/50K/80K in Table 3). MetaVLA collapses these into one unified model in 75K steps (-68.75% steps) and beats avg by 4.4 pts. OpenVLA-OFT (Kim '25) optimizes inference speed but per the paper's intro still demands ~150K–500K SFT steps (incl. diffusion + non-diffusion parts); MetaVLA is orthogonal — could in principle be stacked.

Vs π₀ / π₀.₅ / π₀.₆ / π₀.₇. π₀-class models invest heavily in pretraining (large robotics corpus + flow matching). MetaVLA is purely post-training: π₀.₅ scores 96.9 avg LIBERO via massive pretraining; MetaVLA+aux 79.3 with much cheaper compute. Different points on the cost/performance frontier.

Vs CoT-VLA / OneTwoVLA / EO-1. Those add reasoning data into pretraining (CoT-VLA, OneTwoVLA) or interleaved Vision-Text-Action (EO-1, "high inference latency"). MetaVLA stays at post-training and avoids inference-time overhead.

Vs other efficient-adaptation peers in 2026. Sits with FASTER, HyperVLA, Hybrid Training, LBM Cotraining — all treat training/adaptation efficiency as a first-class objective. MetaVLA is unique in leveraging meta-learning (ANP) rather than hypernetworks, LoRA-style routing, or curriculum.

Vs GR00T-N1.5. Interestingly, MetaVLA uses GR00T data as auxiliary context — flips the relationship from competitor to fuel.

The deeper claim is structural: post-training is an under-explored axis vs the dominant pretraining-scaling story. Naive multi-task SFT fails worse with more diverse auxiliary tasks (aggregate -67.6 pt collapse with 6 aux tasks). The ANP context-bank lets the model selectively attend to relevant context, turning auxiliary diversity from a liability into an asset. Reframes VLA adaptation as a meta-learning problem rather than a fresh fine-tune per task.

• OpenReview: https://openreview.net/forum?id=E1K2Ph3LtS

• HyperVLA
• FASTER
• Hybrid Training
• LBM Cotraining
• OpenVLA

← Back to ICLR-2026