VLA Architectures — Categories A–M, compared (companion to Review-VLA-Architecture)

Split out of the main VLA Architectures review for faster GitHub-wiki rendering. This is the per-category deep-dive (§5 of that review): pros, cons, exemplars, and when to pick each of the 13 families.

Category A — Autoregressive action tokens

• Mechanism: Extend the VLM's vocabulary with discretized EE-pose-delta bins; generate actions left-to-right with standard cross-entropy.
• Exemplars: OpenVLA, RT-X, π0-FAST, RT-2, VLA-0.
• Pros: ✅ Zero architectural surgery · ✅ inherits LLM training/inference tooling · ✅ trivially extensible to any output type.
• Cons: ❌ Discretization caps precision · ❌ token-sequential decoding is slow · ❌ rigid decoding order.
• Key datapoint: VLA-0 (NVIDIA 2025) shows that representing actions directly as text beats π0.5-KI, OpenVLA-OFT, SmolVLA, GR00T-N1, MolmoAct on LIBERO — reopens the AR-vs-diffusion debate.
• Pick when: You want the simplest possible recipe and LLM tooling compatibility.

Category B — Flow-matching action expert

• Mechanism: Separate continuous-valued head attached to the VLM; produces a full action chunk in a handful of ODE integration steps by matching a learned vector field.
• Exemplars: π0 → π0.5 → π0.6 → π0.7 (canonical), RFS (residual), FLOWER (efficient 950M). NeurIPS 2025 additions: Knowledge Insulation (PI Spotlight — the published training recipe that enables this category to coexist with a VLM), Real-Time Chunking (async chunk inpainting for boundary smoothness), ReinFlow (learnable noise injection → RL-trainable flow policies). ICRA 2026 additions: FPO (Flow Policy Optimization — a likelihood-free RL fine-tuning objective that reuses the conditional-flow-matching loss differential as the PPO importance ratio, so off-the-shelf π0 is improvable from sparse reward with no sampler change; LIBERO avg 87.2), ACG (Action Coherence Guidance — a flow-specific test-time guidance term that suppresses the jitter/noise-sensitivity that flow policies' high generative capacity introduces under imitation). The ICRA cohort's clear signal: flow matching is now the default continuous-action head, and its two open problems — reward fine-tuning and action coherence — are where the work has moved.
• Pros: ✅ Smooth continuous actions · ✅ 5 Euler steps on H100 → 63 ms on π0.6 · ✅ clean separation of reasoning vs. control.
• Cons: ❌ Two-stage objective is finicky; action-head gradients corrupt VLM features → Knowledge Insulation needed · ❌ separate head complicates the pipeline.
• Pick when: Production system with latency budget and continuous-action smoothness requirements.

Category C — Continuous diffusion action head

• Mechanism: Same structural split as B (separate head), trained with denoising-diffusion loss rather than flow matching; multi-step sampling.
• Exemplars: Diffusion Policy (ancestor), RDT-1B (1.2B + Physically Interpretable UAS), DexVLA (~1B plug-in), RoboDual (specialist + generalist).
• Pros: ✅ Models multimodal action distributions well · ✅ mature theory · ✅ stable training.
• Cons: ❌ More NFE at inference than flow matching · ❌ separate head complicates architecture.
• Pick when: Training stability matters and you can afford extra denoising steps.

Category D — Discrete diffusion inside the VLM

• Mechanism: Actions are tokenized (like A) but generated via masked discrete diffusion within the same transformer — adaptive easy-first unmasking, sometimes with re-masking for uncertain tokens.
• Exemplars: Discrete Diffusion VLA (96.3% LIBERO), Unified Diffusion VLA (joint future-frame + action), dVLA (with CoT), DIVA & Fast-dVLA (latency opt), MMaDA-VLA, Dream-VLA (whole backbone is a diffusion LM).
• Pros: ✅ Unified transformer (one objective) · ✅ parallel decoding · ✅ adaptive order · ✅ inherits VLM KV-cache and decoding ecosystem.
• Cons: ❌ Discretization loss persists · ❌ newer — training recipes less mature · ❌ latency still catching up to flow matching (DIVA targets parity).
• Pick when: You want the unification argument and can tolerate early-adopter risk.

Category E — World-model / video-based action

📖 Key empirical reference for E4 (video-backbone WAMs): WAM vs VLA Robustness — Huawei + UToronto, Mar 2026 (arXiv 2603.22078). The first controlled benchmark of 7 VLAs + 2 hybrids + 4 WAMs on LIBERO-Plus + the new RoboTwin 2.0-Plus, with the headline mixed verdict: WAMs win noise/light/layout (Cosmos-Policy 82.2% LIB, LingBot-VA 74.2% RT) but lose camera + robot-state, π0.5 leads LIBERO-Plus overall (85.7%), and WAM inference latency is 4.8×–83× π0.5. Empirically grounds the E4 = video-generation-backbone-as-policy sub-pattern below.

This category covers every architectural pattern where a generative model of future observations is part of the policy — whether as a regularizer, a data source, an environment, a backbone, or a substitute for action generation entirely. The §4.2 diagram lays out the five sub-patterns; below is per-sub-pattern detail.

ICRA 2026 adds a goal-state variant. Goal-VLA (ICRA 2026) uses an image-generative VLM as an object-centric world model: it synthesizes a goal image from instruction + observation, validates it through a Reflection-through-Synthesis loop, then recovers the object's rigid transform (feature matching + point-cloud registration) and hands it to a training-free motion planner. No action-labeled data and no learned action head — the generated goal state is the interface. Zero-shot, it averages 59.9% on RLBench (8 tasks) where π0 scores 0.0% and OpenVLA 0.2%; the reflection loop carries a 40.0% baseline to 88.8%. It is the world-model-as-prompt analogue to E2 (generate, then act) but skips the policy entirely, sitting at the E5/Category-F boundary (geometry-derived control via a hierarchical VLM planner).

Why this category exploded in 2025–2026. Three forces converged: (1) video-generation models (LTX-Video, OpenSora, Wan2.1, Cosmos, DynamiCrafter) became cheap enough to fine-tune on robot data; (2) action-conditioned world models with bounded long-horizon error (Ctrl-World's 20+ s consistency, WMPO's reward F1 ≥ 0.95) made E3-style policy optimization viable; (3) ICLR 2026 confirmed strong empirical wins — Genie-Envisioner's 1M-episode AgiBot-World-Beta scale, WMPO's 53→70% real-robot lift, ViPRA's +16% SIMPLER. The thread is now the largest growth area in the ICLR 2026 VLA cohort.

E1 — World model as auxiliary loss

• Mechanism: Predict future frames / depth / segmentation as an auxiliary objective on top of VLM features; action head still drives behavior.
• Exemplars: Cosmos Policy (NVIDIA Cosmos foundation + control tokens), Vid2World (DynamiCrafter + Diffusion Forcing + Causal Action Injection — CS:GO FVD −71%, RT-1 FVD 18.5 vs 24.2 baselines), Genie Envisioner (GE-Base: LTX-Video 2B trained on 1M AgiBot episodes), Geometry-aware 4D Video (Stanford+MIT+TRI — joint geometry + appearance grounding), WorldVLA, DreamVLA (multi-modal world-knowledge forecasting: depth + geom + seg + dynamic-region mask), VideoVLA.
• Pros: ✅ Strong representation regularizer · ✅ no inference-time generation cost (WM is off-policy at deploy) · ✅ composes with any action head (A/B/C/D).
• Cons: ❌ Loss-balancing finicky · ❌ doesn't directly improve action prediction — gain is via shared features only.
• Pick when: You already have a VLA that works and want extra grounding without changing the action path.

E2 — World model as data factory

• Mechanism: Train a controllable video generator on real robot data + novel language prompts; harvest the generated rollouts (with re-captioning + IDM-derived pseudo-actions) as synthetic SFT data.
• Exemplars: DreamGen (NVIDIA's GR00T-Dreams blueprint — claim: 36h sim ≈ 3 months of teleop), ViPRA (NSVQ 8-codebook + DINOv2 video pretrain → flow head; +16% SIMPLER, +13% real, 22 Hz, 240k pretrain + 50k finetune steps on 8×H100 over 312h), Genie Envisioner (the SFT half of the GE-Base/GE-Act pipeline), "Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations" (ICLR 2026).
• Pros: ✅ Decouples from real data collection · ✅ enables novel-verb training data · ✅ benefits from web-video pretraining of the generator.
• Cons: ❌ Filtering generated rollouts is hard — N1 used a commercial-LLM judge on 8-frame samples · ❌ generator hallucinations propagate to policy · ❌ generator + IDM training is itself costly.
• Pick when: You have an abundant world model + scarce target-robot data.

E3 — World model as environment for policy optimization

• Mechanism: Use the learned video / state predictor as a simulator — rollout policy inside it, compute imagined rewards, optimize with policy-gradient or model-based RL. Distinct from E2 because the policy queries the WM at training time, not just learns from frozen rollouts.
• Exemplars: Ctrl-World (pose-conditioned controllable WM, 20+ s consistency for closed-loop policy evaluation), WMPO (OpenSora world model + VideoMAE reward model F1 ≥ 0.95 → GRPO policy update; 4-task Mimicgen 33.6 → 47.1 → 57.6 across budgets; real-world 53 → 70%), WorldGym (world-model-as-benchmark), RLVR-World (RL for world-model quality, NeurIPS 2025).
• Pros: ✅ Avoids expensive real rollouts during RL · ✅ closed-loop evaluation pipeline · ✅ scales with WM quality, not robot count.
• Cons: ❌ Sim2real-style WM-to-reality gap reappears · ❌ WM artifacts can short-circuit the reward (model-exploiting policies) · ❌ training-time inference cost is high.
• Pick when: You want RL on expensive embodiments and you can train a high-fidelity WM cheaply.

E4 — VAM: video-generation model replaces the VLM backbone

• Mechanism: Skip the VLM entirely. The backbone is a pretrained video-generation model (Cosmos, LTX-Video, DynamiCrafter); a small action decoder rides on top. Argument: web-video pretraining beats web-image+text pretraining for embodied tasks.
• Exemplars: mimic-video (Dec 2025 — "VAM" position piece), DiT4DiT (dual-DiT video-action, 2603.10448), S-VAM (shortcut VAM via self-distilling foresight, 2603.16195), Genie Envisioner (GE-Act: 160M action decoder block-aligned with LTX-Video 2B features, 5 Hz video + 30 Hz action async), Cosmos-Policy (ICLR-2026-Cosmos-Policy), LingBot-VA (Wan2.2-5B), Fast-WAM (Wan2.2-5B, joint-denoise + skip-video-at-test), DreamZero (Wan2.1-14B), GigaWorld-Policy (action-conditioned video).
• Pros: ✅ Exploits true internet-scale video data · ✅ video features encode dynamics natively · ✅ "scale beats architecture" empirical bet · ✅ strong visual-perturbation robustness (noise/light/layout) per the Huawei robustness study — Cosmos-Policy 82.2% LIBERO-Plus, LingBot-VA 74.2% RoboTwin 2.0-Plus.
• Cons: ❌ Loses language-grounding flexibility — verb generalization unclear · ❌ video backbones are 1–3B params with high latency · ❌ data is even larger than VLM-style · ❌ weak on geometric perturbations (camera viewpoint, robot init state) — video priors don't transfer to scene geometry; latency 4.8×–83× π0.5 even with denoising-step reductions (Review-WAM-vs-VLA-Robustness).
• Open empirical reference: "Do WAMs Generalize Better than VLAs?" Huawei Mar 2026 — the controlled benchmark across 7 VLAs + 2 hybrids + 4 WAMs on LIBERO-Plus + RoboTwin 2.0-Plus.
• Pick when: You believe web-video pretraining is the right prior, you have the compute, you can tolerate the latency, and visual-perturbation robustness matters more than camera-geometry robustness.

E5 — Geometry-first: predict 4D / point cloud → IK

• Mechanism: Predict future point cloud, 4D Gaussian field, or scene geometry; derive action via inverse kinematics / contact search. Skip action token generation entirely.
• Exemplars: Avi (2510.21746, NeurIPS 2025 Workshop — predict future point cloud → IK; sidesteps cross-embodiment action-vocabulary problem), Geometry-aware 4D Video (E1 + E5 hybrid — predicts video and geometry for the action policy), PA3FF (part-aware 3D feature field — could be either L or E5 depending on use).
• Pros: ✅ No action-vocabulary commitment · ✅ naturally cross-embodiment · ✅ explicit physical grounding.
• Cons: ❌ Generating future geometry is harder than future frames · ❌ IK adds a second optimization at inference · ❌ contact-rich manipulation requires physics-aware planning, not just IK.
• Pick when: Cross-embodiment is the primary axis and contact dynamics are tractable.

When E vs. its alternatives

• E1 vs. plain B/C/D: add E1 if you want a free representation regularizer with no inference cost. Not a substitute for the action head.
• E2 vs. teleop scaling: E2 only wins if your WM is genuinely better than what you'd get from 100h additional teleop on the target robot.
• E3 vs. real-robot RL: E3 is the path when real RL is impossible (expensive embodiment, safety constraints). Otherwise real RL is still cleaner.
• E4 vs. B (π-style flow matching): E4 is the contrarian bet — strong on video-rich domains, weaker on language-rich ones.
• E5 vs. category J (sensor-augmented 3D): E5 generates geometry; J senses geometry. Different commitments.

Pick category E when: No robot data for the target (E2/E4), expensive embodiment without sim (E3), strong video-pretraining ecosystem (E1/E4), or cross-embodiment is the primary axis (E5).

Category F — Hierarchical / dual-system / MoE

📖 Deeper treatment of the cognitive framing (Kahneman → Bengio → Chiriatti, Figure Helix-02, Sharpa CraftNet, ~26 dual/triple-system papers): see Review: System 0 / 1 / 2 for Humanoids.

• Mechanism: Decompose the policy into asynchronous components: slow System 2 VLM for planning + fast System 1 for control, optionally with sparse mixture-of-experts in either.
• Exemplars: GR00T N1/N1.5/N1.6, Hi-Robot, π0.5 / π0.7 (subtask + action expert), HiMoE-VLA, WholeBodyVLA, AdaMoE, RoboDual, OpenHelix, VITA-VLA (reverse distillation), Steerable Policies (Feb 2026 — replaces NL S2/S1 interface with a 5-level multi-modal command vocabulary), DuoCore-FS (Dec 2025 — Astribot's truly-parallel fast-slow with a written-and-read latent buffer + whole-body 3-stream RVQ-VAE tokenizer; 32.3 Hz on 25-DoF mobile dual-arm with 3B PaliGemma slow side), Galaxea + G0 (ICRA 2026 — Qwen2.5-VL System-2 planner + PaliGemma-3B flow-matching System-1 actor; its evidence cuts against cross-embodiment scaling: a single-embodiment 500 h / 100 K-trajectory dataset drives few-shot generalization more than raw OXE scale, and the System-2 G0-VLM hits 83.3% on Table Bussing vs 32.0% for Gemini-2.5-pro). NeurIPS 2025 additions: Fast-in-Slow (S1 embedded inside S2 via shared parameters, not cascaded), ChatVLA-2 (Dynamic MoE that routes reasoning vs. action), ThinkAct (NVIDIA — MLLM plans rewarded via RL, compressed to visual latent for action head), VLA-OS (controlled study that validates Hierarchical > Integrated > Action-Only).
• Pros: ✅ Decouples control frequency from reasoning frequency · ✅ capacity scaling via sparse experts · ✅ best empirical results on long-horizon tasks.
• Cons: ❌ Coordination between systems is finicky · ❌ sparsity training is unstable · ❌ MoE memory footprint · ❌ hand-designed inter-tier interface.
• Pick when: Long-horizon / multi-embodiment / you can afford the coordination complexity.

Category G — Reasoning-augmented (CoT)

• Mechanism: Emit natural-language or symbolic intermediate tokens (plans, affordances, trajectory sketches, point-traces) before or alongside action tokens.
• Exemplars: ECoT / ECoT-Lite, Hybrid Training, CoT-VLA, Embodied-R1, InstructVLA, MolmoAct, Vlaser, VLA-R1, CoA-VLA, ACoT-VLA, TraceVLA, dVLA (CoT channel), Steerable Policies (CoT reasoner emits grounded steering primitives — points / traces — alongside text rationales). NeurIPS 2025 additions: Chain-of-Action (ByteDance — backward trajectory AR from goal keyframe), ThinkAct (RL-shaped MLLM plans as visual latent), Robot-R1 (R1-style RL on keypoint reasoning — 7B beats GPT-4o on low-level spatial), DreamVLA (multi-modal forecasting as implicit CoT). ICRA 2026 addition: VLA-Reasoner moves the reasoning from emitted tokens to test-time search — online MCTS over candidate action chunks, each branch rolled out through a 600M action-aware world model (iVideoGPT) and scored by an offline value head, so imagined futures are the "rationales." It is plug-and-play on a frozen policy and lifts OpenVLA real-world 22%→41% and Octo-Small LIBERO 26.5%→37.3% without retraining.
• Pros: ✅ Interpretable · ✅ works with few demos · ✅ composes with any action-decoder category (A/B/C/D).
• Cons: ❌ Longer sequences → slower inference (until dVLA solved this for D) · ❌ reasoning quality is weakly supervised · ❌ hallucinated reasoning decouples from actions.
• Pick when: Long-horizon / need interpretability / few-shot adaptation.

Category H — Tokenizer-centric efficiency

• Mechanism: Optimize the action representation — learned vector quantizers, DCT compression, B-spline bases, quantile binning — to shrink tokens per action or enable structured parallel decoding. Orthogonal to categories A / D.
• Exemplars: FAST (RSS 2025), FASTER (RVQ + DCT loss), OmniSAT (B-spline), HyperVLA (hypernetwork), AutoQVLA (channel-aware quantization).
• Pros: ✅ Large latency / training-cost wins · ✅ enables entirely new decoding strategies · ✅ stacks with A or D.
• Cons: ❌ Tokenizer quality is the new bottleneck · ❌ harder to interpret · ❌ can clash with cross-embodiment transfer.
• Pick when: You're cost / latency constrained and already in category A or D.

Category I — Small / efficient VLAs

• Mechanism: Architectural choices targeting parameter / latency / single-GPU budgets — smaller backbones, Mamba SSMs, layer pruning, distillation.
• Exemplars: TinyVLA (2409.12514), SmolVLA (<0.5B), RoboMamba (SSM), FLOWER (950M flow-matching), NORA (Qwen-2.5-VL-3B), Lite VLA (CPU edge), ChatVLA, CogACT. NeurIPS 2025 additions: CogVLA (2508.21046 — FiLM-based instruction-driven routing + token pruning; 97.4% LIBERO, 2.5× training / 2.8× inference speedup over OpenVLA). ICRA 2026 addition: LightVLA (H+I — parameter-free, hyperparameter-free differentiable visual-token pruning via Gumbel-softmax over cross-attention saliency, ~78 tokens retained; −59.1% FLOPs / −38.2% latency while raising OpenVLA-OFT success 94.5%→97.4%, where VLM-oriented pruners like FlashVLA/SP-VLA/VLA-Cache collapse to ~74%). It inverts the efficiency/accuracy trade-off by making the keep/drop decision performance-driven.
• Pros: ✅ Deployable on edge / consumer HW · ✅ democratizes research · ✅ fast iteration.
• Cons: ❌ Performance gap on hardest tasks · ❌ less cross-embodiment headroom.
• Pick when: Edge deployment, consumer robot, or limited GPU budget.

Category J — Sensory-augmented input (native 3D / tactile / event)

Contrast with Category L (3D-foundation-aligned). J = the robot has a non-RGB sensor (depth camera, GelSight tactile, event camera) and the VLA ingests it natively. L = the robot has only RGB but training aligns features to a 3D foundation model. Different deployment costs, different data-collection requirements.

• Mechanism: Modify the input side, not the action decoder — feed the policy point clouds, tactile, event-camera streams, or depth maps from real sensors.
• Exemplars:
  • Native 3D point-cloud input: PointVLA (3D point-cloud features into frozen VLA), DexMove (ICLR-2026-DexMove — R-Tac tactile sensor + 3D contact perception), Avi (input and output side — also E5)
  • Tactile / force: Tactile-VLA, VLA-Touch (pretrained tactile-language), TaF-VLA (tactile + 6-axis force/torque), OmniVTLA (vision-tactile-language-action), DexMove (R-Tac 120 FPS markers + Farneback flow), FD-VLA (ICRA 2026 — force awareness without a force sensor: a Force Distillation Module distills a force token from vision+state, aligned at training time against the latent of a real F/T signal, so the sensor is needed only during training; the distilled token reportedly outperforms the raw sensor reading — recasting force sensing as a learnable-representation problem rather than a hardware requirement). → For the full touch-centric taxonomy (how tactile enters the policy, sensor hardware, ICRA/ICLR/CVPR 2026 trends, limitations), see the Tactile VLA cross-paper review.
  • Event cameras: E-VLA (2604.04834 / 2026) — first event-camera VLA for dark/blurred scenes
  • Depth + RGB-D: AugVLA-3D
• Pros: ✅ Unlocks contact-rich manipulation · ✅ direct geometric/force grounding · ✅ closer to real robot sensing stack · ✅ tactile is the only path to deformable / slip / friction-sensitive tasks.
• Cons: ❌ Extra sensor pipelines + data collection · ❌ less internet-pretraining benefit for non-RGB modalities (no web-scale tactile data) · ❌ harder to scale cross-embodiment · ❌ sensor calibration / synchronization is a real deployment cost.
• Pick when: Contact-rich / deformable / 3D-critical tasks where RGB+language alone fails. Combine with L if you have both sensors and foundation-model alignment.

Category K — Hybrid AR + diffusion

• Mechanism: Combine autoregressive next-token prediction with diffusion denoising in a single policy — interleave the two objectives or ensemble their outputs.
• Exemplars: HybridVLA (diffusion interleaved into AR; +14–19% over SOTA), AR-VLA (AR over continuous action chunks), MMaDA-VLA, "Unifying Diffusion and Autoregression" (ICLR 2026 OpenReview H1KDMNOKQn).
• Pros: ✅ Captures multimodal distribution (diffusion) without giving up token-level training (AR) · ✅ action-ensemble improves robustness.
• Cons: ❌ Architectural complexity · ❌ two objectives to tune · ❌ unclear when to pick over A or D alone.
• Pick when: You already have AR baseline and want diffusion's multimodality without migrating.

Category L — 3D-foundation-aligned VLAs (new in this rebuild)

Why it's its own category, distinct from J. Category J (sensory-augmented) is about the input modality: the robot has a depth camera / tactile sensor / event camera, and the VLA ingests it. Category L is about the representation prior: the VLA only sees RGB+language at deploy, but during training its features are explicitly aligned to a 3D foundation model (VGGT, ESM, Sonata/PTv3, DINOv2-3D, SuperPoint). This is a 2025–2026 thread that bypasses the data-collection cost of native-3D sensing while keeping the geometric grounding.

• Mechanism: Train the VLA to predict / align with outputs of a frozen 3D foundation model — depth maps, point clouds, part-aware feature fields, SE(3) transformations — typically via an auxiliary loss at a specific VLM layer or a parallel head. Action prediction is otherwise standard (flow-matching, AR, or discrete diffusion).
• Exemplars:
  • Spatial Forcing — implicit alignment to VGGT (Wang et al. 2025) at VLM layer 24 of 32; LIBERO 98.5% average, 5.9× data efficiency vs the OpenVLA-OFT base
  • Spatially Guided Training (ST4VLA) — Qwen2.5-VL-3B + DiT actor + 8.7 MB querying transformer with 0.5 gradient decay; SimplerEnv 66 → 84% on Visual Aggregation
  • Spatial-to-Actions / FALCON — Kosmos-2 + 1.0B Embodied Spatial Module; stochastic Bernoulli p=0.66 conditioning of depth/pose; full CALVIN + SimplerEnv tables
  • EquAct — SE(3)-equivariant multi-task transformer + iFiLM language conditioning; 18 RLBench tasks (Northeastern)
  • PA3FF — Sonata/PTv3 part-aware dense 3D feature field + SigLIP semantic supervision; PartInstruct 28.79 vs GenDP 19.36, PartNetE 70.6 mAP50 zero-shot
  • SpatialVLA (RSS 2025) — 3D Egocentric Position Encoding + Adaptive Spatial Grids derived from foundation 3D priors
  • GeoVLA (2508.09071 / 2025) — parallel 2D VLM + Point Embedding Network derived from RGB
  • BridgeVLA (2506.07961 / NeurIPS 2025) — projects 3D to multi-view 2D heatmaps for unified I/O; 96.8% real on 10 tasks with 3 trajectories each
  • Evaluation companion: Seeing Across Views (MV-RoboBench) shows even GPT-5 reaches only 56.4% vs human 91.0% on multi-view spatial reasoning — the gap is the motivation for this whole category
• Pros: ✅ Geometric grounding without 3D sensors at deploy · ✅ stacks with any action head · ✅ 5–10× data efficiency reported across the cluster · ✅ benefits scale with 3D foundation model quality, not robot data.
• Cons: ❌ Depends on a strong 3D foundation model (VGGT, Sonata, ESM, etc.) — quality is the ceiling · ❌ alignment layer choice is empirical (Spatial Forcing settled on layer 24/32 by sweep) · ❌ adds compute during training even though inference is RGB-only.
• Pick when: You want geometric grounding but cannot deploy depth sensors, or you want to multiply RGB sample efficiency.

Category M — Latent-action / video-pretraining-derived VLAs (promoted from "Bonus" given 2025–2026 maturity)

• Mechanism: Extract latent action tokens from unlabeled video, then ground them to robot actions — cleanly decouples "what" (latent) from "how" (per-embodiment decoder). Multiple sub-flavors: VQ-VAE quantization (LAPA), DINO-space task-centric (UniVLA), enhanced latent with proprio FDM (villa-X), Perceiver-resampler queries (BayesianVLA).
• Exemplars:
  • LAPA (ICLR 2025) — first unsupervised VLA pretraining via VQ-VAE latent actions from video
  • UniVLA latent (2505.06111) — task-centric latent actions in DINO space, ~1/20 compute vs OpenVLA
  • villa-X — PaliGemma 3B + 18-layer ACT-latent + 18-layer ACT-robot; proprio-FDM with 50%/50% stochastic masking; SIMPLER Google 77.7% / WidowX 62.5% vs π0 58.7%; 1.6M robot trajectories + 3.6M human-video clips
  • BayesianVLA (2601.15197 / 2026) — Perceiver-resampler-style Latent Action Queries + PMI objective
  • XR-1 — Unified Vision-Motion Codes (UVMC) as latent action representation for cross-embodiment
  • NavFoM (latent token application) — Embodied Navigation Foundation Model uses TVI (Temporal-Viewpoint Indicator) tokens as a latent embodiment-context representation
  • GR00T's "LAPA embodiment" head — pre-quantized VQ-VAE latents as pseudo-actions for unlabeled human ego-video (in the GR00T series data pyramid base layer)
• Pros: ✅ Unsupervised pretraining from web video at scale · ✅ cross-embodiment transfer is natural (see Review-Cross-Embodiment §4.10) · ✅ strong sample efficiency (UniVLA ~1/20 compute vs. OpenVLA, villa-X transfers to XHand 12-DoF dexterous w/o dexterous pretraining data) · ✅ a clean separation of "intent" (latent) and "execution" (per-embodiment decoder) for the right tasks.
• Cons: ❌ Latent quality is the ceiling — and quality depends on the video corpus · ❌ fine-grained contact often doesn't fit a low-dim latent · ❌ task-centric latents collapse on busy but task-irrelevant video · ❌ a separate latent codebook complicates the training pipeline · ❌ LBM Co-training Study empirically reports that latent-action variants of human-video co-training do not help at LBM scale — the gain comes from text captions, not from the latent codebook (Modality 4 verdict).
• Pick when: You have abundant unlabeled video and limited robot data, and you can tolerate the latent-codebook engineering cost. Be aware of the LBM-Cotraining caveat before committing.

---

← Back to VLA Architectures review