Scope: the 2026 convergence where a VLA and a world/video model fuse into one policy — with a focused comparison of the papers that split vision (video / visual-foresight) into its own expert tower in a three-expert Mixture-of-Transformers (MoT). Companions: VLA Architectures §4.2b (the taxonomy this expands) · World Models · NVIDIA WAM thesis · WAM vs VLA Robustness. Written as a design reference for building a new architecture — §6 is a decision guide.

A VLA grounds language → action directly; a world/action model (WAM) inserts an imagine stage (predict how the scene evolves, then act). Through 2026 the two converged: NVIDIA's WAM blog predicts the next generation is a WAM+VLA hybrid (Review-NVIDIA-WAM-Cosmos3 §2.6), and the field settled on Mixture-of-Transformers (MoT) as the vehicle. This page is the deep-dive on the sharpest version of that idea — giving vision its own tower.

Two orthogonal design axes organize the whole hybrid space (full taxonomy in Review-VLA-Architecture §4.2b):

• Axis 1 — fusion style: modular / bolted-on (π0.7, Cortex 2.0) vs unified single model (this page).
• Axis 2 — where the world/vision model sits relative to the action path: in the action path (generate/consume frames to act — heavy, ~3–4× latency) · co-training tower, dropped at inference (reactive) · training-time critic only (🆕 IROS 2026 — the WM never touches deployment; it scores action chunks during offline RL post-training, then is absent — AtomVLA).

The canonical unified hybrid is a three-tower Mixture-of-Transformers: an understanding/VLM tower, a vision tower (video generation or visual-foresight), and an action tower. Each tower keeps its own weights and its natural generative objective — autoregressive text for reasoning, diffusion / flow-matching for visual and action — while the towers exchange information through shared self-attention (the BAGEL recipe, §3: separate QKV projectors + FFNs per expert, shared attention layers).

flowchart LR
  IN[obs · language] --> U[Understanding / VLM tower<br/>AR text · reasoning, subtask]
  IN --> V[Vision tower<br/>diffusion — video / keyframe / latent future]
  IN --> A[Action tower<br/>flow-matching — action chunk]
  U <-. shared self-attention .-> V
  V <-. shared self-attention .-> A
  U <-. shared self-attention .-> A
  A ==> OUT[action]
  V -. run in-path / cheap 1-step / dropped .-> A

Why split vision off? A single VLM fine-tuned to also generate video and actions suffers gradient conflict; per-expert QKV/FFN with shared attention lets the visual-generation objective and the action objective coexist without one clobbering the other, while still cross-conditioning. It also makes the vision tower droppable at inference (Axis 2) — the key to buying a world prior without its latency.

Structural variants (not a vision-separate three-expert MoT, but the same fusion family — useful contrasts):

(a) What the vision tower predicts — granularity is the real lever.

• Full video (Motus, DYNA-2, Cosmos 3): richest physical prior, heaviest. Best for dynamics-scarce pretraining; worst for latency if run in-path.
• Keyframe / subgoal (BagelVLA, HALO, π0.7): a single predicted image/subgoal, aimed squarely at long-horizon, multi-stage tasks (BagelVLA's motivating example: solve an arithmetic step, then place the block). Much cheaper than a rollout.
• Latent future (Being-H0.7, ω-0): predict a representation, never pixels — cheapest, naturally reactive, but the world prior is only as good as the latent space.

(b) Per-tower generative mechanism. The consensus (explicit in HALO) is to keep each expert's natural objective: AR for text reasoning, diffusion/flow-matching for visual and action. The MoT's separate-QKV/FFN-shared-attention wiring (BAGEL) is what makes heterogeneous objectives cohabit one model.

(c) Does the vision tower run at inference? — the latency frontier. This single choice dominates deployability:

• In-path (Cosmos-Policy/DreamZero, Cosmos 3): generate/consume frames to act → strongest prior, 3–4× latency (Review-WAM-vs-VLA-Robustness).
• Cheap single-step (BagelVLA's RFG): one denoising step extracts predictive visual features — "foresight without full image synthesis" — recovering real-time rates (40–72 Hz).
• Dropped → reactive (DYNA-2, Being-H0.7, ω-0): the vision tower is training-only; inference is a plain fast policy. The world prior is baked into shared weights.
• Critic-only, at post-training (🆕 AtomVLA, IROS 2026): the WM is used neither in-path nor as a co-training tower — it scores candidate action chunks against LLM-derived subtasks in latent space during offline GRPO, enabling RL-quality post-training without online robot rollouts, then plays no role at deployment. This is the cheapest way to inject a world prior — it never costs inference latency and never needs the WM at train-time-in-the-loop; the tradeoff is the prior only shapes the policy indirectly (via reward), not the representation.

(d) How the vision tower yields action supervision from label-free video. A recurring sub-problem, solved three ways: optical-flow "delta action" (Motus), hand-pose pseudo-actions (DYNA-2), future-informed latent posterior (Being-H0.7). This is what lets the vision tower pretrain on human/web video the action tower can't.

(e) Scale, openness, evidence. Motus (8B, open) and BagelVLA (9B, open on Bagel) are the reproducible references; HALO reports only a relative +34.1% (ICML); the strongest industrial datapoints (Cosmos 3, DYNA-2) are vendor-reported and closed. No study yet ablates the third tower's marginal value against a 2-tower VL+action baseline at matched compute.

(f) IROS 2026 — what changed (🆕). The insight is reinforced, not overturned: convergence continues and the latent / low-inference-cost corner keeps winning. Two refinements: (1) AtomVLA adds the critic-only hybrid mode above — a third answer on Axis 2 that pushes "cheapest world prior" to its limit (no inference cost, no in-the-loop WM at train time). (2) The WAM cluster broadened — cross-embodiment world models for dexterous manipulation, DreamMimic (humanoid loco-manip via a WM), RoboDream (WM as a data factory) — confirming World Models's "one WM, many roles" thesis inside the hybrid frame (backbone · co-training tower · reactive prior · offline critic · data engine). The marginal-value gap still stands — even with AtomVLA, no head-to-head isolates the world-model contribution at matched compute.

Pros

• Interference-free multi-objective training — per-expert QKV/FFN means video-generation gradients don't corrupt the VLM's language grounding or the action head (the failure mode of monolithic VLA+video fine-tuning).
• Modular, staged training — Motus's 3-phase pipeline and BagelVLA's "add a 2B action expert last" both exploit tower separation to stage capabilities and freeze what's done.
• Droppable at inference — the vision tower can be omitted (reactive) or run cheaply (RFG), decoupling the world-prior benefit from latency — the property monolithic designs can't offer.
• Natural-objective per modality — AR text + diffusion visual/action, each at its best.

Cons

• Parameter/compute cost — 8–14B is the entry ticket; prices out small labs.
• Loss balancing is delicate — "video training dilutes action-learning gradients" (DYNA-2, Motus); the video/action loss weight λ is a live knob.
• Latency if run in-path — full generation is 3–4× a VLA; only cheap-1-step or dropped variants hit real-time.
• Unproven marginal value — no controlled head-to-head shows the third tower beats a well-tuned 2-tower VL+action at equal budget.

1. How many towers? 2 (VL+action, or video+action) is simpler and often enough; add a third vision tower only if you need both language grounding and generative visual foresight (long-horizon, multi-stage, or human-video pretraining).
2. Pick the vision granularity to your task horizon. Long-horizon/compositional → keyframe/subgoal (BagelVLA/HALO). Dynamics-scarce pretraining → full video (Motus). Latency-critical → latent (Being-H0.7/ω-0).
3. Decide the inference contract first. In-path (accept 3–4× latency), cheap 1-step (RFG), or dropped/reactive (co-train the vision tower, don't run it). This choice constrains everything else.
4. Plan action-from-video supervision if pretraining on label-free video: optical-flow latent, hand-pose, or a posterior/prior latent.
5. Keep each expert's natural objective (AR text, diffusion/flow visual+action) and fuse via separate QKV/FFN + shared self-attention (BAGEL) to avoid interference.
6. Budget the λ (video↔action loss weight) and expect to tune it — it is the reported failure knob.

1. Does the third (vision) tower earn its parameters vs a 2-tower VL+action at matched compute? No paper isolates this.
2. Which vision granularity wins per task class (video vs keyframe vs latent)? Motus/BagelVLA/Being-H0.7 each argue a different point on the curve; no shared benchmark compares them.
3. Where should the world model sit — in-path, reactive co-training tower, or offline critic? DYNA-2 and Being-H0.7 bet reactive; Cosmos 3 keeps it in-path; AtomVLA (IROS 2026) bets critic-only (WM shapes the policy via offline-RL reward, never at inference). Which mode wins on the hardest contact-rich / long-horizon tasks is open — and it may be task-dependent (in-path foresight for precise contact, critic-only for cheap long-horizon shaping).
4. Is the MoT a transient stage that collapses into the dual-system (Cat F) taxonomy, or a durable family? (See Review-VLA-Architecture §8.)

• Core papers (in-depth): Motus (2512.13030) · BagelVLA (2602.09849) · HALO (2602.21157) · BAGEL (2505.14683)
• Variants: DYNA-2 · Being-H0.7 · ω-0 · Cosmos 3 / NVIDIA WAM · Cortex 2.0 · AtomVLA (critic-only, IROS'26)
• IROS 2026 context: IROS 2026 survey §5.1
• Taxonomy & theory: VLA Architectures §4.2b · World Models · WAM vs VLA Robustness
• Latest Papers · Reviews

← Back to Home · Reviews