Paper: A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation Lead author: Fanqi Lin · Senior author / corresp.: Jose Barreiros ([email protected]) Co-authors include: Kushal Arora, Jean Mercat, Haruki Nishimura, Paarth Shah, Chen Xu, … Affiliation: Toyota Research Institute (TRI) arXiv: 2602.01067 · Feb 1, 2026 Code / data: internal TRI; no public release announced as of Feb 2026

This page sits in the VLA architectures review as a training-recipe contribution and pairs with Knowledge Insulation (the canonical "how to protect the VLM during action training" result) as the what data should you co-train on companion.

Toyota Research Institute trains 89 LBM (Large Behavior Model) policies — a flow-matching VLA built on PaliGemma2-3B — under a controlled 5×3 grid of data modality × training strategy, and runs 58,000 simulation rollouts + 2,835 real-robot rollouts to measure each combination.

Three findings overturn folk wisdom that had been propagating through 2025:

1. Web vision-language data and cross-embodiment robot data dominate — they account for almost all of the +36 pp simulation gain and +45 pp real-world language-following gain.
2. Discrete action tokens (FAST, VQ-VAE) and latent video tokens do not help at LBM scale — possibly hurt — despite being widely adopted in 2025 VLAs.
3. Explicit chain-of-thought conditioning at inference is harmful for direct perception-to-action tasks; the gain people attributed to CoT is from training-time supervision, not inference-time conditioning. This empirically corroborates from a different angle the Hybrid Training thesis.

Two further findings refine the recipe:

4. Cross-embodiment data is best used in phase 1 only — keeping it in phase 2 hurts.
5. Robot-only training erodes the VLM's MMBench/GQA scores; balanced co-training restores them — i.e., VL co-training is partly a regularization-against-forgetting story, not just a generalization-gain story.

By Q1 2026 the community had three loud co-training claims fighting for default-recipe status:

• PI / Knowledge Insulation (NeurIPS 2025): the gradient from the action expert must not back-prop into the VLM, or you lose the VLM's pretrained knowledge.
• OpenVLA / FAST / OmniSAT: discrete action tokenization is the way to share parameters between vision-language and action.
• ECoT family: chain-of-thought traces (text or latent) attached to actions improve generalization.

The TRI paper is the first controlled comparison that grades all three with the same backbone, the same robot platform, and the same evaluation suite. The verdict for each:

This is the empirical companion paper that the field had been waiting for since the Sergey Levine et al. What Matters in Learning from Large-Scale Datasets (ICRA 2025) line — same spirit, but at LBM scale and including the language/CoT axes that the older paper did not cover.

flowchart LR
  IMG[Multi-camera observation] --> VLM
  LANG[Language instruction] --> VLM
  STATE[Proprio state] --> VLM
  VLM[PaliGemma2-3B-PT VLM<br/>trained jointly with action expert]
  VLM -- single observation token,<br/>extracted from last 4 layers --> COND[adaLN conditioning]
  COND --> AFT[ActionFT — 8-layer flow-matching transformer]
  AFT --> ACT[Action chunk: 16 steps<br/>relative EE pose + gripper width]

  VLM -.discrete-token CE losses.-> AUX[Auxiliary discrete heads<br/>FAST tokens / VQ-VAE / language captions]

  classDef vlm fill:#bbdefb,stroke:#1565c0,color:#000
  classDef act fill:#c8e6c9,stroke:#2e7d32,color:#000
  classDef aux fill:#fff9c4,stroke:#f57f17,color:#000
  class VLM,COND vlm
  class AFT,ACT act
  class AUX aux

• Backbone: PaliGemma2-3B-PT, trained jointly with the action expert (not frozen) — the VLM "can optionally be trained to generate text or discrete action tokens." This is essential to the paper's catastrophic-forgetting result: only because the VLM weights move can robot-only training erode its MMBench/GQA scores and VL co-training restore them.
• Action expert: ActionFT, an 8-layer flow-matching transformer, conditioned via adaLN on a single observation token built from the last 4 layers of the VLM.
• Action representation: continuous flow-matching for the primary head; auxiliary discrete heads added under specific co-training settings.
• Action horizon: 16-step chunks.

Loss = flow-matching MSE on continuous actions + cross-entropy on auxiliary token / language streams when present, weighted by computed masks.

Note (correction): an earlier version of this page described the VLM as "frozen during action training." The source describes a jointly-trained end-to-end system; the VLM is not frozen.

This architecture is deliberately close to the public PaliGemma-3B + flow-matching recipe so the conclusions transfer to other LBM-class systems built on the same template (π0.5 / π0.6 / Steerable-π0.5, GR00T-N1.x family modulo backbone size).

flowchart TB
  subgraph M1[Modality 1 — Standard VL data · 50M samples]
    direction LR
    RP[RoboPoint<br/>1.3M images · 8.2M QA]
    RS[RefSpatial<br/>2.5M images · 20M QA]
  end
  subgraph M2[Modality 2 — Dense robot annotations · 523h]
    direction LR
    SC[Scripted heuristics<br/>EE-motion descriptions]
    GC[GPT-5 captions<br/>rich semantic descriptions]
  end
  subgraph M3[Modality 3 — Cross-embodiment robot · 1,150h]
    OXE[OXE-Ramen<br/>12 robots · 924 tasks · 466k demos]
  end
  subgraph M4[Modality 4 — Human videos · 2,271h]
    direction LR
    LAM[Latent Action Model<br/>32 codebook · 8 tokens/chunk]
    HC[GPT-5 captions of motion]
  end
  subgraph M5[Modality 5 — Discrete robot tokens · 523h]
    direction LR
    FAST[FAST tokens<br/>~42 / chunk · vocab 2,048]
    VQ[VQ-VAE tokens<br/>8 / chunk · codebook 32]
  end

  classDef good fill:#c8e6c9,stroke:#2e7d32,color:#000
  classDef bad fill:#ffcdd2,stroke:#c62828,color:#000
  class M1,M3 good
  class M2 good
  class M5 bad

Color coding: green = the paper's "effective" verdict, red = "no significant benefit", mixed for M2/M4 (caption variants effective, token variants not).

• Sources: RoboPoint (1.3M images, 8.2M QA pairs — robotics-relevant pointing and spatial QA) and RefSpatial (2.5M images, 20M QA — refer-and-locate spatial reasoning). Combined ≈ 50M VL samples.
• Loss: standard CE on the VLM language head.
• Verdict: ✅ largest single-modality contribution to both unseen-task generalization (sim) and language following (real). Cheap to scale because no robot data collection is required.

• Source: TRI-Ramen (~523 h of TRI's in-house manipulation demos — 403 tasks, 53,411 demonstrations on the dual-Franka platform).
• Two caption variants:
  • Scripted heuristics — derive end-effector motion descriptions from action stream ("gripper moves left and down 5 cm, closes").
  • GPT-5 captions — vision-language model emits rich semantic captions for short trajectory segments at ~1–2 second intervals.
• Verdict: ✅ GPT-5 captions clearly beat scripted heuristics. The semantic richness — object names, contact events, intent — is what matters. Scripted heuristics give the action a redundant motion description that the VLM does not need.

• Source: OXE-Ramen — a TRI-curated subset of Open X-Embodiment covering 12 robot setups, 924 tasks, 466k demos, totaling 1,150 h.
• Used as: native continuous actions (not retokenized).
• Verdict: ✅ effective, but best used in phase 1 only. Keeping it in phase 2 hurts — the morphology gap between the OXE robots and the target dual-Franka platform creates conflicting gradients during specialization.

• Sources (2,271 h after filtering): Ego4D, EgoDex, Something-Something V2, Epic Kitchen, HoloAssist.
• Two representations tested:
  • Latent action tokens — Latent Action Model (LAM) trained to compress short clips into 8 tokens per chunk over a codebook of 32; supervised via CE.
  • GPT-5 motion captions — VLM-generated language descriptions of the human motion.
• Verdict: ✅ caption variant works. ❌ latent-action variant only helps in the low-data regime; gains evaporate at scale. The paper's framing: human videos contribute visual diversity through language, not through action-token transfer.

• Source: TRI-Ramen retokenized two ways:
  • FAST — DCT-based, ≈ 42 tokens per 16-step chunk over a 2,048 vocab.
  • VQ-VAE — 8 tokens per chunk over a 32 codebook.
• Used as: auxiliary CE prediction targets alongside the primary flow-matching head.
• Verdict: ❌ no significant benefit; FAST sometimes hurts unseen-task performance. This is one of the paper's most important findings — Knowledge Insulation NeurIPS-2025-Knowledge-Insulation uses FAST tokens to give the VLM a CE signal, but TRI's controlled study shows the signal itself is not what matters; KI's gain must come from the gradient routing, not from the token target.

flowchart LR
  subgraph SP[Single-phase joint]
    SP1[Mix target + co-training<br/>train one phase end-to-end]
  end
  subgraph TP1[Two-phase · 1st-phase only]
    TP1a[Phase 1: co-training data only]
    TP1b[Phase 2: target robot continuous actions only]
    TP1a --> TP1b
  end
  subgraph TP2[Two-phase · full co-training]
    TP2a[Phase 1: co-training data only]
    TP2b[Phase 2: target + co-training jointly]
    TP2a --> TP2b
  end

  classDef ph fill:#fff9c4,stroke:#f57f17,color:#000
  class SP1,TP1a,TP1b,TP2a,TP2b ph

The strategy that wins depends on the modality:

Pattern: language-bearing modalities want to be co-trained throughout (phase 1 + phase 2). Action-stream modalities from non-target morphologies want to be dropped after phase 1 so the policy can specialize.

This pattern is consistent with how the π-series uses Knowledge Insulation: the VLM keeps getting language-side gradients (CE on tokens / captions) throughout training, while the action expert specializes — TRI's empirical result is the data-side counterpart of KI's gradient-routing result.

• Tasks: 13 seen + 8 unseen (semantic / multi-step / compositional generalization).
• Conditions: nominal + four distribution shifts — lighting, textures, camera viewpoint, backgrounds.
• Rollouts: 50 per task per condition → 58,000+ total.

flowchart LR
  subgraph LF[Language-following · 45 rollouts/condition]
    LF1[Seen objects from training]
    LF2[Instruction generalization<br/>paraphrases · category · attribute]
    LF3[Unseen objects · 52 novel]
  end
  subgraph LH[Long-horizon dexterous · 30 rollouts each · ~93 s · 13 steps avg]
    LH1[PackItemsIntoStringBag<br/>fine-grained bottle capping]
    LH2[PourIngredientsIntoSoup<br/>scooping with spatula]
    LH3[StoreCleanDishes<br/>transparent objects]
  end

• Total real-robot rollouts: 2,835.

In addition to manipulation, the LBM's PaliGemma2 backbone is evaluated on MMBench, GQA, and other standard VL benchmarks before/after each training regime — to detect catastrophic forgetting of visiolinguistic capability.

→ +36.4 percentage points from the right co-training mix; gains stack additively rather than saturating.

→ +45.3 pp. The unseen-objects axis is the largest single beneficiary — VL data + human-video captions push the model to recognize objects it never saw on the target robot.

After pretraining, fine-tune on 200 demonstrations of three unseen long-horizon tasks (bottle-capping in bag, scooping into soup, storing transparent dishes):

The +22.8 pp delta over the same fine-tune procedure on the no-co-training baseline is the cleanest demonstration in the paper that co-training learns transferable representations, not just multi-task amortization.

Robot-only training drops PaliGemma2 MMBench / GQA scores measurably vs. the original VLM weights. Adding standard VL co-training restores them. This frames part of the gain as preventing catastrophic forgetting rather than learning new capability.

This is the section that will be most cited because it directly contradicts the prevailing 2024-2025 ECoT narrative.

Three CoT trace types are tested:

• Scripted captions (Modality 2a)
• VLM-generated captions (Modality 2b / Modality 4b)
• Latent action tokens (Modality 4a)

For each, three CoT strategies:

All three CoT inference strategies either match or degrade the implicit two-phase co-training baseline.

The paper's diagnosis:

• The eval tasks are direct perception-to-action (pick-and-place, simple long-horizon scoops). They do not require multi-step planning.
• VLM-generated captions hallucinate at non-trivial rates; latent action tokens contain reconstruction noise.
• When the action prediction is conditioned on the noisy generated CoT, errors propagate: a hallucinated subgoal directly distorts the action chunk that follows.
• When CoT is only used as an auxiliary CE training target and skipped at inference (option 1), the model still doesn't gain — the implicit co-training already captures whatever benefit was available.

This is the paper's most direct dialogue with the broader 2026 reasoning-augmented VLA literature:

• Hybrid Training had argued: the ECoT generalization gain is from training-time supervision; drop the reasoning at inference. TRI's controlled experiment empirically validates that thesis for the LBM-scale setting.
• Actions as Language had argued: subtask relabeling is the lever. TRI confirms: language captions of robot/human trajectories are the most valuable form of co-training, but only as CE training targets, not as inference conditioning.
• Embodied-R1 and dVLA are the two papers that do make CoT-at-inference work — they pay either via RL (E-R1) or via parallel decoding (dVLA). TRI's paper sharpens their justification: vanilla CoT-at-inference doesn't pay off; you need a specific architectural or training reason to use it.

This is the second most important finding for the field's recipes.

The 2024–2025 consensus had been that discrete action tokens (FAST, BPE, VQ-VAE) are the way to:

1. Share parameters with VLM language heads (one CE loss).
2. Make actions look like language so that a frozen LM can be co-trained.
3. Compress high-frequency action streams.

Knowledge Insulation — the foundation of π0.6/π0.7 — uses FAST tokens specifically as the CE training target for the VLM half, while the continuous flow-matching head remains the action head.

TRI's controlled experiment shows that, with this exact architecture (PaliGemma2 + flow-matching + auxiliary FAST CE head), the FAST CE loss adds no measurable benefit and sometimes hurts unseen-task performance.

Two ways to reconcile:

1. The KI gain comes from gradient routing, not from the token target. That is, what matters is that the VLM's gradient is insulated from the continuous head — the auxiliary CE target was a vehicle for that, not the cause of the gain.
2. The benefit of FAST tokens depends on the eval suite. Possible — TRI's eval is dual-Franka manipulation; PI's flagship evals include very different domains (mobile manipulation, laundry folding). The TRI paper does not claim a universal verdict, only the LBM-scale verdict on its specific eval suite.

Either way, this is now an open empirical question the field needs to settle in 2026 — not a closed-recipe assumption.

Distilled from §III of the paper:

1. Always include large-scale standard VL data (RoboPoint + RefSpatial or equivalent). It gives the largest single-modality gain and prevents VLM forgetting. Cheap; no robot data collection.
2. Always include GPT-5-generated captions over your robot demos. Better than scripted heuristics. Keep them as a CE target throughout phase 2.
3. Include cross-embodiment robot data (OXE) only in phase 1. Drop it for specialization in phase 2.
4. Include human-video captions (not latent action tokens) and keep them throughout both phases.
5. Skip discrete action tokenization (FAST / VQ-VAE) as an auxiliary head unless you have a specific reason (low-data regime, KI-style gradient routing argument).
6. Skip explicit chain-of-thought conditioning at inference. If you want CoT's training-time benefit, use it as a CE auxiliary loss only and predict actions without generating CoT at inference.
7. Two-phase training is the default. Single-phase joint training works too but does not match the two-phase ceiling.

This section consolidates everything needed to reproduce the Final Model ("+Cross-Embodiment-Robot-Data" row in Table S1) — the recipe that achieved the best generalization in the paper. Numbers are pulled from Appendix 2 (Training Details), Table S1 (per-phase data ratios), Table S2 (shared hyperparameters), and Table S3 (human-video data composition).

Note on "two-phase" vs "three-phase": The paper's main text (Fig. 9) reports that the best single-modality policies are either two-phase full co-training (standard VL, human-video annotations) or two-phase 1st-phase-only (scripted/VLM robot annotations, cross-embodiment data). That statement is about isolated modalities. The cumulative Final Model (§III-C, "+Cross-Embodiment-Robot-Data") is a separate model that stacks all effective modalities, and in Table S1 it has populated 1st-/2nd-/3rd-phase columns — i.e. it is genuinely three-phase. The two are not in conflict: "two-phase" is the per-modality finding; "three-phase" is the stacked-recipe schedule. (Audit 2026-06: all numbers in this section were cross-checked against the arXiv HTML full text incl. appendix and confirmed — see report.)

The best-performing policy is a three-phase schedule that mixes five modalities at carefully tuned ratios:

Excluded by ablation: FAST tokens, VQ-VAE tokens, latent video tokens (no measurable gain at LBM scale), CoT-at-inference (degrades).

flowchart TB
  P1["Phase 1 — pure co-training (BS=384)<br/>VL : Lang-anno-TRI-Ramen : Lang-anno-human-video<br/>= 1 : 1 : 1<br/>(no continuous robot actions yet)"]
  P2["Phase 2 — joint specialization (BS=256)<br/>TRI-Ramen : VL : OXE-Ramen : Lang-anno-human<br/>= 4 : 1 : 4 : 1<br/>(introduce target + cross-embodiment continuous actions)"]
  P3["Phase 3 — target specialization (BS=128)<br/>TRI-Ramen : VL : Lang-anno-human<br/>= 9 : 0.5 : 0.5<br/>(drop OXE; keep CE language regularizers)"]
  P1 --> P2 --> P3
  classDef ph fill:#fff9c4,stroke:#f57f17,color:#000
  class P1,P2,P3 ph

Reading the schedule:

• Phase 1 (warm-start the VL side). No continuous-action robot data yet — only language streams. This builds spatial/semantic priors in the VLM half before action gradients touch it.
• Phase 2 (introduce robot continuous actions). Target robot + cross-embodiment robot at 4:4 with VL + human-video captions kept at 1:1 as regularizers. This is the only phase where OXE's morphology gradients enter.
• Phase 3 (drop cross-embodiment, specialize on target). Robot:VL:human-caption = 9 : 0.5 : 0.5 — the 9:1 robot-to-co-training ratio established by the ablation in Fig. S3.

For other "special" policies (latent-action three-phase, FAST-only, etc.), Table S1 ratios are:

Architecture-comparison note (Fig. S1): π0-equivalent and π0.5-equivalent variants use BS=112 for 230k steps (higher memory); LBM (this paper) and πFAST-equivalent use BS=128 for 200k steps.

The combined objective (Eq. 3 of the paper):

$$\mathcal{L} ;=; M_{FM} \cdot \mathcal{L}_{FM} ;+; w \cdot M_{CE} \cdot \mathcal{L}_{CE}$$

where:

• $\mathcal{L}_{FM}$ — flow-matching MSE on continuous actions (target robot + cross-embodiment robot)
• $\mathcal{L}_{CE}$ — cross-entropy on language tokens / discrete action tokens / latent action tokens
• $M_{FM}, M_{CE}$ — per-sample masks selecting which loss applies to which batch item
• $w = 0.02$ — the CE loss weight; tuned in Appendix 1B / Fig. S2

Why w = 0.02? Fig. S2 ablation under single-phase VL co-training:

• $w$ too high → VLM gradient dominates → in-distribution action performance degrades
• $w$ too low → no generalization gain from co-training
• 0.02 is the empirical sweet spot

Robot-to-co-training ratio sweet spot (Fig. S3):

• Higher robot ratio → better in-distribution, worse generalization
• Lower robot ratio → better generalization, worse in-distribution
• 9 : 1 (robot : co-training) chosen for the unpaired modalities (VL, OXE, human videos) in the specialization phase
• 6 : 4 (target : OXE) used during phase-1 cross-embodiment mixing — both contribute to the same flow-matching objective so the higher OXE share is tolerable

LAM training (when used):

• Batch size 1024 on 16 H100s
• 300k steps (~68 hours)
• LR 1e-4 with 1k-step linear warmup and 20k-step cosine decay
• Trained on the full 2,271.6 h human-video corpus

If you only have a fixed budget and want the Final Model:

1. Architecture: PaliGemma2-3B-PT (google/paligemma2-3b-pt-224, trained jointly — not frozen) + 8-layer ActionFT flow-matching transformer, single observation token aggregated from last 4 VLM layers via adaLN.
2. Phase 1 (200k steps, BS=384, 16 H100 ≈ 64 h): VL : TRI-Ramen-LangAnno : Human-LangAnno = 1 : 1 : 1. Pure CE — no continuous actions.
3. Phase 2 (200k steps, BS=256, ≈ 64 h): TRI-Ramen : VL : OXE-Ramen : Human-LangAnno = 4 : 1 : 4 : 1. Introduce flow matching + keep CE.
4. Phase 3 (200k steps, BS=128, ≈ 64 h): TRI-Ramen : VL : Human-LangAnno = 9 : 0.5 : 0.5. Drop OXE.
5. Loss: $\mathcal{L} = M_{FM}\mathcal{L}{FM} + 0.02 \cdot M{CE}\mathcal{L}_{CE}$ throughout.
6. Optimizer: AdamW; LR 2e-5 with 1,000-step linear warmup; cosine decay 2e-5 → 2e-6 over the last 60k of each 200k phase.
7. Total compute: ≈ 192 H100-GPU-days (3 phases × 64 h × 16 H100s) for the multi-task pretrain; +~10 H100-GPU-days per single-task fine-tune (30k steps).

The paper is admirably explicit about its scope:

• One backbone (PaliGemma2-3B-PT, trained jointly). The paper's verdicts may shift for a much larger backbone or a different VLM family.
• One action head (8-layer ActionFT, flow matching). Does not test, e.g., discrete-diffusion action heads (ICLR-2026-Discrete-Diffusion-VLA) or AR action heads (OpenVLA-class).
• Eval is mostly bimanual Franka tabletop. Mobile manipulation (π0.5), humanoid (GR00T), and dexterous-hand (UniHM) are not tested.
• VLM is trained jointly (not frozen) — the co-training-vs-VLM-forgetting result is precisely an artifact of the VLM weights moving: robot-only training drifts them away from the pretrained VL optimum, and VL co-training pulls them back. KI's gradient-routing argument is the alternative design point (insulate the VLM from the continuous-action gradient) that this paper does not adopt.
• Caption-generator quality (GPT-5) is the upper bound on the gain from caption-style modalities. Different vision-language captioners would change the absolute numbers.
• No long-horizon multi-step planning eval at the level where CoT would actually be expected to pay. The CoT-doesn't-help conclusion is therefore a verdict for direct perception-to-action manipulation, not a universal claim. This is the single most likely place where follow-up work will find a counterexample.

What the field still needs:

• The same controlled study at unfrozen-VLM scale.
• The same controlled study with discrete-diffusion action heads — to ask whether the action-head class changes which co-training modalities help.
• A planning-heavy eval suite (CALVIN long-horizon, kitchen-style) to test whether CoT-at-inference pays in environments where multi-step reasoning is genuinely required.

• Per-paper page: LBM Co-training Study
• Knowledge Insulation — the gradient-routing companion
• Hybrid Training — same conclusion on CoT, by curriculum
• Actions as Language — language-relabeling lever
• Human-Video Pretraining — adjacent on human-video data
• VLM4VLA — VLM-benchmark vs VLA-perf disambiguation
• FASTER — discrete-token tokenizer this paper finds unhelpful at LBM scale
• VLA Architectures review
• VLM↔Action Connection review — KI vs. cross-attention vs. shared-stack tradeoffs
• π0.6 long-form review · π0.7 long-form review — the production baseline using KI

Verdict: co-training is the best-measured question in the field — VL and cross-embodiment data compound; discrete action tokens don't; mixing must be joint, not sequential. Ratios remain art.

From folklore to measurement: this page's ICLR 2026 study → the RSS 2026 89-policy sequel (4,000 h, 50M VL samples, 58k sim + 2,835 real rollouts) → independent confirmations from the Qwen program and VLM4VLA.

• Mixture ratios have no theory (λ=0.1 manipulation vs 1.0 navigation in the Qwen suite, undiscussed).
• Scheduling (single- vs multi-phase) shows effects without explanation.
• Per-sample attribution of co-training gains does not exist.
• Budget-constrained adaptation has its own trade-off: ICML 2026's Escaping the Diversity Trap identifies a coverage–density tension (anchor-first, then expand to risky boundaries) that pure mixture-ratio thinking misses.

← Back to Home · ICLR-2026