Venue: ICLR 2026 Category: Embodied reasoning · Planning Trend tag: 3D grounding · Embodiment-aware planning Authors: Huawei Noah's Ark Lab. Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang (equal); corresponding Yuzheng Zhuang.

flowchart LR
  subgraph Inputs[Inputs]
    Img[RGB images / video<br/>16 frames train · 32 inference] 
    Depth[Depth maps + intrinsics K + extrinsics M_i]
    Text[Instruction T]
  end
  Text --> ST[all-MiniLM-L6-v2<br/>sentence transformer · d_st=384]
  Img --> ViT[InternVL3-8B ViT encoder<br/>448×448 patches]
  Depth --> P3D[3D world coords<br/>0.1 m voxel · sinusoidal patch PE]
  ViT --> AvgP[Avg-pool → V_avg^I · 1024-d]
  ST --> Concat[Concat<br/>1024+384]
  AvgP --> Concat
  Concat --> MLP[2-layer MLP · hidden 256] --> Logits[V^g ∈ R²]
  Logits --> Gumbel["Gumbel-Softmax · hard gating g ∈ {0,1}"]
  ViT --> Hybrid[V_hybrid = V^I + g · V^p]
  P3D --> Hybrid
  Hybrid --> LLM[InternVL3-8B LLM]
  Text --> LLM
  LLM --> Resp[Response: NL · 2D/3D pts · 3D bbox]
  LLM -.SFT then GRPO.- TEGRPO[TE-GRPO: r_format + r_task + λ_t · r_embod]

Two well-defined gaps in MLLM-based embodied planners:

1. Geometric Adaptability Gap. 2D-only models (SpatialVLM, RoboPoint, RoboBrain) fail at tasks requiring depth, occlusion handling, or stacking; 3D-LLMs (3D-LLM, LLaVA-3D, 3DRS) hard-code 3D injection regardless of task — wasting compute and adding noise when 3D inputs are incomplete or irrelevant.
2. Embodiment Constraint Gap. Models trained on web-scale or rule-based-synthetic data often produce plans that are theoretically valid but kinematically infeasible — they ignore object affordances, workspace limits, and arm reachability.

OmniEVA addresses both with a single architecture.

• Base MLLM: InternVL3-8B (Zhu et al., 2025).
• Sentence encoder for the gate: all-MiniLM-L6-v2 (Reimers & Face, 2021) — 384-d.
• Image resolution: 448 × 448.
• Video sampling: 16 frames train / 32 frames inference.
• 3D voxel size: 0.1 m (coordinates discretized for both inputs and outputs).
• Output formats: natural language; 2D points in [0, 1000]; 3D points/boxes in 0.1 m grid.

Patch-level 3D positional encoding. Each depth image D_i ∈ R^H×W is reprojected into world coordinates P_i ∈ R^H×W×3 using camera intrinsics. Coords are partitioned into ViT-aligned patches and averaged → P′_i ∈ R^H_p×W_p×3. A sinusoidal encoding maps each patch to a d_v-dimensional vector → V^p.

Two conditioning signals to the gate:

• Task condition: V^T = SentenceTransformer(T) ∈ R³⁸⁴
• Scene condition: V_avg^I = AvgPool(V^I) ∈ R¹⁰²⁴

Gate logits: V^g = MLP_ψ(Concat([V^T, V_avg^I])) ∈ R²

Hard gating via Gumbel-Softmax (the paper argues against soft gating because soft-weighted sinusoidal PEs distort numerical stability):

g = GumbelSoftmax(V^g, τ) ∈ {0, 1} V_hybrid^I = V^I + g · V^p

This is mathematically equivalent to a 2-expert Mixture-of-Experts between pure-vision and 3D-augmented tokens. A KL term against Bernoulli(0.5) regularizes gating:

L_total^ψ,θ = L^CE(o_label, o) + α · L^KL_ψ(V^g ∥ P_prior)

Stage 1 — TAGR Pretraining. ViT frozen; sentence transformer frozen; MLP gate trained with lr=1e-4; LLM updated with low lr=5e-7. Datasets include depth-aware spatial reasoning data from ScanNet, Matterport3D, 3RScan, ArkitScenes. After pretraining, TAGR parameters are frozen and the fine-tuned LLM is discarded to avoid interference. Gumbel temperature scheduled exponentially: τ = τ_init·exp(−4.5·steps / max_steps), τ_init=1.0 → τ_min=0.05.

Stage 2 — Supervised Fine-Tuning. Builds OmniEVA-Base. Hybrid dataset of (a) general embodied reasoning (2D images, video, 3D for spatial referring + scene captioning) and (b) custom embodied task data (navigation + manipulation). ViT frozen, LLM fine-tuned at lr=1e-5. Batch size 256; 1 epoch; cosine LR schedule.

Stage 3 — TE-GRPO RFT. Builds OmniEVA-ER. Builds on GRPO (Shao et al., 2024) with three rewards:

• r_format — encourages "......" structure.
• r_task = EvalTask(q, o_i) ∈ [0,1] — semantic task completion (e.g., fraction of points inside a target region).
• r_embod = EvalExec(q, o_i) ∈ {0, 1} — kinematic feasibility check via a motion planner in simulation (verifies reachability + workspace + collisions).

Progressive embodiment curriculum:

r_i,t^acc = r_i^task · (λ_t · r_i^embod + (1 − λ_t))

λ_t increases over time: early training rewards task completion alone; later steps demand strict embodiment feasibility. Group size G = 8 candidates per training step; LLM at lr=1e-5; batch size 128; AdamW; bf16; weight decay 0.1; gradient clip 1.0.

Trained only on SQA3D + ScanQA + Scan2Cap + ScanRefer training splits, fair comparison vs Video-3D-LLM and 3DRS:

Cross-attention is catastrophic on Scan2Cap (-50 points). Soft gating consistently underperforms hard gating because soft weights distort the sinusoidal positional encodings.

3D activation rate by prompt cluster: Shape 76.9%, Action/Activity 50.9%, Visibility/Occlusion 33.0%, Direction/Orientation 30.5%, Identification 27.5%, Color 24.2%, Comparison 21.4%, Material/Texture 19.4%, Arrangement Pattern 13.6%, Counting 9.0%, State Condition 6.3%. The router cleanly learns that geometric and dynamic prompts need 3D, while descriptive/numerical ones don't.

Compact 8B model surpasses 32B and 72B baselines; +10.45 average gain over RoboBrain2.0-32B. PACO-LVIS jump (16.2 → 21.0) and RoboRefit jump (70.0 → 91.2) are particularly large.

Leads on three of four benchmarks. On ScanRefer (3D visual grounding) it slightly trails; the paper notes OmniEVA-Base reaches 55.8 [email protected] using only text I/O without external detectors or task-specific heads — vs prior text-only best of 44.4.

+5.4 SPL on HM3D over UniNavid; first work to score >40% SPL on HM3D-OVON in this setting.

Six tasks comparing RoboBrain2.0-7B, OmniEVA-Base, OmniEVA-ER w/o r_embod, OmniEVA-ER (full):

Where2Approach goes from 6.25 → 39.86 (a +33.6 jump); Mobile Placement (Hard) jumps 7 → 52. Removing r_embod systematically degrades performance.

Models trained on arm lengths {75, 88, 110 cm}; evaluated on seen and unseen lengths from 72 cm to 105 cm:

OmniEVA-ER attains 80.5% on unseen arm lengths — +17 points over OmniEVA-Base.

Two off-the-shelf wheeled mobile robots with different dual 6-DoF arm spans (75 cm vs Galaxea R1 at 70 cm), 10 trials per task:

OmniEVA-ER particularly excels on the more-constrained 70 cm embodiment for placement.

1. Scene-level gating. TAGR makes one binary decision per (scene, task) — heterogeneous environments can lead to suboptimal 3D feature integration. The activation analysis shows unexpectedly low gate activation on some spatial-relationship prompts. Future work: patch-level gating for finer-grained 3D adaptation.
2. Unmodeled physical constraints. Embodiment-aware planning currently models arm length; arm DoFs, installation height, joint torque limits are unmodeled.

OmniEVA is a counterpoint to "always-3D" embodied planners and a complement to "always-2D" VLAs:

• vs SpatialVLM / RoboPoint: they trained on 2D-only spatial QA; OmniEVA shows that adding 3D, but only when needed outperforms both alternatives — including hard-coded 3D fusion (Scan2Cap +2.4, ScanRefer +1.9, average +1.4).
• vs 3D-LLM / LLaVA-3D / Video-3D-LLM / 3DRS: these always inject 3D; OmniEVA's gate matches or beats them with cleaner 2D performance.
• vs RoboBrain 2.0 (7B/32B): RoboBrain has high-level planning + low-level pointing; OmniEVA-Base at 8B beats RoboBrain2.0-32B by +10.45 average across in-house embodied benchmarks while adding 3D awareness.
• vs π-family / Embodied-R1: OmniEVA stays at the planning layer (predicts subgoals/points/3D boxes); it is complementary to low-level VLA controllers.
• vs From Seeing to Doing: both push planners toward producing feasible (not just coherent) plans. OmniEVA does this through TE-GRPO with a kinematic-feasibility reward.

The TAGR + TE-GRPO combination — gate the 3D pathway dynamically, then RL-finetune on a feasibility reward — is a clean recipe other 3D-LLM efforts can adopt.

• OpenReview: https://openreview.net/forum?id=tkEmIJv1tB
• PDF: https://openreview.net/pdf?id=tkEmIJv1tB

• From Seeing to Doing
• RoboInter
• Embodied-R1

← Back to ICLR-2026