ICML 2026 Joint Navigation and Manipulation Planning - Heungwoo/research GitHub Wiki

Joint Navigation and Manipulation Planning with 3D Interaction Chains — Unifying OVMM planning over a shared 3D map

Venue: ICML 2026 (Poster) Category: VLA Architecture / Mobile Manipulation Affiliations: Keming Zhang, Sixian Zhang, Xinhang Song, Hongyu Wang, Yiyao Wang, Yingjie Wang, Shuqiang Jiang

Problem

Open-vocabulary mobile manipulation (OVMM) requires both long-horizon navigation in unseen environments and object-centric manipulation. The common practice of treating navigation and manipulation as separate, sequential planning stages introduces inefficiencies: a navigation goal chosen without regard to downstream graspability can leave the robot poorly positioned to manipulate, forcing wasteful re-positioning. The paper argues these two phases should be planned jointly.

Method

The proposed framework, 3D Interaction Chains (3D-IC), integrates navigation and manipulation planning over a single shared representation. It maintains a shared 3D feature map of the scene, generates stage-aligned interaction waypoints from that map, and chains them together into candidate multi-stage plans. A hierarchical policy then scores each chain by jointly considering (1) feasibility, assessed through vision-language-model (VLM) reasoning, and (2) transition cost between waypoints. The robot executes the selected waypoints and replans as new observations arrive, closing the loop between perception and action.

flowchart TD
    A[RGB-D observations] --> B[Shared 3D feature map]
    B --> C[Generate stage-aligned interaction waypoints]
    C --> D[Chain waypoints into candidate plans]
    D --> E[Hierarchical policy scoring]
    E --> F[VLM feasibility reasoning]
    E --> G[Transition cost]
    F --> H[Select best 3D interaction chain]
    G --> H
    H --> I[Execute waypoints]
    I --> J[New observations -> replan]
    J --> B

Results

The method was evaluated both in simulation and on a physical Stretch 3 mobile manipulator. Across these settings it showed improvements in task success rates and in trajectory efficiency relative to staged baselines, indicating that jointly scoring navigation-and-manipulation waypoint chains yields both more reliable task completion and shorter, less wasteful robot trajectories. (Detailed numeric tables were not available from the abstract source.)

Significance

3D Interaction Chains reframes OVMM as a single coupled planning problem over a shared 3D map rather than a navigate-then-manipulate pipeline. By using VLM reasoning to judge the feasibility of open-vocabulary interaction waypoints and folding transition cost into the same objective, the approach directly targets the positioning inefficiencies that arise when the two phases are decoupled — a step toward more efficient general-purpose mobile manipulation in unseen environments.