ICML 2026 Tabero - Heungwoo/research GitHub Wiki

Tabero: Gentle Manipulation with Closed-Loop Force Feedback — a vision-tactile-language benchmark and VTLA model

Venue: ICML 2026 (Poster) Category: Tactile Affiliations: Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, Renjing Xu Traction (2026-06): 0 citations (arXiv)

Tabero-VTLA system overview: tactile inputs encoded and fused with vision/language, with real-time force feedback (Figure 3 from Wu et al., 2026)

Problem

Tactile sensing is essential for human-like gentle manipulation, but existing VLA models cannot exploit touch for force-aware control. Two gaps block progress: (1) aligned vision-tactile-language data is scarce, and (2) there is no effective closed-loop force-feedback mechanism — policies trained on synthetic data without force feedback frequently damage objects during manipulation.

Method

Tabero is both a benchmark and a model suite. Benchmark: a high-fidelity simulation platform integrating Isaac Lab with advanced tactile simulation, plus a data-efficient pipeline that repurposes open-source robot manipulation trajectories to generate synchronized streams of multi-view vision, tactile images, marker displacement fields, contact force, and proprioception — yielding diverse vision-tactile-language tasks. It also defines a multidimensional evaluation protocol that measures task success and physical interaction quality (e.g., average grip force), going beyond pure success rate.

Tabero-VTLA is built on the π0 infrastructure with flow matching, enabling continuous joint prediction of future gripper poses and fingertip force setpoints from vision, language, and touch. The default tactile modality is the marker motion field (encoding normal/shear force and torque, generalizing across piezoelectric, magnetic, and vision-based sensors). Tactile signals enter through three injection options: a Force-Field Tokenizer (a lightweight TCN over (H+1)×N×2 marker displacement frames), a Tactile-Image Adapter (composite left/right-finger tactile image through the RGB-D visual encoder, fused via cross-attention), and a Force Tokenizer (6D fingertip force vectors through an MLP). A decoupled force-position command interface separates grip force from applied translational force; a fixed admittance-based hybrid controller executes these commands, so the policy reasons about appropriate interaction forces while the controller supplies physical compliance. Grip force is F_grip = 2·min(|F_left,z|, |F_right,z|) and applied force F_applied = −(F_left+F_right).

High-fidelity multimodal data-generation pipeline that repurposes open-source trajectories (Figure 2 from Wu et al., 2026)

Results

Headline: "reduces average grip force by over 70% under gentle instructions" while maintaining high task success. In the tactile-modality ablation (Table 3), the no-tactile baseline fails completely (0% success); adding tactile tokens plus force supervision recovers strong performance, with the best configuration Field+FS reaching 0.86 firm / 0.52 gentle success and dropping average grip force from 32.4 N (firm) to 3.7 N (gentle). Semantic-force generalization (Table 4) shows the model modulates force by linguistic adverb — "firmly/tightly" → 32.4 N average contact force vs. "gently/softly" → 3.7 N — and extrapolates to out-of-domain adverbs ("lightly" → 14.5 N, "forcefully" → 18+ N). Code: github.com/NathanWu7/Tabero.

Significance

Tabero tackles the two practical blockers to force-aware VLA — data scarcity and closed-loop force control — by recycling existing manipulation datasets into tactile-rich tasks and by cleanly decoupling learned force intent from a fixed compliant controller. The semantic-force results show language can directly govern interaction force, a key step toward safe, object-preserving manipulation.