TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation

Venue: RSS 2026 (Sydney, Jul 13–17) · Session: Manipulation 2 · paper #60 Authors: Rishabh Madan, Angchen Xie, Samantha Saak, Andres Blanco, Dohyeok Lee, Sarah Grace Brown, Yunting Yan, Mark Zolotas, Jose Barreiros, Tapomayukh Bhattacharjee arXiv: 2607.09218 · program page

Summary compiled from the arXiv paper (v2); all numbers quoted from the paper. Trend context: RSS 2026 survey.

Left: why whole-arm contact matters — end-effector-only side rollover exceeds torque limits, while human caregivers brace with the whole arm. Center/right: the robot rolls a life-size manikin using distributed contacts, with the fused observation stack (RGB, depth, proximity mask, tactile taxels) and a force-vs-time trace kept under a safety threshold. Bottom: real roll-outs of side rollover and limb repositioning enabled by TACTIC.

Problem

Whole-arm manipulation — completing tasks by distributing contact across multiple arm links as contacts form, slide, and break — violates assumptions of standard learned manipulation pipelines: configuration tightly couples motion and forces, contact state is partially observed under occlusion, and multi-link contact configurations are sparse in datasets, making purely learned rollouts physically inconsistent under distribution shift.

Method

TACTIC is a receding-horizon sampling-based MPC (MPPI, built on the STORM sampler) with three components: (1) a contact-centric multimodal state fusing RGB-D, distributed tactile sensing, and a compact 2D proximity mask; (2) contact-aware action sampling that uses tactile-derived contact Jacobians to project sampled perturbations toward force-modulating directions (σ_force > σ_null); (3) a hybrid predictive model coupling a ViT-based action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, scored by objectives over predicted proximity and interaction forces plus an IQL-learned terminal value. A two-fidelity scheme keeps ViT evaluations tractable; on hardware the MPPI loop runs at ~12 Hz above a 1 kHz joint-space compliant controller.

Results

In simulation (Maze, Bed Bathing, Tabletop Reach, Granular): full TACTIC reaches 87.2% success on Maze with 39.0 force violations vs DreamerV3 (75.9%, 82.4) and TD-MPC2 (65.1%, 93.9), and 79.5% vs 51.6% on Bed Bathing; ablations show removing tactile+mask drops success to 63.5%, and hybrid rollouts beat latent-only (64.1%) and kinematics-only (43.6%). Hybridization also helps pretrained world models (V-JEPA2 comparable performance with ~34% less finetuning data; ~26% Chamfer reduction for DINO-WM). On a Kinova Gen3 with 22 FlexiForce taxels manipulating a ~46 lb manikin, TACTIC scores 12/20 (Side Rollover) and 14/20 (Limb Repositioning) vs Diffusion Policy's 3/20 and 5/20, with fewer force violations than even expert teleoperation.

Significance

A strong argument that contact-rich physical HRI needs explicit force/contact structure rather than end-to-end policies alone — directly relevant to Review-Tactile-VLA and the model-based side of Review-Dexterous-Manipulation; the open-sourced exoskeleton teleop and tactile skin lower the entry barrier for whole-arm caregiving research.

← Back to RSS 2026 survey · RSS-2026-Papers · Home