IROS 2026 Articulated Tools Sim2Real - Heungwoo/research GitHub Wiki

IROS 2026 — In-Hand Manipulation of Articulated Tools with Sim-to-Real Transfer

Venue: IROS 2026 (Pittsburgh) · paper #3176 · UC San Diego (Atar, Huang, Richter, Yip). Paper: arXiv 2509.23075 · project (UCSD ARCLab). Representative of: sim-to-real robustness × articulated-object manipulation — a sim-trained base policy refined online from hardware demos to handle real-world friction/backlash on articulated tools (scissors, pliers, staplers, surgical instruments). Companions: Dexterous Manipulation · VLA Evaluation · IROS 2026 survey.

Pipeline — a simulation-trained base policy for the articulated-tool skill is augmented with a sensor-driven refinement learned from hardware demonstrations that fuses whole-hand tactile + force–torque feedback with the policy's action intent via cross-attention, adapting online to instance-specific articulation (friction/stiction/backlash) (pipeline figure from Atar et al., arXiv 2509.23075, © the authors)

1. Problem

RL + sim-to-real advanced rigid-object manipulation, but policies stay brittle for articulated mechanisms: contact-rich dynamics need stable grasping and simultaneous free in-hand articulation. Real articulated objects and hands have under-modeled friction, stiction, backlash that widen the sim-to-real gap, and real tactile sensing falls short of the idealized sim.

2. Method

A two-stage recipe that closes the sim-to-real gap for articulated in-hand manipulation:

  • Sim-trained base policy for the articulated-tool skill.
  • Sensor-driven refinement from hardware demonstrations — conditions on proprioception + target articulation state while fusing whole-hand tactile + force–torque feedback with the policy's action intent via cross-attention.
  • The controller adapts online to instance-specific articulation properties, stabilizes contact, and regulates internal forces under perturbation.

3. Results

  • Validated on diverse real tools — scissors, pliers, minimally-invasive surgical instruments, staplers — with robust sim-to-real transfer, disturbance resilience, and generalization across structurally-related articulated tools without precise physical modeling.

4. Why it matters

This is IROS 2026's sim-to-real robustness representative, and it extends dexterity from rigid/cube in-hand (Proprioceptive Transformer) to articulated tools — a distinct capability (grasp + articulate simultaneously). The transferable idea is sim base + hardware-demo sensor refinement fused by cross-attention: rather than closing the sim-to-real gap purely with domain randomization, it learns a real-world correction conditioned on tactile/F-T — the "servo on real contact" answer to under-modeled friction/backlash. Connects to the data-pyramid L5→L6 (sim + a small real refinement tip) and the evaluation-robustness themes.

Limitations (reviewer): needs hardware demos for the refinement stage (not pure sim-to-real); articulated-tool-scoped; tactile/F-T-equipped hand required.

5. Links

← Back to IROS 2026 survey · Home