IROS 2026 Proprioceptive Transformer - Heungwoo/research GitHub Wiki

IROS 2026 — Proprioceptive Transformer: Robust In-Hand Manipulation from Joint Sensing Only

Venue: IROS 2026 (Pittsburgh) · paper #3215 · ETH Zurich (Yao, Yang, Kim, Sympetheros, Katzschmann) · Best Paper / Best Student Paper Award session. Paper: arXiv 2605.21330. Representative of: in-hand dexterity (reorientation) — continuous cube rotation on a tendon-driven hand using joint sensing alone, no vision or tactile. Companions: Dexterous Manipulation · RL for VLA · IROS 2026 survey.

Proprioceptive Transformer — a teacher trained by RL with privileged object info is distilled into PT, which consumes only joint position/velocity histories (16 magnetic angle sensors on the ORCA hand) through a Transformer that extracts implicit object state from temporal joint patterns; no vision/tactile at inference (figure from Yao et al., arXiv 2605.21330, © the authors)

1. Problem

In-hand manipulation usually leans on vision or tactile to track object state, while joint sensing — the most readily available modality on any hand — is largely overlooked, especially for tendon-driven hands. How far can joint sensing alone go?

2. Method

The Proprioceptive Transformer (PT) is an exteroception-free policy for continuous cube rotation using only joint feedback:

  • A teacher policy is trained by RL with privileged object info, then distilled into PT, which operates solely on joint position + velocity histories.
  • A Transformer extracts implicit object-state information from temporal patterns in joint-sensor readings (proprioception → implicit exteroception).

3. Results

  • On the real ORCA hand: 3.1× higher rotation speed than baselines, and 23.4% lower RMSE for cube-pose estimation — i.e. joint-only sensing recovers object state better than baselines and controls faster, with no external perception.

4. Why it matters

PT is IROS 2026's in-hand-reorientation representative and a pointed result: proprioception alone can carry dexterous in-hand control, recovering implicit exteroceptive state via a Transformer over joint-history. It's the minimalist counterpoint to the tactile-heavy dexterity line (T-Rex, MrGrasp) — showing that the cheapest, always-present modality is underexploited. The teacher-distillation recipe echoes DreamMimic's privileged-teacher → deployable-student pattern. (An award finalist — a rare "technically representative AND recognized" pick.)

Limitations (reviewer): demonstrated on cube rotation (the canonical in-hand benchmark) on one hand (ORCA); joint-only sensing may not extend to arbitrary object geometries / fine contact tasks where tactile is decisive; RL-teacher needs sim privileged info.

5. Links

← Back to IROS 2026 survey · Home