CoRL 2026 World In Your Hands - Heungwoo/research GitHub Wiki

CoRL 2026 โ€” World In Your Hands: Open Ecosystem for Human-Centric Manipulation in the Wild

Venue: CoRL 2026 (Austin, TX, Nov 9โ€“12). Paper: arXiv 2512.24310. Representative of: scaling human manipulation data beyond teleoperation โ€” 1,000+ h in-the-wild human data + open capture hardware. Companions: Egocentric Video Pre-Training ยท Human Video โ†’ Robot Transfer ยท CoRL 2026 survey.

WIYH ecosystem teaser: wearable Oracle Suite capture, in-the-wild human manipulation data, and downstream robot transfer (figure from the authors, arXiv 2512.24310, ยฉ the authors)

1. Problem

Robot manipulation data is bottlenecked by teleoperation: expensive, slow, and confined to lab setups. Human hands are the most abundant source of dexterous manipulation, but capturing it in the wild with the geometric precision robots need โ€” and labeling it at scale โ€” has been unsolved. WIYH targets this gap with an open, end-to-end ecosystem for collecting and learning from human-centric manipulation data.

2. Method

World In Your Hands (WIYH) has three parts:

  • Oracle Suite โ€” a wearable capture kit: an H-FPVHive chest module (two fisheye + two pinhole cameras, four infrared modules), H-Gloves (each: a Manus glove, six IMUs, an onboard MCU, three fisheye cameras), and an H-Backpack with an NVIDIA Orin for onboard compute, storage, and power. An auto-labeling pipeline fuses infrared, RGB, and IMU via online sync + offline motion reconstruction to recover 6D wrist-pose trajectories for both hands.
  • WIYH Dataset โ€” 1,045 hours of in-the-wild egocentric manipulation, ~100K annotated atomic-action episodes across 100+ skills, with millimeter-scale accuracy (mean positional error < 5 mm). Modalities: RGB, depth, masks, calibrated observations, 3D hand/wrist trajectories, hand skeletons, and language.
  • Annotations & benchmarks โ€” a multi-stage pipeline yielding atomic-action instructions, perception labels (masks, depth), and vision-language supervision (task / sub-task descriptions, reasoning annotations), supporting tasks from perception to action.

3. Results

Co-training robot policies with WIYH human-centric data lifts manipulation success in cluttered scenes from 8% โ†’ 60% (500 robot clips, +52 pts); with only 200 robot clips, 0% โ†’ 32%. Reported real-robot transfer improves success from 15% โ†’ 70% with WIYH pre-training.

4. Why it matters

WIYH shows that in-the-wild human data โ€” captured with a portable, open kit rather than robot teleop โ€” can substantially close the clutter gap that stalls real-world policies, and it releases the whole stack (data, hardware designs, code) rather than just a model. That makes it a scaling recipe others can reproduce and extend.

Limitations (reviewer): human-hand data still faces an embodiment gap (morphology, dynamics) that co-training only partially bridges; headline gains are on a limited set of task-matched robot scenes; and mm-scale capture depends on the specialized multi-sensor suite, so quality away from that rig is unverified.

5. Links

โ† Back to CoRL 2026 survey ยท Home