Cornell University
Library
Cornell UniversityLibrary

eCommons

Help
Log In(current)
  1. Home
  2. Cornell University Graduate School
  3. Cornell Theses and Dissertations
  4. Learning Robot Manipulation Skills from Human Videos

Learning Robot Manipulation Skills from Human Videos

File(s)
Dan_cornell_0058O_12686.pdf (37.27 MB)
Permanent Link(s)
https://doi.org/10.7298/j2mr-nj45
https://hdl.handle.net/1813/126248
Collections
Cornell Theses and Dissertations
Author
Dan, Prithwish
Abstract

Teaching robots to manipulate objects requires large amounts of demonstrated behavior, yet collecting expert robot demonstrations is expensive, slow, and difficult to scale. Human videos offer an abundant alternative — they are cheap to record, capture a rich diversity of manipulation strategies, and require no robot involvement. However, directly leveraging human videos for robot policy training poses a fundamental challenge: humans and robots differ significantly in embodiment, making human actions difficult or impossible to replicate on a robot. This thesis presents two complementary frameworks that address this cross-embodiment gap. The first contribution, X-Sim, sidesteps execution differences entirely by avoiding action retargeting. Rather than extracting robot actions from human hand motion, X-Sim identifies object motion — observable in any RGBD video — as a dense, embodiment- agnostic supervisory signal. A photorealistic simulation is reconstructed from the human video, and tracked object trajectories define object-centric rewards for training reinforcement learning policies in simulation. These policies are distilled into image- conditioned diffusion policies and transferred to the real world via an online domain adaptation technique that aligns simulated and real observations at test time, requiring zero robot teleoperation data. Across five manipulation tasks, X-Sim improves task progress by 30% over hand-tracking baselines and achieves comparable performance to behavior cloning with 10× less data collection time. The second contribution, X-Diffusion, takes an alternate approach that avoids the complexity of simulation and RL by incorporating human demonstrations directly into imitation learning. It treats human hand actions as noisy counterparts of robot actions, grounded in the observation that as Gaussian noise is added during the forward diffusion process, low-level embodiment-specific differences fade while high-level task structure is preserved. X-Diffusion trains a classifier to identify, for each human action, the minimum noise level at which it becomes indistinguishable from a robot action, and selectively integrates human demonstrations into diffusion policy training only beyond this threshold. This adaptive co-training avoids learning kinematically infeasible be- haviors while still extracting useful guidance from the full human dataset, improving average task success by 16% over naive co-training and manual data filtering across five real-world manipulation tasks. Together, these works demonstrate two principled strategies for bridging the human– robot embodiment gap — one through simulation as an intermediary, and one through the structure of the diffusion process itself — establishing human video as a viable and scalable source of supervision for robot manipulation learning.

Description
85 pages
Date Issued
2026-05
Keywords
Learning from Human Videos
•
Representation Learning
•
Sim-to-Real
Committee Chair
Choudhury, Sanjiban
Committee Member
Ma, Wei-Chiu
Bhattacharjee, Tapomayukh
Degree Discipline
Computer Science
Degree Name
M.S., Computer Science
Degree Level
Master of Science
Rights
Attribution 4.0 International
Rights URI
https://creativecommons.org/licenses/by/4.0/
Type
dissertation or thesis

Site Statistics | Help

About eCommons | Policies | Terms of use | Contact Us

copyright © 2002-2026 Cornell University Library | Privacy | Web Accessibility Assistance