Learning Robot Manipulation Skills from Human Videos
Teaching robots to manipulate objects requires large amounts of demonstrated behavior, yet collecting expert robot demonstrations is expensive, slow, and difficult to scale. Human videos offer an abundant alternative — they are cheap to record, capture a rich diversity of manipulation strategies, and require no robot involvement. However, directly leveraging human videos for robot policy training poses a fundamental challenge: humans and robots differ significantly in embodiment, making human actions difficult or impossible to replicate on a robot. This thesis presents two complementary frameworks that address this cross-embodiment gap. The first contribution, X-Sim, sidesteps execution differences entirely by avoiding action retargeting. Rather than extracting robot actions from human hand motion, X-Sim identifies object motion — observable in any RGBD video — as a dense, embodiment- agnostic supervisory signal. A photorealistic simulation is reconstructed from the human video, and tracked object trajectories define object-centric rewards for training reinforcement learning policies in simulation. These policies are distilled into image- conditioned diffusion policies and transferred to the real world via an online domain adaptation technique that aligns simulated and real observations at test time, requiring zero robot teleoperation data. Across five manipulation tasks, X-Sim improves task progress by 30% over hand-tracking baselines and achieves comparable performance to behavior cloning with 10× less data collection time. The second contribution, X-Diffusion, takes an alternate approach that avoids the complexity of simulation and RL by incorporating human demonstrations directly into imitation learning. It treats human hand actions as noisy counterparts of robot actions, grounded in the observation that as Gaussian noise is added during the forward diffusion process, low-level embodiment-specific differences fade while high-level task structure is preserved. X-Diffusion trains a classifier to identify, for each human action, the minimum noise level at which it becomes indistinguishable from a robot action, and selectively integrates human demonstrations into diffusion policy training only beyond this threshold. This adaptive co-training avoids learning kinematically infeasible be- haviors while still extracting useful guidance from the full human dataset, improving average task success by 16% over naive co-training and manual data filtering across five real-world manipulation tasks. Together, these works demonstrate two principled strategies for bridging the human– robot embodiment gap — one through simulation as an intermediary, and one through the structure of the diffusion process itself — establishing human video as a viable and scalable source of supervision for robot manipulation learning.