eCommons

 

HOLISTIC SCENE PERCEPTION FOR COLLABORATIVE HUMAN-ROBOT TEAMS

dc.contributor.authorDong, Junyi
dc.contributor.chairFerrari, Silviaen_US
dc.contributor.committeeMemberFrazier, Peteren_US
dc.contributor.committeeMemberMacMartin, Douglasen_US
dc.date.accessioned2024-01-31T21:18:49Z
dc.date.issued2023-05
dc.description.abstractHolistic scene perception aims to provide comprehensive knowledge of the objects and environment in a scene, as well as the intrinsic relationships between them, which plays an important role in human cognition. Driven by the desireto build future cognitive robots that can form collaborative teams with humans, scene perception for autonomous robots has been at the frontier of the interdisciplinary research combining computer vision and robotics in the recent decades. Although humans are capable of perceiving variegated visual scenes effortlessly, such tasks remain difcult for autonomous robots. The primary challenge lies in the extraction of implicit and hidden context, such as the interaction between objects, and the integration of it with the explicit and task-relevant visual features to interpret the scene. This dissertation tackles the above challenge by providing a novel framework for holistic scene perception that integrates domain knowledge, image recognition, state estimation, inference of hidden variables, and anticipation of future actions. The proposed approach is tested in dynamic scenes that depict human team activities, such as the team sport of volleyball, with complex goals and variegated interactions. The approach relies on a novel dynamic Markov random feld model to infer hidden variables in the scene, which are then combined with visual features and domain knowledge to perform action anticipation using a multi-layer perceptron. In addition, recent advancements in robotics and processing capabilities pointto a future in which mobile robots equipped with onboard sensors will be able to perceive the environment as humans do. Therefore, the second part of this dissertation investigates scene perception from the aspect of employing a network of mobile robots to effectively recognize and track a larger number of dynamic targets. Of critical interest in this problem is the maximization of tracking quality by simultaneously determining the coordination and control of the robot network. This dissertation presents a new decomposition based framework to efciently solve the network optimization problem in two stages. Two novel decentralized coordination methods are proposed to find adaptive and conflict-free target assignments. Then, robots locally and concurrently determine their control to maximize a new tracking utility function in real time. Physical experiments with a network of ground robots tracking human targets validate the applicability of the proposed approach in real-world applications. Finally, the long term vision for holistic scene perception is to have intelligent robots share common goals and perceive the targets and environments collaboratively with humans to gain improved efciency and robustness. A collaborativehuman-robot team has the advantage of leveraging complementary skills such as human feld experience and domain knowledge, and robot data processing and integrated sensor modalities. This dissertation develops a new collaborative control and communication framework applicable to human-robot teams engaged in visually detecting and tracking many targets in an obstacle-populated environment. In both numerical simulations and physical experiments, this new collaborative control and communication framework is shown to be capable of providing robust performance in the presence of uncertainties such as state estimation errors and intruders.en_US
dc.description.embargo2024-06-13
dc.identifier.doihttps://doi.org/10.7298/qsj0-w592
dc.identifier.otherDong_cornellgrad_0058_13380
dc.identifier.otherhttp://dissertations.umi.com/cornellgrad:13380
dc.identifier.urihttps://hdl.handle.net/1813/114022
dc.language.isoen
dc.subjectHuman-robot collaborationen_US
dc.subjectMarkov random fielden_US
dc.subjectMulti-target trackingen_US
dc.subjectOptimizationen_US
dc.subjectRobot network controlen_US
dc.subjectRole Inferenceen_US
dc.titleHOLISTIC SCENE PERCEPTION FOR COLLABORATIVE HUMAN-ROBOT TEAMSen_US
dc.typedissertation or thesisen_US
dcterms.licensehttps://hdl.handle.net/1813/59810.2
thesis.degree.disciplineMechanical Engineering
thesis.degree.grantorCornell University
thesis.degree.levelDoctor of Philosophy
thesis.degree.namePh. D., Mechanical Engineering

Files

Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
Dong_cornellgrad_0058_13380.pdf
Size:
5.47 MB
Format:
Adobe Portable Document Format