Other Titles



Holistic scene perception aims to provide comprehensive knowledge of the objects and environment in a scene, as well as the intrinsic relationships between them, which plays an important role in human cognition. Driven by the desireto build future cognitive robots that can form collaborative teams with humans, scene perception for autonomous robots has been at the frontier of the interdisciplinary research combining computer vision and robotics in the recent decades. Although humans are capable of perceiving variegated visual scenes effortlessly, such tasks remain difcult for autonomous robots. The primary challenge lies in the extraction of implicit and hidden context, such as the interaction between objects, and the integration of it with the explicit and task-relevant visual features to interpret the scene. This dissertation tackles the above challenge by providing a novel framework for holistic scene perception that integrates domain knowledge, image recognition, state estimation, inference of hidden variables, and anticipation of future actions. The proposed approach is tested in dynamic scenes that depict human team activities, such as the team sport of volleyball, with complex goals and variegated interactions. The approach relies on a novel dynamic Markov random feld model to infer hidden variables in the scene, which are then combined with visual features and domain knowledge to perform action anticipation using a multi-layer perceptron. In addition, recent advancements in robotics and processing capabilities pointto a future in which mobile robots equipped with onboard sensors will be able to perceive the environment as humans do. Therefore, the second part of this dissertation investigates scene perception from the aspect of employing a network of mobile robots to effectively recognize and track a larger number of dynamic targets. Of critical interest in this problem is the maximization of tracking quality by simultaneously determining the coordination and control of the robot network. This dissertation presents a new decomposition based framework to efciently solve the network optimization problem in two stages. Two novel decentralized coordination methods are proposed to find adaptive and conflict-free target assignments. Then, robots locally and concurrently determine their control to maximize a new tracking utility function in real time. Physical experiments with a network of ground robots tracking human targets validate the applicability of the proposed approach in real-world applications. Finally, the long term vision for holistic scene perception is to have intelligent robots share common goals and perceive the targets and environments collaboratively with humans to gain improved efciency and robustness. A collaborativehuman-robot team has the advantage of leveraging complementary skills such as human feld experience and domain knowledge, and robot data processing and integrated sensor modalities. This dissertation develops a new collaborative control and communication framework applicable to human-robot teams engaged in visually detecting and tracking many targets in an obstacle-populated environment. In both numerical simulations and physical experiments, this new collaborative control and communication framework is shown to be capable of providing robust performance in the presence of uncertainties such as state estimation errors and intruders.

Journal / Series

Volume & Issue



Date Issued




Human-robot collaboration; Markov random field; Multi-target tracking; Optimization; Robot network control; Role Inference


Effective Date

Expiration Date




Union Local


Number of Workers

Committee Chair

Ferrari, Silvia

Committee Co-Chair

Committee Member

Frazier, Peter
MacMartin, Douglas

Degree Discipline

Mechanical Engineering

Degree Name

Ph. D., Mechanical Engineering

Degree Level

Doctor of Philosophy

Related Version

Related DOI

Related To

Related Part

Based on Related Item

Has Other Format(s)

Part of Related Item

Related To

Related Publication(s)

Link(s) to Related Publication(s)


Link(s) to Reference(s)

Previously Published As

Government Document




Other Identifiers


Rights URI


dissertation or thesis

Accessibility Feature

Accessibility Hazard

Accessibility Summary

Link(s) to Catalog Record