2026-09-21T07:50:22Zhttps://ecommons.cornell.edu/server/oai/requestoai:ecommons.cornell.edu:1813/516192026-05-15T19:46:13Zcom_1813_35col_1813_47
Sung, Jaeyong
Saxena, Ashutosh
Salisbury, J. Kenneth
Selman, Bart
Guimbretière, François
Marschner, Steve
2017-07-07T12:48:41Z
2017-12-08T07:00:46Z
2017-05-30
ProQuest Submission ID: 10207
ProQuest Publication ID: 10258261
https://hdl.handle.net/1813/51619
https://doi.org/10.7298/X43R0R0W
9948842
The ability to reason about different modalities of information, for the purpose of physical interaction with objects, is a critical skill for assistive robots. For a robot to be able to assist us in our daily lives, it is not feasible to train each robot for a large number of tasks with all instances of objects that exist in human environments. Robots will have to generalize their skills by jointly reasoning with various sensor modalities such as vision, language and haptic feedback. This is an extremely challenging problem because each modality has intrinsically different statistical properties. Moreover, even with expert knowledge, manually designing joint features between such disparate modalities is difficult.
In this dissertation, we focus on developing learning algorithms for robots that model tasks involving interactions with various objects in unstructured human environments --- especially on novel objects and scenarios that involve sequences of complicated manipulation. To this end, we develop algorithms that learn shared representations of multimodal data and model full sequences of complex motions. We demonstrate our approach on several different applications: understanding human activities in unstructured environment, synthesizing manipulation sequences for under-specified tasks, manipulating novel appliances, and manipulating objects with haptic feedback.
en_US
machine learning
Multimodal Data
Robotic Manipulation
Robot Learning
Artificial intelligence
Deep Learning
Computer science
Robotics
Learning to Manipulate Novel Objects for Assistive Robots
dissertation or thesis
application/pdf
Computer Science
Cornell University
Doctor of Philosophy
Ph. D., Computer Science
https://hdl.handle.net/1813/59810
Publication
Cornell Theses and Dissertations
Sung, Jaeyong
5893a6ea-7af3-41d7-abc6-04bcd26ab5df
open.access
open.access
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="0c34d76d-ee96-4277-9ad5-b97d212838ea">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Language>en_US</Language>
<Title>Learning to Manipulate Novel Objects for Assistive Robots</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2017-05-30</PublicationDate>
<DOI>https://doi.org/10.7298/X43R0R0W</DOI>
<Authors>
<Author>
<DisplayName>Sung, Jaeyong</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<OrgUnit />
</Publisher>
</Publishers>
<Keyword>machine learning</Keyword>
<Keyword>Multimodal Data</Keyword>
<Keyword>Robotic Manipulation</Keyword>
<Keyword>Robot Learning</Keyword>
<Keyword>Artificial intelligence</Keyword>
<Keyword>Deep Learning</Keyword>
<Keyword>Computer science</Keyword>
<Keyword>Robotics</Keyword>
<Abstract>The ability to reason about different modalities of information, for the purpose of physical interaction with objects, is a critical skill for assistive robots. For a robot to be able to assist us in our daily lives, it is not feasible to train each robot for a large number of tasks with all instances of objects that exist in human environments. Robots will have to generalize their skills by jointly reasoning with various sensor modalities such as vision, language and haptic feedback. This is an extremely challenging problem because each modality has intrinsically different statistical properties. Moreover, even with expert knowledge, manually designing joint features between such disparate modalities is difficult.
In this dissertation, we focus on developing learning algorithms for robots that model tasks involving interactions with various objects in unstructured human environments --- especially on novel objects and scenarios that involve sequences of complicated manipulation. To this end, we develop algorithms that learn shared representations of multimodal data and model full sequences of complex motions. We demonstrate our approach on several different applications: understanding human activities in unstructured environment, synthesizing manipulation sequences for under-specified tasks, manipulating novel appliances, and manipulating objects with haptic feedback.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>