2026-09-19T18:14:29Zhttps://ecommons.cornell.edu/server/oai/request

oai:ecommons.cornell.edu:1813/594142026-05-15T19:50:42Zcom_1813_35col_1813_47

Lee, Moontae Mimno, David Frazier, Peter Bindel, David S. 2018-10-23T13:22:40Z 2020-06-04T06:00:32Z 2018-05-30 ProQuest Submission ID: 10890 ProQuest Publication ID: 10823238 https://hdl.handle.net/1813/59414 https://doi.org/10.7298/X4FB514V 10489499 Co-occurrence information is powerful statistics that can model various discrete objects by their joint instances with other objects. Transforming unsupervised problems of learning low-dimensional geometry into provable decompositions of co-occurrence information, spectral inference provides fast algorithms and optimality guarantees for non-linear dimensionality reduction or latent topic analysis. Spectral approaches reduce the dependence on the original training examples and produce substantial gain in efficiency, but at costs: a) The algorithms perform poorly on real data that does not necessarily follow underlying models; b) Users can no longer infer information about individual examples, which is often important for real-world applications; c) Model complexity rapidly grows as the number of objects increases, requiring a careful curation of the vocabulary. The first issue is called model-data mismatch, which is a fundamental problem common in every spectral inference method for latent variable models. As real data never follows any particular computational model, this issue must be ad- dressed for practicality of the spectral inference beyond synthetic settings. For the second issue, users could revisit probabilistic inference to infer information about individual examples, but this brings back all the drawbacks of traditional approaches. One method is recently developed for spectral inference, but it works only on tiny models, quickly losing its performance for the datasets whose underlying structures exhibit realistic correlations. While probabilistic inference also suffers from the third issue, the problem is more serious for spectral inferences because co-occurrence information easily exceeds storable capacity as the size of vocabulary becomes larger. We cast the learning problem in the framework of Joint Stochastic Matrix Factorization (JSMF), showing that existing methods violate the theoretical conditions necessary for a good solution to exist. Proposing novel rectification paradigms for handling the model-data mismatch, the Rectified Anchor Word Algorithm (RAWA) is able to learn quality latent structures and their interactions even on small noisy data. We also propose the Prior Aware Dual Decomposition (PADD) that is capable of considering the learned interactions as well as the learned latent structures to robustly infer example- specific information. Beyond the theoretical guarantees, our experimental results show that RAWA recovers quality low-dimensional geometry on various textual/non-textual datasets comparable to probabilistic Gibbs sampling, and PADD substantially outperforms the recently developed method for learning low-dimensional representations of individual examples. Although this thesis does not address the complexity issue for large vocabulary, we have developed new methods that can drastically compress co-occurrence information and learn only with the compressed statistics without losing much precision. Providing rich capability to operate on millions of objects and billions of examples, we complete all the necessary tools to make spectral inference robust and scalable competitor to probabilistic inference for unsupervised latent structure learning. We hope our research serves an initial basis for a new perspective that combines the benefits of both spectral and probabilistic worlds. en_US Statistics Applied mathematics Anchor Word Algorithm Co-occurrence Modeling Joint-Stochastic Matrix Factorization Rectification Spectral Inference Topic Modeling Artificial intelligence Joint-stochastic Spectral Inference for Robust Co-occurrence Modeling and Latent Topic Analysis dissertation or thesis application/pdf Computer Science Cornell University Doctor of Philosophy Ph. D., Computer Science https://hdl.handle.net/1813/59810 Publication Cornell Theses and Dissertations Lee, Moontae 5893a6ea-7af3-41d7-abc6-04bcd26ab5df open.access open.access <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="0e43dfb5-d421-4fd4-9163-978c125adf9d"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>en_US</Language> <Title>Joint-stochastic Spectral Inference for Robust Co-occurrence Modeling and Latent Topic Analysis</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2018-05-30</PublicationDate> <DOI>https://doi.org/10.7298/X4FB514V</DOI> <Authors> <Author> <DisplayName>Lee, Moontae</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <OrgUnit /> </Publisher> </Publishers> <Keyword>Statistics</Keyword> <Keyword>Applied mathematics</Keyword> <Keyword>Anchor Word Algorithm</Keyword> <Keyword>Co-occurrence Modeling</Keyword> <Keyword>Joint-Stochastic Matrix Factorization</Keyword> <Keyword>Rectification</Keyword> <Keyword>Spectral Inference</Keyword> <Keyword>Topic Modeling</Keyword> <Keyword>Artificial intelligence</Keyword> <Abstract>Co-occurrence information is powerful statistics that can model various discrete objects by their joint instances with other objects. Transforming unsupervised problems of learning low-dimensional geometry into provable decompositions of co-occurrence information, spectral inference provides fast algorithms and optimality guarantees for non-linear dimensionality reduction or latent topic analysis. Spectral approaches reduce the dependence on the original training examples and produce substantial gain in efficiency, but at costs: a) The algorithms perform poorly on real data that does not necessarily follow underlying models; b) Users can no longer infer information about individual examples, which is often important for real-world applications; c) Model complexity rapidly grows as the number of objects increases, requiring a careful curation of the vocabulary. The first issue is called model-data mismatch, which is a fundamental problem common in every spectral inference method for latent variable models. As real data never follows any particular computational model, this issue must be ad- dressed for practicality of the spectral inference beyond synthetic settings. For the second issue, users could revisit probabilistic inference to infer information about individual examples, but this brings back all the drawbacks of traditional approaches. One method is recently developed for spectral inference, but it works only on tiny models, quickly losing its performance for the datasets whose underlying structures exhibit realistic correlations. While probabilistic inference also suffers from the third issue, the problem is more serious for spectral inferences because co-occurrence information easily exceeds storable capacity as the size of vocabulary becomes larger. We cast the learning problem in the framework of Joint Stochastic Matrix Factorization (JSMF), showing that existing methods violate the theoretical conditions necessary for a good solution to exist. Proposing novel rectification paradigms for handling the model-data mismatch, the Rectified Anchor Word Algorithm (RAWA) is able to learn quality latent structures and their interactions even on small noisy data. We also propose the Prior Aware Dual Decomposition (PADD) that is capable of considering the learned interactions as well as the learned latent structures to robustly infer example- specific information. Beyond the theoretical guarantees, our experimental results show that RAWA recovers quality low-dimensional geometry on various textual/non-textual datasets comparable to probabilistic Gibbs sampling, and PADD substantially outperforms the recently developed method for learning low-dimensional representations of individual examples. Although this thesis does not address the complexity issue for large vocabulary, we have developed new methods that can drastically compress co-occurrence information and learn only with the compressed statistics without losing much precision. Providing rich capability to operate on millions of objects and billions of examples, we complete all the necessary tools to make spectral inference robust and scalable competitor to probabilistic inference for unsupervised latent structure learning. We hope our research serves an initial basis for a new perspective that combines the benefits of both spectral and probabilistic worlds.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>