Cornell University
Library
Cornell UniversityLibrary

eCommons

Help
Log In(current)
DigitalCollections@ILR
ILR School
  1. Home
  2. ILR School
  3. Centers, Institutes, Programs
  4. Labor Dynamics Institute
  5. NSF Census Research Network
  6. Presentations of the NCRN Coordinating Office
  7. Why infinite exchangeable mixture models fail for “sparse” data sets yet microclustering succeeds

Why infinite exchangeable mixture models fail for “sparse” data sets yet microclustering succeeds

File(s)
How Will Statistical Agencies Operate When All Data Are Private-.pdf (1.49 MB)
Permanent Link(s)
https://hdl.handle.net/1813/50060
Collections
Presentations of the NCRN Coordinating Office
Author
Steorts, Rebecca C.
Abstract

Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points. Finite mixture models, Dirichlet process mixture models, and Pitman--Yor process mixture models make this assumption, as do all other infinitely exchangeable clustering models. However, for some tasks, this assumption is undesirable. For example, when performing entity resolution, the size of each cluster is often unrelated to the size of the data set. Consequently, each cluster contains a negligible fraction of the total number of data points. Such tasks therefore require models that yield clusters whose sizes grow sublinearly with the size of the data set. We address this requirement by defining the microclustering property and introducing a new model that exhibits this property. We compare this model to several commonly used clustering models by checking model fit using real and simulated data sets.

Description
joint work with Jeff Miller, Brenda Betancourt, Abbas Zaidi,
and Hanna Wallach, Giacomo Zanella
Sponsorship
Thank you to the John Templeton Foundation (Metaknowlege
Network) and to NSF SES 1534412 for support of this research.
Disclaimer: This work is the view point of the researchers alone
and not the funding agencies/foundations.
Date Issued
2016-04-06
Type
presentation

Site Statistics | Help

About eCommons | Policies | Terms of use | Contact Us

copyright © 2002-2026 Cornell University Library | Privacy | Web Accessibility Assistance