Model selection is the task of selecting a "good" model from a set of candidate models given data. In machine learning, it is important that models fit the training data well, however it is more important for a model to generalize to unseen data. Additionally, it is desirable for a model to be as small as possible allowing for deployment in low-resource settings. In this thesis, we focus on three problems where the structure of the problem benefits from reducing the size of the models. In the first problem setting, we explore word embedding models and propose a framework in which we connect popular word embedding methods to low-rank matrix models and suggest new models for computing word vectors. In the second problem setting, we explore sparsity-inducing penalties for deep neural networks in order to obtain highly sparse networks which perform competitively with their over-parametrized counterparts. Finally, we explore the task of robot planar pushing and propose a novel penalty which adapts the parameters of a neural network to unseen examples allowing for robots to better interact in unseen environments. Our results on simulations and real-world data applications indicate that penalization is effective for learning models which perform well in settings with less computational budget, storage, or labeled data.