One overarching theme of the past decade of machine learning is that scaling any of the many axes of modeling generally improves model quality. By increasing data, compute, or the number of parameters, we can usually expect improvements in downstream performance. To the casual observer, it may seem that large, unstructured models are the key to deep learning. However, careful analysis shows that modern models are actually rich in structure. In this thesis, I show that it is possible to exploit these structures to improve the efficiency of deep models. In Part 2, I demonstrate that by using the geometric properties of Hyperbolic space to model hierarchies in attention mechanisms, we can significantly improve modeling performance on hierarchical data. In Part 3, I consider the problem of LLM quantization. I first show that by taking advantage of both the geometric structure of the model weights as well as the model loss landscape, we can compress models to as little as 1/8 of their original size without catastrophic loss of fidelity. Then, I apply these concepts to low-precision training, where I introduce the first near-lossless FP4 LLM training recipe, accelerating training by over 2x. Finally, in Part 4, I explore explicitly embedding structure into models with sparse embedding tables, resulting in more performant and faster structured sparse models. Many of these findings have been adopted in industry, confirming that even in the age of scaling, structural information is crucial for efficient learning.