Artificial intelligence has advanced rapidly in recent years, driven largely by the success of neural networks trained at unprecedented scale.A defining feature of these models is their scalability: as model size, data, and compute increase, performance often improves in a predictable manner, enabling the solution of increasingly complex problems. Despite this success, fundamental questions remain. Neural networks are typically highly overparameterized and trained on noisy, finite datasets, yet they often generalize well without explicit regularization. Understanding why this is possible remains a central challenge in modern machine learning. Two closely related concepts play a key role in addressing this challenge: implicit regularization and inductive bias. Implicit regularization refers to the tendency of the training process itself --- through optimization algorithms, parameterization, and initialization --- to favor certain solutions over others, even in the absence of explicit regularization terms. Inductive biases describe the structural preferences encoded by a model’s architecture and design choices, which guide learning toward solutions that generalize beyond the training data. This dissertation focuses on understanding these two aspects of neural network training.We study the implicit regularization and inductive biases of several widely used model classes, including multi-layer perceptrons, state-space models, and Transformer architectures. Across these settings, we analyze how frequency structure, rank structure, and long-range dependencies emerge during training, and how they are shaped by architectural choices and parameterization. Importantly, we go beyond characterization and show how these effects can be tuned in a principled way through data design, model construction, and training procedures. By clarifying how implicit regularization and inductive biases arise and evolve, this work provides a unified perspective on the generalization behavior of modern neural networks and offers practical guidance for designing scalable models that are better aligned with the structure of the tasks they are intended to solve.