Despite the proliferation of machine learning (ML) in a multitude of applications, current black-box models, such as deep learning, remain hard to understand, critique, and judge by decision makers. This in turn limits their adoption in high-stakes environments (e.g., credit lending, college admissions, medicine) where ML is often used as a tool to support a human decision maker. Moreover in applications where decisions have a significant societal impact, practitioners need ML models that can guarantee that their output is fair to sensitive demographic groups, a challenging constraint to integrate into existing algorithms. Integer Programming (IP) is a natural tool for these problems as many simple interpretable ML models can be represented by low-complexity discrete objects, and it allows for the flexible incorporation of domain-specific constraints such as fairness criteria. However, despite its success in numerous industrial applications, such as scheduling and logistics, exact IP methods are considered to be too computationally demanding to be used in many ML applications and are eschewed for fast heuristics. This thesis endeavors to bridge this gap between exact optimization and fast heuristics by leveraging large-scale integer programming techniques to build scalable algorithms that can outperform existing ML heuristics in a fraction of the time of exact IP-based methods. In particular, this thesis develops novel formulations for ML problems built upon strong combinatorial structure that can flexibly incorporate domain-specific constraints such as fairness. Underpinning these formulations are large-scale optimization procedures that are informed by exact optimization methods but leverage heuristics tailored for ML settings that allow them to scale to large data sets. Finally, towards democratizing these IP-based machine learning tools, this thesis explores how to leverage Large Language Models to enable non-expert users to interact and customize mathematical optimization models.