Over the past several years, we have witnessed fundamental breakthroughs in machine learning, largely driven by rapid advancements of the underlying deep neural network models and algorithms. This has consequently spurred the development of new, powerful machine learning systems, with emerging applications in specialized, high-stakes domains such as medicine. However, the increased capabilities of these novel systems come at a cost of much greater complexity, with the design of machine learning systems becoming ever more laborious, computationally expensive and opaque. This can result in catastrophic failures and significantly hinders effective collaboration with human experts, often central to successful deployment. In this thesis, we present research results that take steps to addressing these challenges. Having first overviewed some of the key deep learning models, algorithms and use cases, we begin by introducing quantitative techniques that can give insights into neural network hidden representations, which provide both fundamental understanding on central aspects of machine learning, and also inform algorithms for efficiently learning and training these complex systems. Finally, we study how these fully trained AI systems can be adapted to work effectively with human experts, resulting in better outcomes than either humans or AI alone. We conclude with a discussion on the many rich further directions and open questions for future study.