This dissertation focuses on developing new machine learning models and algorithms for the task of learning from data that originates from human-system interactions i.e., the interactive learning paradigm. A wide array of modern technologies involve significant interaction between the humans users and the system. These technologies - which range from everyday applications such as search engines and retail services, to more disruptive ones such as self-driving cars and smart homes - can greatly benefit from the world knowledge implicit in these human interactions. However, as a consequence of interactive learning data being derived from observed human behavior, standard machine learning models are a poor fit. This thesis develops a fundamentally new approach to interactive learning. The guiding principle in this dissertation is to jointly design the three key components of interactive learning: the learning algorithm, the user behavioral model and the feedback interventions. The learning algorithms developed in this thesis strive to learn from preference data in a robust manner. Furthermore, they come with theoretical performance guarantees and are shown to work well in practice. For sound learning from human interaction data, we need plausible models of user behavior while interacting with these systems. The approaches discussed here explicitly account for the different factors that impact the user decisions, such as their motivations, expertise, skills, needs and decision context. A unique advantage interactive learning systems possess is the ability to intervene and alter the content presented to users so as to maximize learning. This dissertation covers different examples that illustrate that even small changes can greatly improve learning in these systems. The potency of this joint design methodology is illustrated using different interactive learning examples including: (a) a scholarly text search engine for arxiv.org, that autonomously, robustly, and cost-effectively improve its performance; (b) web search and recommender systems that can model and facilitate complex user tasks; and (c) peergrading.org, a peer-grading service which collates grades from all the students in a principled manner.