Systems that support expressive, situated natural language interactions are essential for expanding access to complex computing systems, such as robots and databases, to non-experts. Reasoning and learning in such natural language interactions is a challenging open problem. For example, resolving sentence meaning requires reasoning not only about word meaning, but also about the interaction context, including the history of the interaction and the situated environment. In addition, the sequential dynamics that arise between user and system in and across interactions make learning from static data, i.e., supervised data, both challenging and ineffective. However, these same interaction dynamics result in ample opportunities for learning from implicit and explicit feedback that arises naturally in the interaction. This lays the foundation for systems that continually learn, improve, and adapt their language use through interaction, without additional annotation effort. This thesis will focus on these challenges and opportunities.