Natural language systems grounded in environmental contexts, such as visual perceptions, offer versatile interactions with human users suitable for a wide array of applications. For instance, these systems can be used for robots to collaboratively interact with human users in a visual environment. However, the internal reasoning processes of these systems remain opaque, making it challenging to understand their learning patterns. In this thesis, we introduce a framework to explore the reasoning abilities of such grounded language systems. Moreover, while grounded language systems open doors to expansive user interactions due to their versatility, there is a need for methodologies to continually improve these systems based on user interactions. To this end, we explore learning frameworks to improve grounded language systems by leveraging implicit user feedback.