Understanding people in complex dynamic environments is important for many applications such as robotic assistants, health-care monitoring systems, self driving cars, etc. This is a challenging problem as human actions and intents are not always observable and often contain large amounts of ambiguity. Moreover, human environments are complex with lots of objects and many possible ways of interacting with them. This leads to a huge variation in the way people perform various tasks. The focus of this dissertation is to develop learning algorithms for understanding people and their environments from RGB-D data. We address the problems of labeling environments, detecting past activities and anticipating what will happen in the future. In order to enable agents operating in human environments to perform holistic reasoning, we need to jointly model the humans, objects and environments and capture the rich context between them. We propose graphical models that naturally capture the rich spatio-temporal relations between human poses and objects in a 3D scene. We propose an efficient method to sample multiple possible graph structures and reason about the many alternate future possibilities. Our models also provide a functional representation of the environments, allowing agents to reactively plan their own actions to assist in the activities. We applied these algorithms successfully on our robot for performing various assistive tasks ranging from finding objects in large cluttered rooms to working alongside humans in collaborative tasks.