Fully automatic processing of images is a key challenge for the 21st century. Our processing needs lie beyond just organizing photos by date and location. We need image analysis tools that can reason about photos like a human. For example, we need algorithms that can identify the people, activities, materials, illumination and 3D shapes depicted in photos. These scene understanding tasks are the key building blocks of many practical applications: automatic album creation & video summarization, intelligent sharing & privacy policies for social networks, automatic image captioning, visual question & answer, visual search & recommendation systems, acquisition of physical properties (e.g., reflectance and 3D shape), augmented reality object insertion, artistic stylization, smart photo manipulation, robot task planning and self-driving cars. The goal of scene understanding is to infer a structured model of reality from a photo. This cannot be done perfectly because there can be many realities which produce the same image. Humans excel at using prior experience to guess the reality which underlies an image. A new approach to scene understanding has grown around the idea that computers can mimic human-level scene understanding by fitting an artificial neural network to labeled image data. This approach is called deep learning and it is dramatically more successful than previous approaches. Deep learning can be summarized in three key steps: (1) acquire millions of labeled images; (2) create a multi-layer convolutional neural network (aka deep CNN) model parameterized by millions of variables; and (3) adjust the parameters so that the CNN predicts the same as a human. In this thesis we explore the three steps of deep learning through the lens of recognizing materials in a real-world scene and making structured changes to an image: we describe a practical method for efficiently gathering crowdsourced labels; a method for assigning a material label to every pixel of an image; a method for improving the accuracy of crowdsourced labels; and surprising new applications of CNNs for photorealistic image manipulation.