Over the last decade, researchers have made significant progress toward training and understanding larger and more powerful deep neural networks. This thesis contains several contributions toward this effort. The first chapter gives a brief introduction. The next two chapters describe applications and extensions of existing techniques to specific problems. First we consider the task of creating gaits for legged robots and describe the performance of various learning algorithms for automating generation of quadruped gaits. Notably, labels employed for learning are gather entirely for real-world experiments run in the learning loop. This results in a challenging learning scenario with limited labels, but through the use of models with the appropriate assumptions, gaits are found that are nine times faster than those designed by hand. In the next chapter, we discuss the task of enabling easier design of three dimensional (3D) shapes via machine learning. We describe the front end presentation and back end algorithms that enabled non-expert users to generate millions of shapes online. The remaining chapters describe approaches to understand network training and behavior. The first details several visualizations useful for training Restricted Boltzmann Machines (RBMs) and an application to a synthetic 3D data set. The second investigates a basic property of trained neural networks: to what extent parameters learned on one task can be transferred to another task. The third discusses the Deep Visualization Toolbox, open source software written to aid in understanding individual neurons in the middle of a large neural network. An additional chapter is supplied in the appendix as the author of this thesis and another researcher contributed equally to it. In this final section, the local vs. distributed nature of neural network representations is studied experimentally.