Recent advances in deep learning have provided fruitful applications for natural language processing (NLP) tasks. One key advance was the invention of word vectors, representing every word in a dense, low-dimensional vector space. Even though word vectors provide very strong results for word level NLP tasks, producing appropriate representation for phrases and sentences is still an open research problem. In this dissertation, we focus on compositional approaches to representation learning. In particular, we employ the notions of compositionality in which the sequence or structure information is utilized, via recurrent or recursive neural networks. We investigate the effectiveness of such approaches for specific natural language understanding tasks including opinion mining and sentiment analysis, and extend some of the approaches to provide better representation hierarchies. In particular, we propose two novel variants: bidirectional recursive neural networks, which are capable of producing context-dependent structural representations and deep recursive neural networks, which provide representation hierarchies in the structural setting. Additionally, we qualitatively investigate such models, and describe how they relate to alternative compositional approaches. Finally, we discuss challenges in interpretation and understanding of compositional neural models, propose simple tools for visualization, and perform exploratory analyses over features learned by such a model.