In the past decade, we have witnessed rapid progress in machine learning with the help of neural network models. Recently, neural models have achieved superhuman performance on various tasks such as image recognition, graph classification, and reading comprehension. With tremendous computation, researchers and engineers are able to train unprecedentedly large models and achieve superhuman performance on several benchmarks. While the performance (or accuracy) of neural models receives a significant amount of attention, there is an increasing concern about the efficiency and the environmental impact of these models since training a state-of-the-art model can be both costly and time-consuming. Moreover, while being accurate in offline settings, state-of-the-art models are seldom used in real-life applications due to their high latency and cost. To improve the performance of neural networks, researchers introduce more complex operations and models, which allow them to capture more complicated functions. While achieving higher accuracy, the reasons behind the success of several modern neural components, including graph convolution and self-attention remain concealed. Does the improvement come from additional parameters and model capacity or the induction bias? Do they utilize additional information? Besides new operations, stacking layers to increase the depth of a neural network is a common practice to enlarge the model capacity and improve the performance. While examples near the decision boundaries are hard to be classified, a shallow network with little computation can already deal with a large number of tasks. How can we dynamically allocate different computing resources to different examples at inference time? We will address these questions. This dissertation aims to expose unnecessary computation in modern neural networks and provide insights for designing efficient neural architectures. In chapters 3 and 4, we analyze two popular neural models, the graph convolution networks (GCNs) and the Transformers with self-attention mechanism, and propose simpler yet more efficient alternatives which achieve better or comparable performance. We point out that the non-linearity is unnecessary in GCNs and their success comes from feature propagation among local neighbors. We analyze the success of the Transformer models and demonstrate that convolution layers with particular emphasis on the local features can be very competitive as well. In chapters 5 and 6, we introduce novel neural architectures tailored to conditional inference strategies. On image recognition and machine reading comprehension, models with conditional inference strategies can achieve several times speedup through saving unnecessary computation on easy examples.