Computing with analog physical systems, including optical ones, has experienced a reawakening in recent years, driven by the rapid expansion of artificial intelligence. As the size of deep neural networks continues to grow, there is an increasing demand for more efficient computation methods. Neural-network computations inherently tolerate lower precision in individual steps, particularly when optimized for certain tasks, making low-precision analog computing a viable alternative to universal digital electronic computers. Unlike well-calibrated digital electronic devices, analog physical computing may offer more uncertainty, especially when operated at low power, but leverages inherent physical processes to achieve more efficient computation. In this thesis, I summarize some of my Ph.D. work that explored energy-efficient computation through optical systems. Initially, we constructed a large-scale optical setup capable of performing linear operations with large vector sizes. This setup features substantial spatial parallelism, leveraging the optical energy advantage in computation and making many optical computing tasks experimentally feasible. Using this setup, we demonstrated an ultra-low-optical-energy implementation of optical neural networks, previously only predicted in theory. We also examined the energy scaling advantages for state-of-the-art large deep learning models, ensuring an asymptotic overall energy consumption advantage for very large models. We then ventured beyond the scope of conventional optical computing. We investigated technologies to generate, control, and detect highly multimode quantum states, preparing to exploit their extensive quantum features for computational and sensing innovations. We also explored integrating inherent stochastic physical processes into neural network modeling, achieving significant performance gains with minimal physical resources. Additionally, we studied how this method can benefit image sensing problems under very restricted detection energy. These explorations further advance the potential for efficient computation utilizing low-power stochastic physical systems.