Large-scale continuous optimization arises in many practical problems such as machine learning, signal processing, and imaging. It is usually challenging to analyze the theoretical properties of optimization algorithms and design scalable algorithms that work well in practice. This dissertation provides new perspectives on continuous optimization in both theory and methodology. From the theoretical side, we present a framework for reasoning about equivalence between a broad class of iterative convex continuous optimization algorithms. The notion of algorithm equivalence can make it easier to understand the connections between optimization algorithms, relate many analytical properties of interest such as convergence or robustness, and further give insights to design new algorithms. From the methodological side, we first present NysADMM, a scalable algorithm for faster composite convex optimization by exploiting the low-rank structure. NysADMM accelerates the inexact linearized Alternating Direction Method of Multipliers (ADMM) by constructing a preconditioner for the ADMM subproblem from a randomized low-rank Nystr{"o}m approximation. Second, we present GeNI-ADMM, a framework generalized from NysADMM, which encompasses any ADMM algorithm that solves a first- or (generalized) second-order approximation to the ADMM subproblem and allows inexact subproblem solves. It facilitates the theoretical analysis of various approximate ADMM schemes for large-scale composite convex optimization.