Modern complex stochastic systems face increasing challenges in efficient resource management. This dissertation investigates scalable control in stochastic systems by developing both theoretical foundations and practical data-driven methodologies. The first part of the dissertation presents an analytical solution to this problem from a queueing-theoretic perspective. Specifically, we study the steady-state behavior of multiclass queueing networks with general interarrival and service time distributions, a broad class of models for modern operational systems. We establish a product-form limit in a multi-scale heavy traffic regime, providing the first closed-form performance approximation for general multiclass networks. This product-form limit reveals an approximate independence among service stations, enabling quick performance approximation without time-consuming simulation and subsequently facilitating policy optimization in heavily loaded environments. As networks scale in size and complexity, deriving analytical solutions, as done in the first part, can become increasingly challenging. To address this, the second part of the dissertation explores reinforcement learning (RL) as a data-driven alternative for queueing network control. Using input-queued switch scheduling as a case study, we investigate the effectiveness of RL in learning effective scheduling policies in the face of a combinatorial action space. To better address the large action space, we incorporate an action decomposition technique known as "atomic action" decomposition, which builds a complete matching sequentially via taking "atomic actions" on a reduced action space. Our results show that RL, when combined with atomic action decomposition, can learn efficient policies, demonstrating the potential of RL for scalable control in complex queueing systems. Despite the wide empirical success of RL, its theoretical foundations remain underexplored. To better understand the underlying mechanisms and theoretical properties, the last part of the dissertation studies the theory of constant-stepsize stochastic approximation (SA), a core algorithm underpinning many RL algorithms. Viewing SA as a stochastic system, we analyze its asymptotic and non-asymptotic behavior under Markovian data, which commonly arise in queueing and RL problems, and characterize its asymptotic limiting distribution. Building on the theoretical analysis, we explore the algorithmic implications of our results, such as bias and variance reduction for SA iterates, to improve the data efficiency and enable more effective learning in RL. Together, these contributions present a holistic framework for scalable control in stochastic systems, bridging classical queueing theory with modern RL.