This thesis develops stochastic optimization frameworks for training and coordinating Large Language Models (LLMs) in distributed settings. We study two interconnected problems: (i) controlling the training dynamics of LLMs across distributed datasets, and (ii) modeling and controlling emergent behaviors when multiple LLM agents interact over networks. Part I formulates distributed LLM training as a controlled stochastic optimization problem. We model the training process as a Markov decision process (MDP) where a learner optimally schedules queries to a Markovian oracle providing noisy gradient evaluations. For covert optimization—where a learner must hide the optimal parameters from an eavesdropper—we prove the optimal policy exhibits a monotone threshold structure exploiting supermodularity of the dynamic programming operator. We extend this framework to multi-learner settings via constrained switching control games and develop policy gradient algorithms that provably converge to optimal threshold policies. Applications to privacy-preserving federated learning for hate speech classification demonstrate that our methods reduce eavesdropper accuracy by over 30% compared to greedy baselines. Part II develops analytical frameworks for networks of interacting LLM agents performing distributed inference. We model individual LLM agents as rationally inattentive Bayesian utility maximizers, providing interpretable microeconomic characterizations of their decision-making. For sequential Bayesian social learning, we characterize conditions under which LLM agents form information cascades (herds) and derive optimal stopping policies to delay herding and improve collective accuracy. For large-scale networks, we develop mean-field approximations that accurately predict population dynamics and enable principled network design via preferential attachment mechanisms. Empirical validation on networks of up to 100 LLMs confirms the predictive power of our theoretical framework. Part III presents additional independent contributions in online learning and game theory: (i) sublinear regret algorithms for blocked sparse linear bandits with applications to personalized recommendation and adaptive annotation; (ii) bandit algorithms for multimodal reward distributions with mixture-of-Gaussians structure; (iii) learning dynamics in asymmetric network games using $\alpha$-potential functions that guarantee convergence to approximate Nash equilibria; and (iv) mixture-of-tokens generation methods for reinforcement learning with verifiable feedback in LLMs.