This dissertation investigates stochastic optimization methods where exact gradient computations are prohibitively expensive, necessitating potentially biased gradient estimators. Such challenges arise in applications like empirical risk minimization in machine learning, where unbiased gradient estimates are not always feasible, and in derivative-free settings where only function values are accessible. We focus on stochastic gradient methods with biased estimates from shuffling-based sampling schemes, which lack standard assumptions of independence and unbiasedness. While unbiased methods like i.i.d. SGD are well-studied, the analytical challenges of biased shuffling methods, particularly when combined with momentum, are underexplored. This work addresses these gaps by examining the theoretical and practical impacts of shuffling schemes in stochastic first-order methods, including stochastic gradient descent and its variant with heavy ball momentum and Nesterov's acceleration. Additionally, we analyze a backtracking variant of FISTA and extend a stochastic analysis framework from prior work to both ISTA and backtracking FISTA, showing that, without requiring unbiased gradient estimators, these methods retain their deterministic complexity.