This dissertation develops a more computationally efficient Approximate Dynamic Programming (ADP) method that can be applied to optimal control problems that are stochastic, nonconvex, and either finite or rolling horizon with many stages. The primary application is to a realistic model of Bonneville Power Administration, which is a large wind power and hydropower producer in the Pacific Northwest. The objective is to determine the optimal amount of power to buy and sell on the day and hour ahead markets conditioned on the wind power forecast, as well as how much water to release from each reservoir. The second Chapter in this dissertation develops the Fitting via Unimodal Approximation Optimization (FUA) method for more accurately approximating the value function in ADP with a Feedforward Neural Network with one hidden layer (FFNN1). A major part of FUA is the new Unimodal Approximation Optimization (UAO) algorithm that is used to perform FFNN1 hyperparameter optimization. UAO can be applied to optimization problems with a discrete domain and a noisy unimodal objective function, and it is proven that UAO converges almost surely to the correct solution. Results on two control problems with 4, 12 and 15 state space dimensions show that approximating the value function in ADP with an FFNN1 using FUA yields a more accurate control solution in less time as compared to using other methods of fitting an FFNN1. Chapter 3 presents the Long Term Generation method that generates long-term synthetic wind power scenarios conditioned on historical sequential short-term wind power forecasts. Power systems with wind power integration can be simulated on this data in order to more accurately evaluate the performance of their control policies. Additionally, the Joint Distribution Comparison test is developed to evaluate the quality of these synthetic scenarios. Finally, in Chapter 4 a stochastic rolling horizon model of Bonneville Power Administration is developed and the resulting control problem is solved using the ADP algorithm developed in Chapter 2 and is evaluated using data generated in Chapter 3. The stochastic control formulation has a nonlinear objective function, 24 decision variables, and 16 state space dimensions.