In this thesis, we study approaches to optimal control in situations where there is transitional uncertainty (i.e., uncertainty in the system's next state after initiating a transition) or temporal uncertainty (i.e., uncertainty in the length of the planning horizon). We begin by considering discrete optimal control problems on graphs where the transitional uncertainty is described by a probability distribution over possible successor nodes. We introduce a broad subclass of shortest path problems (SSPs) called opportunistically stochastic shortest path problems (OSSPs). In OSSPs, the availability of actions specifying probabilistic transitions to successor nodes implies the availability of actions resulting in deterministic transitions to each of those nodes. We prove a set of mathematical conditions on the OSSP's transition cost function which allows the problem to be treated with fast algorithms originally designed for deterministic cheapest path problems on graphs (e.g., Dijkstra's or Dial's method). We also illustrate the OSSP's connection to numerical discretizations of partial differential equations (PDEs) encoding continuous optimal control problems and usefulness in determining nuanced driving directions for autonomous vehicles on road networks amidst lane change uncertainty. Next, we tackle two continuous optimal control problems in which the length of the planning horizon T is a random variable. In the first problem, we propose an approach to determine a driver's optimal braking / acceleration strategy through a signalized intersection where the light's turning yellow time T follows a discrete probability distribution. The driver seeks to optimize multiple competing objectives (e.g., fuel use, time to destination, comfort) and we show a selection of numerical experiments which illustrate how the driver handles these tradeoffs, and how to adjust the driving strategy when new information is learned about the true value of T. In the second problem, we develop a framework to determine when an energy-optimizing pursuer should use a movement concealment tactic called "motion camouflage" (MC) to appear less-threatening to an evader amidst uncertainty in the prey's escape attempt time, T. Here, T is a non-homogeneous exponentially distributed random variable governed by a rate function, lambda that is dependent on the pursuer's state and the evader's position. We motivate this approach through the biological model problem of hover fly pursuit-evasion interactions, and our simulations reveal that there is a specific parameter regime for lambda where MC tactics are worthwhile.