Stochastic optimal control theory encompasses various types of stochasticity and notions of optimality. The standard risk-neutral approach minimizes or maximizes an expected total cost, but this approach often yields non-robust results. In this thesis, we introduce a particular type of robust control framework of indefinite-horizon processes, maximizing the probability of desired outcomes while keeping the cumulative cost within a threshold. For diffusive processes, our framework results in second-order parabolic Hamilton-Jacobi-Bellman (HJB) Partial Differential Equations (PDEs).We develop an efficient algorithm to solve these equations by leveraging the inherent causality of the framework. This allows us to recover the optimal "threshold (risk)-aware" feedback policies for all initial configurations and a range of threshold values simultaneously in a single sweep. We first apply this methodology to adaptive cancer therapy under stochastic cancer dynamics. In particular, we aim to maximize the probability of achieving treatment goals while keeping the total treatment cost within a specific cost threshold/budget. We then extend this threshold-aware approach to hybrid control problems, specifically through sailboat routing under stochastically evolving wind conditions. This application involves solving a pair of quasi-variational inequalities in a Hamilton-Jacobi framework. Monte Carlo simulations are used to generate cumulative distribution functions (CDFs), demonstrating the advantages of threshold-aware policies over risk-neutral ones. In the final section, we investigate bacterial competition influenced by environmental extreme events (dilutions). We propose an explanation for why toxin-sensitive bacteria, usually outcompeted by toxin-producers in vitro, can thrive under frequent dilutions. We consider both deterministic periodic dilutions and randomly timed dilutions modeled by a Poisson process. Through a series of optimized toxin-regulation behaviors for toxin-producers, we demonstrate that toxin-sensitive strains still have a reasonable chance of winning. The numerical approach involves solving Hamilton-Jacobi-type equations (including a specific type of non-local coupling emerging from the jump-discontinuities induced by the Poisson process) using semi-Lagrangian schemes.