Humans are exceptional few- and zero-shot learners that use internal representations of the world to reason under uncertainty. These internal world models can encode beliefs about cause-effect relationships, supporting causal reasoning in novel and uncertain environments. Probability theory and causal inference provide principled frameworks that further aid humanity in reasoning about causal mechanisms with limited prior knowledge. As causal reasoning is a hallmark of human cognition, it is also a core desideratum for human-like AI. However, the design of AI systems with robust causal reasoning abilities remains an open problem. This dissertation explores challenges in reasoning under uncertainty from the perspectives of human and AI reasoners. It covers two main directions, both grounded in probabilistic and causal graphical modeling: 1. Methods for supporting expert decision-making and causal inference. This component presents efficient statistical algorithms to aid human experts in reasoning over causal systems with limited prior knowledge. These methods contribute to the literature on structure learning for causal effect estimation and fairness analysis. 2. Methods for evaluating causal reasoning in AI systems. This component presents theoretical and empirical frameworks for the principled evaluation of causal reasoning in deep generative models. These works focus on (i) compositional consistency and commutative reasoning over causal effects and (ii) abstract, logical, and counterfactual reasoning over causal world models, using program synthesis and test-time training.