We formalize agents' knowledge of counterfactuals in two different settings, players' behavior in extensive-form games and the process of agents' conditioning their beliefs. For extensive-form games, we define the notion of subgame-rationality, where players best-respond to what they believe would happen at any subgame where they are due to play. This approach settles a well-known disagreement in the literature between Aumann and Stalnaker - supporting Aumann - regarding whether common knowledge of rationality leads to the backwards induction solution in perfect information games. Subgame-rationality also makes it easier to relate epistemic characterizations of Nash equilibrium to those of subgame-perfect equilibrium. We also turn our attention to adding counterfactuals to agents' language, which leads to definitions of rationality which use iterated counterfactuals. For conditional beliefs, we propose new public-announcement style semantics which factor out the act of conditioning, using two traditional modalities, beliefs and counterfactuals. We investigate the set of validities for these semantics. We also take a closer look at the relationship between traditional plausibility models for conditional beliefs within Dynamic Epistemic Logic and models which use, instead, explicit counterfactual shifts. We identify properties that counterfactual shifts need to satisfy in order to simulate plausibility models.