In this work we explore four questions around the foundations of cooperative artificial intelligence. What types of beliefs are necessary to enable joint behavior? How can we determine the intent an agent has in doing an action? How ought we ascribe blame in scenarios where multiple agents could cooperate to avoid a negative outcome? How can we combine causal information from multiple agents who each have different areas of expertise? We hope that some of the work presented in this dissertation can help kick-start fruitful conversations and work towards developing effective cooperative AI.