Recent advances in artificial intelligence (AI) have expanded the range of possibilities for how human users can make use of computers and autonomous systems, shifting the narrative from collaboration through technology to collaboration with technology. As AI holds the potential to be our new teammates, should they carry on the social experiences that often serve as double-edged swords in human-human teamwork? The present dissertation examines this core inquiry through the following approaches. I begin by reviewing the literature on social cognition and psychology to discuss how social experience arises at the individual level and how it can impact small groups and teamwork (Chapter 2). I then discuss how different AI technologies -- in particular, intelligent agents -- can simulate social actors and elicit social experiences during interaction with human users (Chapter 3). Based on the existing literature, I examine how social experience in human-agent teamwork can be triggered through either a top-down or a bottom-up approach. I begin with studying the influence of social experience when an agent supports individual work in dyadic settings. Specifically, I empirically test how an autonomous teamwork agent's informed identity (Chapter 4) and perceived capability (Chapter 5) can influence users' experiences and behaviors and whether such interventions would facilitate or inhibit individuals' performance. In Chapter 6, I extend to investigate the social experience of human-agent collaboration in multi-player teams. Specifically, I examine whether and how an autonomous agent could support marginalized members in such settings. In summary, while a teamwork agent could bring on benefits commonly introduced by having human teammates, further anthropomorphism and socialization do not lead to significantly more positive teamwork outcomes. Moreover, for individuals who have previously struggled in group settings, such as those who have experienced anxiousness or marginalization, emphasizing social experiences in human-agent teamwork can hinder their performance. Together, the present dissertation takes a principled approach to the research topic and offers theoretical and practical implications that could pave new avenues for future research.