We do not all have equal voices in the design decisions that affect us. Often, the institutions that make these decisions are not committed to designing for members of marginalized groups who may particularly benefit from or be harmed by the outcomes of these decisions. Although voice assistants are a potentially life-altering technology for some, they currently exist principally as luxury goods. The needs and values of marginalized groups for whom voice assistants may be more impactful have received relatively little attention. Among these underrepresented voices in the design process of voice assistants are older adults (age 65 and older), who are at higher risks of experiencing cognitive, functional, visual, hearing, emotional, social, and other impairments, who are often not experienced technology users, and who may depend on caregivers. This dissertation adapts existing concepts from inclusive design to the interaction design of voice assistants, contributing four studies that advocate for improving their inclusivity (and safety). Two of these are empirical studies that explore the use of voice assistants by older adults from two different angles: 1) an approximately two-month long intervention studying private-setting interactions with users who become experienced over the duration of the study, and 2) a close look (second by second) at public-setting interactions with mostly novice users. Together, these two studies examine communication gaps that may lead to exclusion and inform design strategies to increase the inclusivity of voice assistants for older adults (e.g., by integrating into healthcare systems, and by reacting to audio-prosodic and visual cues). The third study explores building voice assistants that perform self-repair as a way to address communication breakdowns such as those seen in the first two studies. We demonstrate that voice assistants that perform self-repair in response to visual cues improve interactions, even after factoring in instances in which these visual cues are not appropriately interpreted. Finally, the fourth study grapples with ethical concerns by developing a framework to reflect on the role of empathy in these interactions and mitigate their potential risk of harm. My studies were carefully designed to complement each other---together they show the power and perils of voice assistants. As a whole, I (i) examine how older adults currently interact with voice assistants; (ii) evaluate alternative designs that could accelerate the development of inclusive voice assistant features; and (iii) develop a framework to mitigate potential harms of empathetic conversational user interfaces. Taken together, this work aspires to relieve some of the burdens placed on underrepresented individuals to adapt to the impositions of digital technologies, serves to promote the inclusion of marginalized groups in mainstream technological activities that can increase justice and equity, and warns us about potential risk of harm.