As interactive systems evolve from smartphones to mixed reality and beyond, they open the door for more seamless human-robot interaction and collaboration. To enable such systems, we need fast and energy-efficient hardware architectures. In this dissertation I present a full-system analysis and modeling of intelligent interactive systems and propose novel architectures that dramatically improve performance and energy-efficiency. I present a methodology to design energy-efficient architectures for real-time human-computer interactions featuring the first simulation-based platform for interactive environments that runs user interface code and recorded user interaction traces. The platform is based on Gem5 ARM simulator and McPAT power models, as well as several idle time power management models. This platform takes input traces from real users that are captured on the handheld hardware, processes them using the interactive application, and measures the response latency and energy. Through detailed analysis of user traces running on various architectures in this simulation environment, I show experimental results for event processor designs that are substantially more energy efficient than general purpose heterogeneous architectures. For efficient support of artificial intelligence workloads, I argue that to address the shortcomings of ASIC accelerators for deep neural networks, the hardware must support some degree of flexibility. To quantitatively evaluate this idea, I analyze the relationship between data types and accuracy by studying multiplier and accumulator designs of various bitwidths across inference and training. Given the appropriate design choices, I propose that the hardware switch between a set of configurations that are more optimal on a per-benchmark basis with low overhead. I design a reconfigurable arithmetic microarchitecture with an efficient memory interface that also enables the accelerator to support training with minimal overheads. I then analyze the microarchitecture of the reconfigurable design in the context of an existing state-of-the-art accelerator chip and present a full system evaluation of the proposed architectural modifications. I demonstrate that the proposed reconfigurable architecture achieves significant improvements in both energy and throughput.