Autonomous driving has evolved significantly from conceptual challenges to commercial deployment in robo-taxis and consumer vehicles. Despite these advances, self-driving systems face fundamental limitations when confronting real-world diversity and distribution shifts not represented in training data. This thesis addresses a critical question: How can we generalize autonomous driving perception systems to diverse scenarios in a principled and cost-efficient manner without requiring infeasibly large labeled datasets? My research improves autonomous vehicle perception by leveraging previously underutilized information sources. I present three main contributions: (1) a novel approach for object discovery from repeated traversals, (2) a method incorporating human priors to accelerate discovery processes, and (3) techniques to enhance road understanding by leveraging standard definition maps. These approaches collectively demonstrate that auxiliary information channels can effectively adapt perception systems to real-world scenarios with greater data efficiency. The work contributes to making autonomous vehicles safer and more adaptable across the diverse challenges of real-world driving environments, particularly addressing the challenging long-tail of rare driving situations.