For the past decades, we have observed tremendous success in developing perception models — computational models that could perceive our world through images, videos, LiDAR point clouds, and so on. Currently, we have perception models that can recognize thousands of concepts commonly seen on the Internet. The ability of these models to recognize concepts is undeniably impressive, but their successes are only limited to concepts or data modalities (e.g., images) commonly seen on the Internet. Beyond applications in the Internet domain such as remote sensing or medical imagery, perception models have yet to show their prowess. The key challenge in building perception models beyond Internet applications is the requirement of extensive expert involvement. Training performant perception models in these domains often requires non-trivial involvement from experts, especially during the data collection process. In this dissertation, we investigate how we could reduce experts’ burden when developing perception models. Specifically, we will focus on the angle of label efficiency, i.e., developing perception models that could be trained with fewer annotations. We will present two broad categories of approaches. The first category relies on minimal assumptions and could be applied to various problem domains; along this vein, we will examine how we could leverage pre-trained models, unlabeled data, and coarsely-labeled data to enhance label efficiency. The second category leverages domain knowledge to enhance label efficiency. For this category of approaches, we will look at two specific domains: autonomous driving and remote sensing. We will investigate how repeated traversals of the same location could be used to improve perception models for self-driving vehicles and how ground images could be used to train vision-language models for remote sensing without any textual annotations. We will end this dissertation with a brief discussion of how we could further reduce experts’ burden when developing perception models, enabling broader success of perception models beyond Internet applications.