From social media to street view and all the way to satellite images, we are capturing visual data at an unprecedented scale. These images tell a story about our planet. With advances in automatic recognition, we can build a collective understanding of world-scale events as recorded through visual media. Such insights have the potential to be useful for various experts in their domain such as cultural anthropologists and ecologists. However, discovering such rare yet interesting insights from the data is very challenging. First, it requires recognition models that have an expert-level understanding of such visual domains. Second, it requires tools that can leverage such models and large-scale spatio-temporal data and discover novel insights. In this dissertation, we first look at ways of building and improving automatic recognition models in such expert domains. More specifically we look at how we can efficiently learn a representation for such domains with either no supervision or with text or attribute-based supervision. We specifically work with domains that require expertise for understanding such as ornithology or remote sensing. These methods not only aim to make the recognition models more cost-efficient but also more practical to be used with experts. More specifically we present an unsupervised method to learn representation in the satellite image domain. Then we look at an attribute-based model for bird classification (and other attribute-based domains) and introduce ways to make it more practical and label-efficient to work with. We then present methods that can discover novel insights without any supervision by looking at large-scale spatio-temporal visual data. These methods make use of domain-specific vision models to make the discovery. More specifically, we use these methods to understand fashion trends and discover cultural phenomena and social events around the world by looking at fashion images from social media. Broadening our domain to include satellite imagery, we introduce completely unsupervised techniques to discover interesting change events across the planet from satellite images. This general framework can be potentially applied in different visual domains ranging from sustainability to online commerce to discover interesting phenomena in those domains.