LEARNING TO MATCH IMAGES WITH KEYPOINTS AND DESCRIPTORS

Abstract

Image matching is a fundamental problem in computer vision. In the context of feature-based correspondence matching, SIFT and its variants have long excelled in a wide array of applications. Under narrow baseline viewing conditions, this problem has been successfully addressed. However, for ultra-wide baselines, as in the case of aerial images captured under large camera rotations, the appearance variation goes beyond the reach of SIFT and RANSAC. In this thesis, the problem of wide baseline matching is studied from various angles. Initially, the use of synthetic view generation and self-similarity to guide a matching procedure is leveraged to address challenges in matching aerial imagery. This is then followed by a data-driven deep-learning-based approach that sidesteps local correspondence by framing the problem as a classification task in a weak-supervision framework. However, local correspondences' usefulness is demonstrated by the incorporation of an attention mechanism to produce a set of probable matches, which allows a further increase in performance. The models outperform the state-of-the-art on ultra-wide baseline matching and approach human accuracy as per a study conducted on Amazon Mechanical Turk. Further, the learning of keypoint detection and description in a fully-supervised setting is then studied, where a large-scale dataset of patches with matching multiscale keypoints is collected. That dataset was used to learn a model capable of identifying and describing meaningful keypoints. Finally, the need for data collection is examined for the case of learning feature descriptors, where the feasibility of learning patch descriptors from synthesized viewpoint changes of random patches is investigated. The research demonstrates the effectiveness of synthetic data in achieving comparable state-of-the-art performance on real-world non-synthetic images.