Learning-based Geometric Image Matching with Modern Deep Learning Techniques

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Learning-based Geometric Image Matching with Modern Deep Learning
Techniques"

By

Mr. Zixin LUO


Abstract:

Geometric image matching requires to establish sparse correspondences on
2D image points, in order to recover camera geometry in a static 3D scene.
This serves as an essential basis for a broad range of computer vision
tasks, including panorama stitching, visual localization,
Structure-from-Motion (SfM), Simultaneous Localization and Mapping (SLAM),
Augmented Reality (AR) and 3D reconstruction. During the past decade,
hand-crafted keypoint features and engineered feature matchers have been
widely used in practical image matching pipelines as the de-facto
standard. However, the performance of traditional methods is found
somewhat saturated, especially in identifying reliable correspondences
across images with large perspective or lighting changes. As a result,
this limitation has become the bottleneck to reconstruct scenes of
next-level accuracy and completeness.

With the emerging of deep learning, a great amount of effort has been
spent on reformulating each component of image matching through modern
neural networks, so as to be optimized in a data-driven and differentiable
manner. In this thesis, we will first review the recent achievements on
learning-based image matching techniques, then reveal the substantial
challenges arisen from practical use, and finally elaborate the methods we
have proposed that give rise to state-of-the-art results on several
important benchmarking datasets.

More specifically, we decompose the learning-based image matching pipeline
into four sub-problems, including a novel local feature extractor 1) with
a keypoint detector and 2) a keypoint descriptor, where we address the
accuracy of keypoint localization, the efficiency of training data
sampling, the aggregation of contextual information, and a joint learning
scheme for both tasks. Next, we develop 3) an image retrieval system that
shortlists the matching candidates from a large image collection, which is
in particular optimized for SfM by explicitly identifying geometric image
overlaps even without clear-defined semantics. Finally, we design 4)
a feature matcher that rejects outlier correspondences and solves
geometric models from two views, where an order-invariant neural network
is designed that builds feature hierarchy and explores spatial context
from point input.

To facilitate above research, we also present a large-scale dataset that
employs an automatic pipeline to generate rich and accurate geometric
training labels from well-reconstructed 3D models. The proposed methods
have been integrated into several important applications and extensively
evaluated across diversified scenarios, where drastic improvements and
strong generalization ability are demonstrated. Furthermore, we show great
potential for future improvements and promising extension to related areas
such as 3D point cloud alignment.


Date:                   Monday, 10 February 2020

Time:                   2:00pm - 4:00pm

Zoom Meeting:           https://hkust.zoom.us/j/680242354

Chairman:               Prof. Matthew MCKAY (ECE)

Committee Members:      Prof. Long QUAN (Supervisor)
                        Prof. Qifeng CHEN
                        Prof. Huamin QU
                        Prof. Shaojie SHEN (ECE)
                        Prof. Wenping WANG (HKU)


**** ALL are Welcome ****