Object Segmentation in Neural Radiance Field

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Object Segmentation in Neural Radiance Field"

By

Mr. Yichen LIU


Abstract:

First, we present one of the first learning-based 3D instance segmentation 
pipelines in Neural Radiance Field (NeRF), dubbed as Instance-NeRF. Taking a 
NeRF pretrained from multi-view RGB images as input, Instance-NeRF can learn 3D 
instance segmentation of a given scene, represented as an instance field 
component of the NeRF model. To this end, we adopt a 3D proposal-based mask 
prediction network on the sampled volumetric features from NeRF, which 
generates discrete 3D instance masks. The coarse 3D mask prediction is then 
projected to image space to match 2D segmentation masks from different views 
generated by existing panoptic segmentation models, which are used to supervise 
the training of the instance field. Notably, beyond generating consistent 2D 
segmentation maps from novel views, Instance-NeRF can query instance 
information at any 3D point, which greatly enhances NeRF object segmentation 
and manipulation.

Next, we introduce the Segment Anything for NeRF in High Quality (SANeRF-HQ) to 
achieve high-quality 3D segmentation of any target object in a given scene. 
SANeRF-HQ utilizes Segment Anything Model (SAM) for open-world object 
segmentation guided by user-supplied prompts, while leveraging NeRF to 
aggregate information from different x viewpoints. To overcome the 
aforementioned challenges, we employ density field and RGB similarity to 
enhance the accuracy of segmentation boundary during the aggregation. 
Emphasizing on segmentation accuracy, we evaluate our method on multiple NeRF 
datasets where high-quality ground-truths are available or manually annotated. 
SANeRFHQ shows a significant quality improvement over state-of-the-art methods 
in NeRF object segmentation, provides higher flexibility for object 
localization, and enables more consistent object segmentation across multiple 
views.


Date:                   Monday, 3 June 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 5510
                        Lifts 25/26

Chairman:               Prof. Pedro SANDER

Committee Members:      Prof. Chi-Keung TANG (Supervisor)
                        Prof. Dit-Yan YEUNG