Utilizing 2D diffusion model for text commanded NeRF editing

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering

Final Year Thesis Oral Defense

Title: "Utilizing 2D diffusion model for text commanded NeRF editing"

by

XU Yanbo

Abstract:

The tasks of media manipulation have always been important. With the 
advancements in AI-based image manipulation in the field of computer 
vision, many methods for editing in the 2D images domain have been 
proposed and proven effective. However, in the realm of 3D modeling, the 
process still requires significant manual labor. Therefore, its automaton 
would greatly benefit the multimedia industry, including fields such as 3D 
photos, Virtual Reality, and Gaming. Recent success of leverage 2D 
diffusion models (DMs) for 3D model generation conditioned on textual 
inputs shows the possibility of guiding 3D generation using 2D models. 
However, the guidance is not yet feasible in the real data domain. In 
light of this, we propose a novel NeRF-based 3D editing method using DMs 
as guidance and text as commands to edit real-world NeRF models. The 
quality of edited model, 3D consistency for rendered images, and 
preservation of original object characteristics are the primary focus for 
evaluation.


Date            : 2 May 2023 (Tuesday)

Time            : 17:00 - 17:40

Venue           : Room 4475 (near lifts 25/26), HKUST

Advisor         : Dr. CHEN Qifeng

2nd Reader      : Dr. WANG Shuai