Robust 3D Object Understanding and Generation: Identifying Vulnerabilities and Addressing Real-World Challenges

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Robust 3D Object Understanding and Generation: Identifying 
Vulnerabilities and Addressing Real-World Challenges"

By

Miss Jaeyeon KIM


Abstract:

The ability to build robust and accurate 3D object understanding and 
generation systems is critical as applications in autonomous systems, 
robotics, and virtual environments continue to grow. As these technologies 
become increasingly integrated into our daily lives, the capacity to 
interpret and manipulate three-dimensional environments with precision and 
resilience is essential. This thesis addresses the core challenge of 
developing robust 3D models that can withstand the complexities of real- 
world scenarios. It focuses on identifying vulnerabilities in 3D object 
understanding and generation, presenting real-world scenarios where 
robustness can be compromised, and proposing effective solutions to address 
these issues. We begin by investigating vulnerabilities in point cloud-based 
models, demonstrating how small perturbations can lead to significant 
misclassifications and highlighting the critical need for improved robustness 
in these systems. The second focus of this thesis is on addressing the 
challenges of robust point cloud inversion and editing. Due to the unordered 
and irregular structure of point clouds, maintaining geometric consistency 
and feature disentanglement during editing is particularly difficult. This 
complexity makes it challenging to map point clouds into the latent space of 
generative models. To address this, we propose novel techniques for point 
cloud inversion that ensure both geometric consistency and feature 
disentanglement are preserved throughout the process. As advancements in 3D 
generation based on large-scale 2D datasets have emerged, our research 
extends beyond point clouds to explore the broader robustness challenges of 
3D-aware image synthesis. This area addresses the challenge of generating 
view- consistent, high-quality images from multiple perspectives without 
relying on extensive 3D data or computationally expensive training processes. 
Leveraging pre-trained 2D generative models, we propose solutions that enable 
scalable and robust 3D-aware image synthesis.

In conclusion, this thesis makes significant contributions to advancing the 
robustness of 3D object understanding and generation. Through the 
identification of adversarial vulnerabilities, the development of robust 
point cloud manipulation techniques, and innovations in scalable 3D-aware 
image synthesis, we provide a comprehensive approach to improving the 
reliability and effectiveness of 3D technologies in real-world scenarios.


Date:                   Friday, 29 November 2024

Time:                   10:30am - 12:30pm

Venue:                  Room 4503
                        Lifts 25/26

Chairman:               Prof. Yu WANG (CIVL)

Committee Members:      Prof. Sai-Kit YEUNG (Supervisor)
                        Prof. Chi-Keung TANG
                        Dr. Tristan BRAUD
                        Prof. Hongbo FU (AMC)
                        Dr. Tian FENG (ZJU)