Building Marine Foundation Models: Problem Formulation, Models and Applications

PhD Thesis Proposal Defence


Title: "Building Marine Foundation Models: Problem Formulation, Models and 
Applications"

by

Mr. Ziqiang ZHENG


Abstract:

The marine ecosystem is the most productive of all ecosystems and shares 
immense ecological, social, and economic value. Performing marine study 
plays a significant role in protecting the marine environment and 
understanding marine science. The marine research involves the study of 
marine biology, oceanography, and environmental science through the lens of 
visual data, enabling scientists and researchers to observe, document, and 
analyze the vast and mysterious creatures beneath the water's surface. 
Existing marine studies highly depend on describing and analyzing the 
collected image/video data based on in-situ marine/underwater surveying 
approaches. There are two main limitations for existing marine studies: 1) 
they cannot support a very large scale data collection and data scarcity has 
become one of the important factors that hinder the further development of 
the marine analysis; 2) further data analysis procedure still requires many 
human labors, time costs, and is also limited to specific biology users. 
Recent foundation models have achieved great success, driven by a 
significant scale of training data and powerful networks. Such foundation 
model recipe leads to efficient and flexible models, supporting a wide 
spectrum of downstream visual analysis tasks. However, few attempts have 
been explored in the marine field and we aim to build effective and 
efficient marine foundation models. Furthermore, most existing marine visual 
analysis algorithms are mainly data-driven, specially designed for some 
tasks and pre-defined conditions. In this work, we try to formulate the 
basic tasks for marine visual understanding and explore the solutions for 
large-scale, efficient, repeated surveying, monitoring and further analysis 
procedures. We first identify the specific and universal challenge of the 
aquatic environments, the visibility degradation due to the water medium. We 
propose to do the underwater image/video enhancement to alleviate the 
visibility issue. We then perform panoptic understanding of the marine world 
comprehensively, formulating how to do the marine visual analysis in theory 
and comprehensively. Based on the intrinsic properties, we design different 
foundation models: CoralSCOP for coral reef segmentation and MarineInst for 
marine creature identification. We extend our research from image domain to 
the video field, ensuring 3D scene reconstruction, understanding and 4D 
animation. We have also reviewed the existing marine datasets and analysis 
platforms designed for marine analysis. The detailed and hierarchical 
discussions about potential applications of built marine foundation models 
are also included. Finally, we discuss the insightful future directions for 
promoting the marine visual analysis.


Date:                   Monday, 6 January 2025

Time:                   10:00am - 12:00noon

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Prof. Sai-Kit Yeung (Supervisor)
                        Prof. Hongbo Fu (Chairperson, AMC)
                        Dr. Qifeng Chen
                        Dr. Dan Xu