LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "LLM-driven Knowledge Enhancement for Multimodal Cancer Survival 
Prediction"

By

Miss Chenyu ZHAO


Abstract:

Current multimodal survival prediction methods typically rely on pathology 
images (WSIs) and genomic data, both of which are high-dimensional and 
redundant, making it difficult to extract discriminative features from them 
and align different modalities. Moreover, using a simple survival follow-up 
label is insufficient to supervise such a complex task. To address these 
challenges, we propose KEMM, an LLM-driven Knowledge-Enhanced Multimodal 
Model for cancer survival prediction, which integrates expert reports and 
prognostic background knowledge. 1) Expert reports, provided by pathologists 
on a case-by-case basis and refined by large language model (LLM), offer 
succinct and clinically focused diagnostic statements. This information may 
typically suggest different survival outcomes. 2) Prognostic background 
knowledge (PBK), generated concisely by LLM, provides valuable prognostic 
background knowledge on different cancer types, which also enhances survival 
prediction. To leverage these knowledge, we introduce the knowledge enhanced 
cross-modal (KECM) attention module. KECM can effectively guide the network 
to focus on discriminative and survival- relevant features from highly 
redundant modalities. Extensive experiments demonstrate that KEMM achieves 
state-of-the-art performance.


Date:                   Wednesday, 25 June 2025

Time:                   4:00pm - 6:00pm

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               Dr. Dan XU

Committee Members:      Dr. Hao CHEN (Supervisor)
                        Dr. Xiaomin OUYANG