Toward Trustworthy Multimodal & Multi-Agent AI in Precision Medicine

Speaker: Dr. Yujin Oh
Assistant Professor
Department of Biomedical Systems Informatics
Yonsei University College of Medicine
  and
AI+X Professor
Department of Artificial Intelligence
Yonsei University

Title: Toward Trustworthy Multimodal & Multi-Agent AI in Precision Medicine

Date: Monday, 26 October 2026

Time: 4:00pm - 5:00pm

Venue: Lecture Theater F (Leung Yat Sing Lecture Theater), near lift 25/26, HKUST

Abstract:

Recent advances in artificial intelligence (AI) foundation models offer opportunities to support precision medicine and reduce clinical workload. By integrating clinical narratives, radiological images, and therapeutic strategies, multimodal models can approximate aspects of the comprehensive, context-sensitive reasoning used by medical professionals. Translating these capabilities into dependable clinical support requires consistency across tasks, generalizability across institutions, and coordination across medical specialties.

We first introduce three large language model (LLM)-driven multimodal AI systems for radiation oncology. LLMSeg incorporates clinical text into target volume contouring, connecting clinical context with anatomical information to support accurate, text-informed delineation. Its strong performance in breast cancer cases demonstrates the value of combining language-based reasoning with image-based analysis.

Building on this foundation, RO-LMM supports interconnected tasks throughout the radiotherapy planning workflow, including clinical context summarization, treatment plan recommendation, and plan-guided target volume segmentation. By integrating task-specific AI agents and their interconnections, RO-LMM allows information generated at one stage to inform subsequent stages, promoting consistency between clinical interpretation, treatment recommendations, and target delineation.

We further extend this approach through the multimodal Mixture of Multicenter Experts (MoME) framework, which integrates specialized institutional expertise through a decentralized architecture. MoME captures diverse clinical strategies without requiring massive data sharing and aligns institutional knowledge with radiological features, enhancing global generalizability while preserving local adaptability. Multi-center validation of LLMSeg and RO-LMM, together with the benefits of MoME, supports the development of reliable and scalable multimodal AI.

Beyond supporting workflows within a single specialty, we propose an LLM-based multi-agent framework for multidisciplinary team discussion, motivated by CoMMa. In this framework, agents representing complementary medical specialties would exchange clinical interpretations, discuss treatment options, and synthesize their perspectives into coordinated recommendations. This extension shifts the focus from individual clinical tasks toward collaborative reasoning across disciplines, reflecting the multidisciplinary nature of oncology decision-making. Together, these directions connect multimodal information integration, institutional expertise, and multi-agent collaboration to advance trustworthy AI for precision medicine.

Biography:

Yujin Oh is an Assistant Professor in the Department of Biomedical Systems Informatics at Yonsei University College of Medicine and an AI+X Professor in the Department of Artificial Intelligence at Yonsei University. Previously, Dr. Oh was a postdoctoral researcher at Harvard Medical School and Massachusetts General Hospital and a Senior Software Engineer in the CTO Division of LG Electronics. Dr. Oh received a Ph.D. in Engineering from KAIST’s Graduate School of AI and B.S. and M.S. degrees in Biomedical Engineering from Korea University.