More about HKUST
Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information
The Hong Kong University of Science and Technology Department of Computer Science and Engineering MPhil Thesis Defence Title: "Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information" By Mr. Yau Wai YIM Abstract: Large language models (LLMs) have succeeded in simple imperfect information games and multi-agent coordination, but their effectiveness in complex collaborative environments remains underexplored. This study evaluates open-source and API-based LLMs in sophisticated text-based games requiring agent collaboration under imperfect information, comparing their performance against established baselines. We propose a Theory of Mind (ToM) planning technique enabling LLM agents to adapt strategies against various adversaries using only game rules, current state, and historical context. An external tool addresses the challenge of dynamic and extensive action spaces. Results reveal that while LLMs underperform state-of-the-art reinforcement learning models, they demonstrate significant ToM capabilities. This improves their performance against opposing agents, indicating their ability to understand both ally and adversary actions and establish effective collaboration. Our findings suggest LLMs' potential for complex multi-agent scenarios despite current limitations. Date: Wednesday, 16 July 2025 Time: 2:00pm - 4:00pm Venue: Room 5501 Lifts 25/26 Chairman: Prof. Kai CHEN Committee Members: Dr. Yangqiu SONG (Supervisor) Dr. Binhang YUAN