A Survey on Advancements and Challenges of Model-based Reinforcement Learning

PhD Qualifying Examination


Title: "A Survey on Advancements and Challenges of Model-based Reinforcement 
Learning"

by

Mr. Hon Hing CHAK


Abstract:

Model-based reinforcement learning is a sub-field in reinforcement learning. It 
studies the problem of optimal control, a sequential decision-making problem 
that maximizes an objective value based on the decisions acted on an 
environment. This sub-field describes a class of heuristics that learns a model 
(called a world model) of the environment, and a second model (called an agent) 
that produces decisions with high objective values. The agent utilizes the 
world model during its learning process.

In the literature, the control problem, or namely the reinforcement learning 
problem, is typically described under a mathematical framework, called the 
Markov Decision Process (MDP). MDP specifies how the environment and the 
objective value change in response to decisions. Many problems with an 
interactive nature can be framed as MDPs, such as games, robotic control, stock 
trading, etc.. Typical reinforcement learning algorithms without environment 
modeling, i.e., model-free reinforcement learning, can only be effective for 
some problems in practice. If interacting with the environment is costly or 
unsafe, such as in self-driving or medical applications, model-free algorithms 
would not be suitable. This is because model-free algorithms rely on large 
amounts of interactive data to learn agents with high performances.

Recent studies have explored new methodologies in model-based reinforcement 
learning. Utilizing the recent advancement of deep learning, current 
model-based methods could create world models that generate high-quality 
simulated interactions in some environments, which are crucial for training 
high-performing agents. For example, modern model-based methods use large 
language models to implement the world models, which have achieved great 
results in the past years. In this survey, we first review different types of 
reinforcement learning problems and their challenges to modern reinforcement 
learning techniques. We then study some recent approaches in the model-based 
reinforcement learning field and their results in common baseline reinforcement 
learning problems in the field. Finally, we conclude this survey by 
highlighting weaknesses that are still present in current model-based 
methodologies and how we might address them in future research.


Date:                   Friday, 28 June 2024

Time:                   2:00pm - 4:00pm

Venue:                  Room 4472
                        Lifts 25/26

Committee Members:      Prof. Raymond Wong (Supervisor)
                        Prof. Dit-Yan Yeung (Chairperson)
                        Prof. Gary Chan
                        Prof. Shing-Chi Cheung