Fine-Tuning Pretrained Language Models with Time Series and Textual Data for Stock Movement Prediction

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Fine-Tuning Pretrained Language Models with Time Series and Textual 
Data for Stock Movement Prediction"

By

Miss Yixin LIU


Abstract:

Predicting stock movement, rise or fall, is a challenging task with high 
practical impact. Traditional methods, ranging from statistical time-series 
models and regression analyses to neural networks, rely on numerical features 
extracted from historical stock price data. Recent efforts have also taken into 
consideration non-numeric factors, such as stock events, industry trends, or 
political issues, summarized from textual data. Nevertheless, as a dynamic 
event in the open world, stock price movement remains a hard problem for 
prediction.

News articles and social media posts are important information for stock 
prediction. Traditionally, it is not easy to combine them because of the 
hardness of natural language understanding and numerical reasoning. The rise of 
LLMs makes this task doable and promising. Large Language Models (LLMs) have 
demonstrated impressive performance on sentiment analysis of financial news and 
social media posts. However, their performance on numerical data sources is 
under-explored. Therefore, in this thesis, we study stock movement prediction 
by utilizing the stock market time series data and the news/tweets textual 
data. Specifically, we use GPT4 to identify news relevant to stocks of our 
interest and summarize these news as well as stock descriptions as 
supplementary data. We then add low-rank adaptation (LoRA) to the pre-trained 
open-source Large Language model LlaMA2 and fine-tune it with these data and 
previous day stock price movement as prompts. Furthermore, we add a stock 
processing block to our modified LlaMA2 to take into past 30-days of stock 
market price embedding. Our prompts give the LLM background information and 
clear question instructions for prediction, whereas the stock price block 
together with LoRA enables the model to train with the stock time-series data. 
The result shows that our method utilizes the reasoning ability of LLMs and 
makes predictions that can outperform the baseline models. This result 
indicates that LLMs can learn both textual and numerical information to 
facilitate time series forecast.


Date:                   Tuesday, 28 May 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 4475
                        Lifts 25/26

Chairman:               Prof. Raymond WONG

Committee Members:      Prof. Qiong LUO (Supervisor)
                        Dr. Xiaojuan MA
                        Dr. Nan TANG (HKUST-GZ)