Investigating Virtual Reality as A Situated Learning Tool for Supporting General Education

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Investigating Virtual Reality as A Situated Learning Tool for 
Supporting General Education"

By

Mr. Zhenjie ZHAO


Abstract

General education courses such as culture, history and oral communication 
are difficult to teach in classrooms due to the lack of authentic 
contexts. Virtual reality (VR) can offer immersive environments for a 
situated learning experience, and it is also well grounded in learning 
theory, in particular constructivism, which encourages learning from 
experiences. However, there are few existing guidelines on how to design 
such VR-based learning systems to support general education. Situated 
cognition theory suggests three key elements of situated learning: an 
authentic learning context, social interaction and collaboration, and 
progressive training. For this thesis, we conducted three empirical 
studies to investigate the three elements separately in the context of 
general education with VR.

For the first study, we developed ShadowPlay2.5D, a 360-degree video 
authoring tool for immersive appreciation of classical Chinese poetry. 
Owing to the lack of authentic contexts, learning and appreciating 
classical Chinese poetry can be challenging. Using Chinese shadow play as 
a metaphor, we designed and implemented a sketch-based authoring tool to 
help novices easily create 360-degree videos about classical Chinese 
poetry. Through two user studies, we show that ShadowPlay2.5D can help 
novices make a short 360-degree video in about 10-15 minutes, and the 2.5D 
stylized illustrations created can bring about a better immersive 
experience for poetry appreciation.

For the second study, we developed Live Emoji, a live storytelling VR 
system with programmable cartoon-style emotion embodiment. While existing 
storytelling systems for democratizing VR technology, such as Google VR 
Tour, use 360-degree images to immerse users in a lifelike environment, 
engaging learners in a socially interactive way is not automatic. In fact, 
it can be quite difficult. Thus, we propose a novel cartoon-style hybrid 
emotion embodiment model to increase a storyteller's presence during live 
performance. We further designed and implemented a system to teleoperate 
the embodiment model in VR for live storytelling. Based on interviews with 
three experts and a workshop study with local secondary school students, 
we show the potential of the emotion embodiment model on VR storytelling 
for education.

In the third study, we explored whether a VR coach with embodied feedback 
could foster a situated learning experience for progressive training. To 
formulate our design, we interviewed experts and observed real elevator 
pitches. We then designed a VR coaching system with three different 
embodied feedback strategies. Through a between-subject experiment with 40 
participants, we found that receiving embodied feedback can create a 
strong sense of cognitive apprenticeship, i.e., coaching and helping from 
experts, and can also help improve the perception of the virtual character 
and the effect of learning.

Through these three studies, we gained practical insights into VR and 
situated learning. Thus in this thesis, we summarize important design 
guidelines of VR systems for supporting general education.

Finally, as a first step towards developing computational intelligence for 
supporting situated learning, we studied language grounding, which 
connects language to the real world, because of the importance of language 
on education and the necessity of an authentic context for situated 
learning. In particular, we used machine learning methods to study 
physical common sense learning and emotion recognition from the aspects of 
model generalization and small sample sizes, respectively. In the first 
study, we formulated physical common sense learning as a knowledge graph 
completion problem. We propose a novel pipeline that combines pre-training 
models and knowledge graph embedding to increase the generalization 
ability of our model to predict physical common sense. In the second 
study, we devised an efficient meta-learning approach to learn text 
emotion distribution from a small training sample.

We conclude this thesis by sketching further plans for building 
conversational agents to support situated learning, reducing cybersickness 
through mixing the physical world into the virtual world and human 
perception-optimized planning, and developing applications for cultural 
heritage education.


Date:			Wednesday, 15 July 2020

Time:			9:00am - 11:00am

Zoom Meeting:		https://hkust.zoom.us/j/97505224713

Chairman:		Prof. Kam Tim TSE (CIVL)

Committee Members:	Prof. Xiaojuan MA (Supervisor)
   			Prof. Huamin QU
 			Prof. Chiew Lan TAI
 			Prof. Richard SO (IEDA)
 			Prof. Holly RUSHMEIER (Yale University)


**** ALL are Welcome ****