Real-time Emotion Sensing In A Computer Game Using Facial Recognition

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering

Final Year Thesis Oral Defense

Title: "Real-time Emotion Sensing In A Computer Game Using Facial 
Recognition"

by

ZHU Ge

Abstract:

In this project, a detective game was built that incorporates real-time 
emotion sensing to detect the player's facial expressions during the game 
play, use a statistical model to determine the player's current emotion 
and then adjust the level of difficulty based on the emotion to help the 
player keep a healthy and positive mood during and after the gameplay.

The game was developed using the Unity Game Engine. In the game, the 
player must act as a detective and solve a murder case by talking to the 
victim's family members and searching for clues in his house. The 
emotion-sensing model was trained separately using Microsoft Visual 
Studio. We first selected 327 images from the CMU CK+ image dataset as a 
training dataset. We used OpenCV to detect the face area from each image 
and extract 18 facial landmarks from the face area using Dlib. The 
distances between those 18 landmarks are normalized and sent to Libsvm for 
training. Then, during gameplay, OpenCV is used to detect the player's 
face area in real time, compare the distances between the 18 landmarks 
with those in the database for emotion estimation. Finally, to utilize the 
estimate emotion in the gameplay, we use a Dynamic Link Library to trigger 
actions in the game.

Date                 : 8 May 2017 (Monday)

Time                 : 18:10 - 18:50

Venue                : 5564 (via lifts 27/28)

Advisor              : Dr. Pan HUI

2nd Reader           : Prof. James KWOK