Learning to Localize Persons from Stereo Footstep Sound

MPhil Thesis Defence


Title: "Learning to Localize Persons from Stereo Footstep Sound"

By

Mr. Minsoo KHANG


Abstract

Human localization is a problem widely explored for its practical 
applications such as autonomous driving, home and campus security. Many of 
the prior works focus mainly on visual cues for localization while some 
include cues from other domain (e.g., radio frequency, speech). However, 
each localization has its own blind spots (such as walking quietly in a 
dark environment) and exploring localization cues from different domain to 
complement each other is of great importance to robust human localization. 
Furthermore, the representations of different localization cues are not 
easily compatible: Direction-Of-Arrival and visual reference frame are the 
common representations for audio cue based and visual cue based 
localization respectively.

In this thesis, we propose a new task by exploring the feasibility in 
using stereo footstep sound as a human localization cue on a visual 
reference frame. Using footstep sound as localization cue is not only 
relatively less explored but even more so for visual reference frame 
representation. In comparison to other audio cues such as music or speech, 
footstep sound typically has much lower SNRs, making localization much 
more challenging. Being the first to attempt on human localization with 
stereo footstep sound on a visual reference frame, we have not only 
verified the feasibility of the new task but also designed a MHSA-SE 
module which has shown to consistently benefit the human localization 
results. Furthermore, we also contribute a new dataset Stereo Footstep 
Dataset dedicated for this new task, which contains both single and double 
person audio and localization coordinates on a visual reference frame 
across 27 unique individuals.


Date:  			Thursday, 21 July 2022

Time:			10:00am - 12:00noon

Zoom Meeting:
https://hkust.zoom.us/j/95986806452?pwd=VTNTWDBIQ3VaelZxWEI4RFNFUFhPZz09

Committee Members:	Dr. Qifeng Chen (Supervisor)
 			Prof. Raymond Wong (Chairperson)
 			Prof. Richard So (IEDA)


**** ALL are Welcome ****