Design and Analysis of Neuromuscular Human-Machine Interfaces

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Design and Analysis of Neuromuscular Human-Machine Interfaces"

By

Mr. Kirill SHATILOV


Abstract:

Traditional human-machine interfaces, such as touch screens, keyboards,
and controllers, often fall short of meeting the requirements for
interacting with real and virtual objects in mobile contexts. They
struggle to provide the necessary mobility, efficiency, and social
acceptance that users demand. In this thesis, we aim to design and
evaluate innovative neuromuscular human-machine interfaces that provide
a direct pathway between the nervous system and machines, addressing
the aforementioned challenges.

To begin with, we explore the use of surface electromyography (sEMG),
a non-intrusive wearable imaging technique, for recognizing gestures.
sEMG works by detecting electrical signals generated by muscles through
the skin's surface, which can then be interpreted to identify performed
or intended gestures.

In our initial study, we developed and assessed a gesture recognition
system intended for controlling a 3D-printed prosthetic hand. Our
primary focus is on the mobility of the solution, where deep learning
algorithms for gesture recognition are executed on a mobile device,
with the option to offload computations to a remote server. In this
study, we design the prosthesis, which includes the necessary
electronics and a modified chassis, as well as a mobile companion
device and communication protocols that link the system's components.
Our findings demonstrate that a low-cost, mobile-centered system can
achieve state-of-the-art accuracy in gesture recognition while
maintaining reliable operation over extended periods. We evaluate the
accuracy of gesture recognition across various gesture sets and assess
the system's response time and power consumption.

Building on the insights gained from our first study, we adapt our
gesture recognition system to function as a virtual keyboard for
extended realities. In this system, users select keys on a standard
QWERTY keyboard by orienting their forearm in space to choose a column
(measured by an IMU sensor) and then selecting a specific key within
that column through a directional gesture (utilizing the sEMG sensor).
Notably, we consider three different usage scenarios for our proposed
solution: one with an empty hand, and two with a busy hand—such as when
holding an umbrella (cylindrical grasp) or a pen (tripod grasp). This
approach results in an input system for extended realities that offers
an uninterrupted experience for subtle and even discreet text entry. We
conduct experiments involving a dozen participants to evaluate text
input rates, identify common errors, and assess the overall usability
of our proposed text input system.

Finally, we look into Electrical muscle stimulation (EMS). EMS provides
direct, felt feedback by actuating muscles, offering a novel haptic
channel for extended reality scenarios. This study investigates whether
EMS enhances immersion, embodiment, and social presence compared to
visual-only and vibrotactile feedback. Participants engaged in virtual
object manipulation and handshake tasks under three feedback
conditions. Presence, realism, and acceptance were assessed using
standardized questionnaires. Results indicate that EMS feedback yields
higher presence scores than visual-only interaction, matches or exceeds
vibrotactile haptics in contact-rich tasks, and increases perceived
realism in social interactions. Within validated parameter bounds, EMS
remains safe and comfortable, with acceptance shaped by both novelty
and trust. These findings provide empirical evidence for EMS as a
socially accepted feedback modality and inform the design of
next-generation extended reality systems that integrate muscle-level
haptics for richer, more embodied experiences.


Date:                   Thursday, 29 January 2026

Time:                   4:00pm - 6:00pm

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               Prof. Ross MURCH (ECE)

Committee Members:      Dr. Tristan BRAUD (Supervisor)
                        Prof. Pan HUI (Co-supervisor, HKUST-GZ)
                        Prof. Gary CHAN
                        Prof. Andrew HORNER
                        Dr. Gareth John TYSON (HKUST-GZ)
                        Prof. Ho Man CHAN (CityU)