Optimizing Multimodal Sensing and Inference for Resource-Constrained Devices: A Literature Review

PhD Qualifying Examination


Title: "Optimizing Multimodal Sensing and Inference for Resource-Constrained 
Devices: A Literature Review"

by

Mr. Runxi HUANG


Abstract:

Real-time multimodal sensing and inference on resource-constrained devices is 
essential for enabling applications such as autonomous driving, human-computer 
interaction, smart health care and mobile agents. Such systems continuously 
collect, process and fuse data from heterogeneous sensor modalities (e.g., 
videos, audios, IMUs) to enhance performance on complex tasks. However, 
deploying end-to-end multimodal perception systems on devices presents 
several major challenges, including the tight coupling between sensing 
dynamics and model execution, the complex inter-modality dependencies, as 
well as limited and dynamic resource availabilities. To systematically 
investigate advancements in this field, this survey categorizes existing 
optimization approaches according to the key stages of on-device multimodal 
sensing and inference, including: 1) data input optimization which focuses on 
adaptively selecting informative sensory data, 2) model optimization which 
aims to model size, 3) inference strategy optimization which designs adaptive 
inference and scheduling mechanisms, and 4) end-to-end optimization which 
jointly optimizes data acquisition and model execution under resource 
constraints. For each category, we outline the fundamental challenges, review 
representative methodologies, and analyze their strengths and limitations. 
Finally, we discuss the current progress and highlight promising research 
directions for advancing end-to-end on-device multimodal inference systems.


Date:                   Monday, 26 January 2026

Time:                   4:00pm - 6:00pm

Venue:                  Room 5501
                        Lifts 25/26

Committee Members:      Dr. Xiaomin Ouyang (Supervisor)
                        Prof. Gary Chan (Chairperson)
                        Dr. Chaojian Li