Integer-Only Training and Model Restructuring for Edge Artificial Intelligence

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Integer-Only Training and Model Restructuring for Edge Artificial
Intelligence"

By

Mr. Jaewoo SONG


Abstract:

Edge Artificial Intelligence (Edge AI) faces unique challenges due to the
limited computational power, memory, and storage of edge devices.
Quantization, the process of converting floating-point parameters to integers,
plays a critical role in addressing these limitations by improving speed and
reducing model size. However, quantization for Edge AI suffers from two major
drawbacks: the lack of integer-based training for deep neural networks (DNNs),
and the high quantization errors caused by linear quantization algorithms
supported by edge devices.

This thesis proposes two complementary solutions to these challenges:
PocketNN and SplitQuant. PocketNN introduces an integer-only training
algorithm and a lightweight C++ framework that allow training and inference of
DNNs using standard integer arithmetic. By replacing backpropagation with
direct feedback alignment and designing specialized integer-only activation
functions, PocketNN prevents integer overflow while achieving accuracy close
to floating-point training on benchmark datasets. The framework ensures
maximum portability and compatibility for low-power edge devices.

SplitQuant addresses quantization errors in pretrained models by restructuring
DNN layers into three smaller, mathematically equivalent layers using k-means
clustering on weights and biases. This split reduces the value range in each
layer, increasing quantization resolution without losing critical information
such as outliers. Evaluations on Llama 3.2 1B and 3B Instruct models and
fine-tuned BERT-Tiny models demonstrate significant improvements in low-bit
quantization accuracy, especially for 2- and 4-bit integers, approaching
original floating-point performance. SplitQuant runs efficiently on CPUs
without GPUs or calibration datasets, making it practical for real-world Edge
AI deployment.

Together, PocketNN and SplitQuant advance integer-based DNN training and
low-bit quantization, enabling more efficient and accurate Edge AI solutions.
Both have been released as open source tools to foster research and
development in Edge AI.


Date:                   Thursday, 16 October 2025

Time:                   3:00pm - 5:00pm

Venue:                  Room 5501
                        Lifts 25/26

Chairman:               Prof. Jinwoo KIM (ECON)

Committee Members:      Prof. Fangzhen LIN (Supervisor)
                        Prof. Gary CHAN
                        Prof. Song GUO
                        Prof. Jun ZHANG (ECE)
                        Dr. Jaechang NAM (Handong Global University)