Improving Privacy and Efficiency of Federated Knowledge Distillation

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence

Title: "Improving Privacy and Efficiency of Federated Knowledge Distillation"

By

Mr. Sheng WAN


Abstract:

Growing concerns over data privacy and security have motivated Federated 
Learning (FL), which enables collaborative model training without exchanging 
raw data. Federated Knowledge Distillation (FedKD) further improves 
communication efficiency and supports heterogeneous model architectures by 
transferring knowledge through logits rather than model parameters. However, 
logit-based knowledge exchange introduces new challenges in communication 
efficiency, robustness, and privacy. This thesis aims to improve the privacy 
and efficiency of FedKD while preserving model utility under diverse 
federated learning scenarios. First, we propose FedPDD, a privacy-preserving 
double distillation framework for cross-silo FL that reduces communication 
overhead through offline knowledge exchange and incorporates differential 
privacy to protect transmitted knowledge. Second, we investigate Byzantine 
vulnerabilities in cross-device FedKD and develop FedTGD, a robust 
aggregation mechanism designed to defend against malicious manipulation of 
shared logits. Third, we study privacy leakage from logit sharing under a 
semi-honest server, introduce the Adaptive Model Stealing Attack (AdaMSA) to 
quantify this risk, and develop a lightweight defense that mitigates privacy 
leakage while preserving training performance. Finally, we establish a 
theoretical framework for decentralized FedKD through KR-DFD, a kernel ridge 
regression-based approach, and derive theoretical bounds to characterize 
learning utility and model privacy under decentralized knowledge exchange. 
Overall, this thesis advances FedKD through efficient cross-silo knowledge 
transfer, robust aggregation against malicious participants, systematic 
analysis and mitigation of privacy leakage, and theoretical characterization 
of decentralized knowledge distillation. The results provide practical 
algorithms and theoretical insights toward efficient, privacy-preserving, and 
trustworthy FedKD systems.


Date:                   Tuesday, 27 October 2026

Time:                   9:00 am - 11:00 am

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               TBC

Committee Members:      Prof. Kai CHEN (Supervisor)
                        Prof. Qiang YANG (Co-supervisor)
                        Dr. Daning HU (Co-supervisor, SUSTech)
                        Dr. Xiaojuan MA
                        Prof. Qian ZHANG
                        Prof. Can YANG (MATH)
                        Dr. Yuxin MA (SUSTech)