Improving Privacy and Efficiency of Federation Knowledge Distillation

PhD Thesis Proposal Defence


Title: "Improving Privacy and Efficiency of Federation Knowledge 
Distillation"

by

Mr. Sheng WAN


Abstract:

Growing concerns over data privacy and security have led to increasingly 
stringent regulations on data sharing and protection. Federated Learning (FL) 
has emerged as a promising paradigm that enables collaborative model training 
without exchanging raw data. Federated Knowledge Distillation (FedKD) further 
extends FL by transferring knowledge through logits rather than model 
parameters, thereby improving communication efficiency and naturally 
supporting heterogeneous model architectures. However, the transmission of 
logits introduces new challenges in terms of communication efficiency and 
privacy protection.

This thesis aims to improve the privacy and efficiency of FedKD while 
preserving model utility under diverse federated learning scenarios. First, 
we address the challenge of achieving efficient and privacy-preserving 
knowledge transfer in cross-silo FL. We propose FedPDD, a privacy-preserving 
double distillation framework that enables local models to learn from both 
the ensemble knowledge of collaborating models and historical local 
predictions. By adopting an offline training strategy and exchanging only 
model outputs, FedPDD significantly reduces communication overhead while 
reducing privacy risks. Differential privacy is further incorporated into the 
communication process to provide formal privacy guarantees.

Second, we investigate the security of FedKD against malicious knowledge 
manipulation in cross-device FL. Existing Byzantine-robust FL methods mainly 
focus on parameter aggregation and are not directly applicable to knowledge 
distillation. To address this limitation, we identify previously unexplored 
vulnerabilities in logit-sharing FL and propose two Byzantine attack 
strategies, namely the Top-k Attack and the Impersonation Attack, which 
specifically target transmitted distilled knowledge. To mitigate these 
threats, we develop Federated Top Guard Distillation (FedTGD), a robust 
knowledge aggregation mechanism that effectively filters malicious knowledge 
while maintaining model utility.

Third, we uncover a previously overlooked privacy vulnerability in 
logit-sharing FL. We demonstrate that a semi-honest server can reconstruct 
clients' private models from shared logits and propose the Adaptive Model 
Stealing Attack (AdaMSA) to quantify this privacy risk. Furthermore, we 
establish theoretical bounds on this vulnerability and introduce a 
lightweight defense strategy that perturbs transmitted logits to minimize 
privacy leakage while preserving training performance.

Overall, this thesis advances Federated Knowledge Distillation from two 
complementary perspectives: improving communication efficiency and 
strengthening privacy protection. Through efficient cross-silo knowledge 
transfer, robust defense against malicious knowledge manipulation and 
systematic analysis and mitigation of privacy leakage in logit-sharing FL, 
this thesis provides practical algorithms and theoretical insights toward 
building efficient, privacy-preserving and trustworthy FedKD systems for 
real-world federated learning applications.


Date:                   Wednesday, 12 August 2026

Time:                   4:40pm - 6:30pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Prof. Kai Chen (Supervisor)
                        Prof. Qiang Yang (Co-Supervisor)
                        Prof. Qian Zhang (Chairperson)
                        Dr. Dan Xu