HideNseek: Federated Lottery Ticket via Server-side Pruning and Sign Supermask

MPhil Thesis Defence


Title: "HideNseek: Federated Lottery Ticket via Server-side Pruning and 
Sign Supermask"

By

Mr. Anish Krishna VALLAPURAM


Abstract

Federated learning is a distributed machine learning paradigm that 
preserves user privacy by only communicating model updates computed 
locally among clients to the central server. However, this significantly 
affects the training performance and user experience because the clients’ 
datasets are statistically heterogeneous and the computation and 
transmission of local model updates are costly for their 
resource-constrained devices. Prior art has addressed these issues by 
incorporating personalization with model compression schemes including 
quantization and pruning. The pruning nonetheless is computationally 
expensive as it is data-dependent and must be performed on the 
client-side. Furthermore, pruning commonly involves learning a binary 
supermask ∈ {0, 1} which restricts the model capacity with no 
computational benefit. In this work, we propose HideNseek which performs 
one-shot pruning on a randomly initialized model on the server-side in a 
data-agnostic manner by selecting the most synaptically salient weights as 
the subnetwork. The clients then collectively learn a sign supermask ∈ 
{−1, +1} that is multiplied to the unpruned weights for faster 
convergence while maintaining the same model compression rate as the 
state-of-the-art. Experiments on three learning tasks reveals that 
HideNseek improves inference accuracies by upto 40.6% compared to the 
state-of-the-art while reducing the communication cost by up to 39.7% and 
training time by up to 46.8%.


Date:  			Monday, 25 July 2022

Time:			4:00pm - 6:00pm

Zoom Meeting:
https://hkust.zoom.us/j/96323377009?pwd=SUovdUV0cWRUcVdYbW9IdmpHMzRKZz09

Committee Members:	Prof. Pan Hui (Supervisor, EMIA)
 			Dr. Tristan Braud (Supervisor)
 			Prof. James Kwok (Chairperson)
 			Dr. Brian Mak


**** ALL are Welcome ****