Federated Knowledge Transfer and Co-Optimization of Large and Small Language Models

PhD Thesis Proposal Defence


Title: "Federated Knowledge Transfer and Co-Optimization of Large and Small
Language Models"

by

Mr. Tao FAN


Abstract:

Large Language Models (LLMs) have achieved remarkable success in language 
understanding, generation, and reasoning. However, their practical deployment 
in domain-specific and privacy-sensitive environments remains difficult. In 
many real-world scenarios, valuable knowledge is distributed across 
institutions, enterprises, and edge devices, where clients possess private 
domain data but cannot directly share them with a central LLM server. 
Meanwhile, these clients often rely on efficient Small Language Models (SLMs) 
because full-scale LLM deployment and fine-tuning are prohibitively expensive. 
This creates a fundamental challenge: how to enable effective knowledge 
transfer and collaborative improvement between powerful server-side LLMs and 
efficient client-side SLMs under constraints of privacy, model ownership, 
resource limitation, and heterogeneity.

This thesis proposal addresses this challenge through a unified perspective of 
federated knowledge transfer and co-optimization of large and small language 
models. We establish the foundations and taxonomy of federated large-small 
language models by organizing the problem space along three complementary 
paradigms: downward transfer from LLMs to SLMs, upward transfer from SLMs to 
LLMs, and bidirectional co-optimization between the two. We further categorize 
transfer mechanisms across data-level, representation-level, and model-level 
interfaces, clarifying the trade-offs among utility, efficiency, privacy, and 
model intellectual-property protection.

Building on this foundation, the proposal presents three completed frameworks. 
First, we propose FedCoT, a privacy-preserving framework for transferring 
reasoning knowledge from LLMs to SLMs. FedCoT perturbs client prompts with 
Differential Privacy and exploits Chain-of-Thought (CoT) rationales generated 
by the server-side LLM to improve the reasoning ability of client SLMs without 
exposing sensitive queries.

Second, we introduce PPC-GPT, a federated task-specific compression framework 
that compresses LLMs into lightweight SLMs through rationale-aware structured 
pruning and CoT-guided synthetic data generation, thereby preserving critical 
reasoning capability under resource constraints.

Third, we develop FedMKT, a framework for bidirectional co-optimization of 
LLMs and SLMs. FedMKT performs selective mutual knowledge distillation and 
resolves architectural and tokenizer mismatch through Minimum Edit Distance 
(MinED) token alignment, enabling both server and client models to benefit 
from each other.

Extensive experiments on diverse text generation and reasoning benchmarks show 
that the completed methods consistently outperform existing federated 
baselines. For future work, we will study FedProxy for heterogeneity-aware 
upward knowledge fusion from client-side SLMs to a proprietary server-side 
LLM. FedProxy aims to address the trilemma of LLM intellectual-property 
protection, client data privacy, and reliable knowledge fusion by using 
high-fidelity Proxy SLMs, heterogeneity-aware aggregation, and training-free 
plug-in fusion. Overall, this proposal establishes a principled and practical 
foundation for privacy-aware federated knowledge transfer between large and 
small language models, and outlines a concrete path toward collaborative, 
efficient, and secure deployment of language intelligence in distributed 
environments.


Date:                   Monday, 21 September 2026

Time:                   2:00pm - 4:00pm

Venue:                  Room 5501
                        Lift 25/26

Committee Members:      Prof. Qiang Yang (Supervisor)
                        Prof. Kai Chen (Co-supervisor)
                        Dr. Dan Xu (Chairperson)
                        Dr. May Fung