Federated Knowledge Transfer and Co-Optimization of Large and Small Language Models

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Federated Knowledge Transfer and Co-Optimization of Large and Small
Language Models"

By

Mr. Tao FAN


Abstract:

Large Language Models (LLMs) have achieved remarkable success in language
understanding, generation, and reasoning. However, their practical deployment in
domain-specific and privacy-sensitive environments remains difficult. In many
real-world scenarios, valuable knowledge is distributed across institutions,
enterprises, and edge devices, where clients possess private domain data but
cannot directly share them with a central LLM server. Meanwhile, these clients
often rely on efficient Small Language Models (SLMs) because full-scale LLM
deployment and fine-tuning are prohibitively expensive. This creates a
fundamental challenge: how to enable effective knowledge transfer and
collaborative improvement between powerful server-side LLMs and efficient
client-side SLMs under constraints of privacy, model ownership, and
heterogeneity.

This thesis addresses this challenge through a unified perspective of federated
knowledge transfer and co-optimization of large and small language models. We
study three complementary paradigms: downward transfer from LLMs to SLMs, upward
transfer from SLMs to LLMs, and bidirectional co-optimization between the two.
Across these paradigms, the thesis develops four frameworks that systematically
tackle privacy preservation, model compression, heterogeneity-aware aggregation,
and mutual knowledge alignment.

First, we propose FedCoT, a privacy-preserving framework for transferring
reasoning knowledge from LLMs to SLMs. FedCoT perturbs client prompts with
Differential Privacy and exploits Chain-of-Thought (CoT) rationales generated by
the server-side LLM to improve the reasoning ability of client SLMs without
exposing sensitive queries. Second, we introduce PPC-GPT, a federated
task-specific compression framework that compresses LLMs into lightweight SLMs
through rationale-aware structured pruning and CoT-guided synthetic data
generation, thereby preserving critical reasoning capability under resource
constraints. Third, we present FedProxy for upward knowledge transfer from
clients to a proprietary server-side LLM. FedProxy replaces weak adapters with a
high-fidelity Proxy SLM, mitigates parameter interference under heterogeneous
client data via H-TIES and conflict-aware regularization, and fuses learned
domain knowledge back into the LLM through a training-free plug-in mechanism.
Fourth, we develop FedMKT, a framework for bidirectional co-optimization of LLMs
and SLMs. FedMKT performs selective mutual knowledge distillation and resolves
architectural and tokenizer mismatch through Minimum Edit Distance (MinED) token
alignment, enabling both server and client models to benefit from each other.

Extensive experiments on diverse text generation and reasoning benchmarks show
that the proposed methods consistently outperform existing federated baselines.
Overall, this thesis establishes a principled and practical foundation for
privacy-aware federated knowledge transfer between large and small language
models, offering a systematic pathway toward collaborative, efficient, and
secure deployment of language intelligence in distributed environments.


Date:                   Monday, 30 November 2026

Time:                   4:30pm - 6:30pm

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               TBC

Committee Members:      Prof. Kai CHEN (Supervisor)
                        Prof. Qiang YANG (Co-supervisor)
                        Dr. May FUNG
                        Prof. Qiong LUO
                        Dr. Hao LIU (HKUST-GZ)
                        Dr. Mingyang ZHOU (SZU)