关键词: 联邦学习, 个性化, 异构, 知识蒸馏

Abstract: In order to address the situation where participants in the Federated Learning framework have heterogeneous models and data, while also considering the issue of trust among participants, this paper proposes a personalized Federated Multi-Task Knowledge Distillation (pFedMK) algorithm. Each client not only learns the global target task but also needs to learn personalized target tasks based on local and information from other clients. The algorithm employs a two-level distillation scheme, including global distillation and mutual distillation, taking into account client reputation issues. Initially, each client trains to obtain its own soft predictions on a public dataset. The central server updates the reputation value of each client based on their soft predictions and previous round's reputation weighting, selecting the k clients with the highest reputation values (excluding itself) to form an alliance. The Shapley Value from cooperative games is calculated to determine a reasonable aggregation coefficient, and the aggregated model knowledge is then transferred to local models through knowledge distillation to complete global distillation. Subsequently, through mutual distillation using a distributed point-to-point method, under a heterogeneous model architecture, each client can learn targets from other clients. By conducting training simulations on two datasets and comparing with other similar algorithms, it is concluded that the pFedMK algorithm can improve system performance and enhance personalized accuracy.

Key words: 联邦学习, 个性化, 异构, 知识蒸馏