A locally sequentially reweighted gradient descent estimator to enhance statistical efficiency for decentralized federated learning

Federated learning has become increasingly important for training collaborative models due to its privacy-preserving capabilities. While many studies have considered the numerical convergence of federated learning algorithms, far less attention has been given to their statistical convergence. In this paper, to enhance statistical efficiency, we propose a novel Locally Sequentially Re-weighted Gradient Descent (LSRGD) estimator for decentralized federated learning. Furthermore, we show that under a linear regression model with homogeneous data, the LSRGD estimator is asymptotically normal. With an appropriate learning rate, the LSRGD estimator achieves statistical efficiency comparable to that of the global estimator. We then extend this optimal statistical efficiency to general models and loss functions for heterogeneous data. Moreover, we also propose a parallel version of the LSRGD algorithm, referred to as LSRGD-P, which exhibits numerical and statistical convergence properties similar to those of LSRGD. Finally, extensive experiments demonstrate that the proposed LSRGD and LSRGD-P estimators exhibit superior statistical efficiency compared to existing competitors, while maintaining comparable convergence speeds and computational efficiency. This advantage is particularly pronounced in scenarios where sample sizes vary significantly across different clients, highlighting the robustness and adaptability of the proposed methods in heterogeneous federated learning environments with imbalanced data.