DualMask: Federated optimization of privacy-utility-efficiency trilemma via orthogonal gradient perturbation and RL-optimized PSO

Federated learning faces a fundamental privacy-utility-communication trilemma, and existing static defense mechanisms suffer from rigid adaptation and poor multidimensional coordination, leaving a critical gap in dynamic trade-off balancing. To address this, we propose DualMask, a cooperative optimization framework that integrates a client-side Adaptive Orthogonal Noise Canceler (AONC) with server-side Distributed Dueling Double Deep Q-Network (D3QN) scheduling and Particle Swarm Optimization (PSO)-based aggregation. The AONC module implements a triple-defense mechanism via orthogonal subspace projection: (1) layer-wise adaptive EMA-quantile clipping to mitigate threshold imbalance, (2) progress-aware noise decay that balances early-stage privacy with late-stage efficiency, and (3) directional tuning that dynamically adjusts parallel-to-orthogonal gradient ratios. On the server side, D3QN enables dynamic resource allocation across heterogeneous devices, while PSO fusion corrects non-IID aggregation bias through particle-swarm-based weight optimization. Experiments on CIFAR-10/100 and Shakespeare datasets demonstrate that DualMask achieves 5.2% higher accuracy (84.1% vs 79.4% in non-IID settings) and 34.4% faster convergence (210 vs 320 rounds) compared to FedAvg. Additionally, DualMask reduces the privacy budget ϵ from 4.5 to 2.8 and communication cost by 37.2% (45 MB vs 65 MB). This constitutes a significant Pareto improvement, substantially expanding the trilemma frontier. The code and data are available at https://github.com/zhou-weib/DualMask.git.