Design of accelerated learning algorithms based on FOPI control

High-performance optimization algorithms are essential for deep learning, and a novel accelerated learning algorithm based on fractional-order proportional-integral is proposed, which achieves a better performance both in convergence speed and global search ability. Firstly, existing accelerated optimization algorithms are expressed by a closed-loop system with different controllers, where the SGDM optimizer is reconstructed as a second-order dynamic system with a proportional controller. By replacing the integer-order integral with the fractional-order integral, the FOPI optimizer is then given, where the long-term memory characteristic of the fractional-order integral is utilized to dynamically modulate the weights of historical gradients. Finally, an explicit Euler discretization is applied to derive a computationally efficient iterative algorithm, and experiments both on benchmark functions and standard datasets demonstrate that the proposed optimization algorithm significantly accelerates the convergence speed and improves training accuracy.