A novel stochastic conjugate gradient algorithm based on a stochastic differential equation perspective

It is widely recognized that data, algorithms, and applications are the three core elements of artificial intelligence and machine learning. These elements are interdependent and jointly drive technological advancement and practical implementation. Among them, algorithms play a bridging role by processing data, extracting useful information, and supporting applications. Efficient algorithm design is therefore of critical importance in scientific research and practical applications. This paper focuses on the stochastic conjugate gradient algorithm, whose stochastic variants require further refinement. The key contributions are listed as follows: (i) A second-order stochastic differential equation is proposed as a continuous-time approximation to analyze the dynamics of stochastic conjugate gradient algorithm. A theoretical framework based on Lyapunov function theory is established, marking the first theoretical analysis of this algorithm in a continuous-time setting. (ii) We introduce a modified Fletcher-Reeves parameter, a two-step extrapolation technique, and a variance reduction strategy. Under suitable assumptions, the proposed algorithm achieves linear convergence. (iii) Numerical experiments on machine learning and deep learning tasks indicate that the proposed algorithm outperforms existing classical algorithms, exhibiting higher optimization efficiency across various tasks.