IDEAS home Printed from https://ideas.repec.org/a/spr/coopap/v91y2025i2d10.1007_s10589-025-00664-1.html
   My bibliography  Save this article

Spectral stochastic gradient method with additional sampling for finite and infinite sums

Author

Listed:
  • Nataša Krklec Jerinkić

    (University of Novi Sad)

  • Valeria Ruggiero

    (University of Ferrara)

  • Ilaria Trombini

    (University of Ferrara
    University of Parma)

Abstract

In this paper, we propose a new stochastic gradient method for numerical minimization of finite sums. We also propose a modified version of this method applicable on more general problems referred to as infinite sum problems, where the objective function is in the form of mathematical expectation. The method is based on a strategy to exploit the effectiveness of the well-known Barzilai–Borwein (BB) rules or variants of these (BB-like) rules for updating the step length in the standard gradient method. The proposed method adapts the aforementioned strategy into the stochastic framework by exploiting the same Sample Average Approximations (SAA) estimator of the objective function for several iterations. Furthermore, the sample size is controlled by an additional sampling which also plays a role in accepting the proposed iterate point. Moreover, the number of “inner” iterations with the same sample is also controlled by an adaptive rule which prevents the method from getting stuck with the same estimator for too long. Convergence results are discussed for the finite and infinite sum version, for general and strongly convex objective functions. For the strongly convex case, we provide convergence rate and worst-case complexity analysis. Numerical experiments on well-known datasets for binary classifications show very promising performance of the method, without the need to provide special values for hyperparameters on which the method depends.

Suggested Citation

  • Nataša Krklec Jerinkić & Valeria Ruggiero & Ilaria Trombini, 2025. "Spectral stochastic gradient method with additional sampling for finite and infinite sums," Computational Optimization and Applications, Springer, vol. 91(2), pages 717-758, June.
    • Handle: RePEc:spr:coopap:v:91:y:2025:i:2:d:10.1007_s10589-025-00664-1
    DOI: 10.1007/s10589-025-00664-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10589-025-00664-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10589-025-00664-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. O. P. Ferreira & M. Lemes & L. F. Prudente, 2022. "On the inexact scaled gradient projection method," Computational Optimization and Applications, Springer, vol. 81(1), pages 91-125, January.
    2. Y. H. Dai, 2002. "On the Nonmonotone Line Search," Journal of Optimization Theory and Applications, Springer, vol. 112(2), pages 315-330, February.
    3. Nataša Krejić & Nataša Krklec Jerinkić & Ángeles Martínez & Mahsa Yousefi, 2024. "A non-monotone trust-region method with noisy oracles and additional sampling," Computational Optimization and Applications, Springer, vol. 89(1), pages 247-278, September.
    4. di Serafino, Daniela & Ruggiero, Valeria & Toraldo, Gerardo & Zanni, Luca, 2018. "On the steplength selection in gradient methods for unconstrained optimization," Applied Mathematics and Computation, Elsevier, vol. 318(C), pages 176-195.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stefania Bellavia & Valentina Simone & Benedetta Morini, 2025. "Preface: New trends in large scale optimization," Computational Optimization and Applications, Springer, vol. 91(2), pages 351-356, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giulia Ferrandi & Michiel E. Hochstenbach & Nataša Krejić, 2023. "A harmonic framework for stepsize selection in gradient methods," Computational Optimization and Applications, Springer, vol. 85(1), pages 75-106, May.
    2. Crisci, Serena & Ruggiero, Valeria & Zanni, Luca, 2019. "Steplength selection in gradient projection methods for box-constrained quadratic programs," Applied Mathematics and Computation, Elsevier, vol. 356(C), pages 312-327.
    3. O. P. Ferreira & M. Lemes & L. F. Prudente, 2022. "On the inexact scaled gradient projection method," Computational Optimization and Applications, Springer, vol. 81(1), pages 91-125, January.
    4. Ubaldo M. García Palomares, 2023. "Convergence of derivative-free nonmonotone Direct Search Methods for unconstrained and box-constrained mixed-integer optimization," Computational Optimization and Applications, Springer, vol. 85(3), pages 821-856, July.
    5. Roberto Andreani & Marcos Raydan, 2021. "Properties of the delayed weighted gradient method," Computational Optimization and Applications, Springer, vol. 78(1), pages 167-180, January.
    6. M. Reza Peyghami & D. Ataee Tarzanagh, 2015. "A relaxed nonmonotone adaptive trust region method for solving unconstrained optimization problems," Computational Optimization and Applications, Springer, vol. 61(2), pages 321-341, June.
    7. Marko Miladinović & Predrag Stanimirović & Sladjana Miljković, 2011. "Scalar Correction Method for Solving Large Scale Unconstrained Minimization Problems," Journal of Optimization Theory and Applications, Springer, vol. 151(2), pages 304-320, November.
    8. Qu, Shaojian & Ji, Ying & Jiang, Jianlin & Zhang, Qingpu, 2017. "Nonmonotone gradient methods for vector optimization with a portfolio optimization application," European Journal of Operational Research, Elsevier, vol. 263(2), pages 356-366.
    9. Serena Crisci & Federica Porta & Valeria Ruggiero & Luca Zanni, 2023. "Hybrid limited memory gradient projection methods for box-constrained optimization problems," Computational Optimization and Applications, Springer, vol. 84(1), pages 151-189, January.
    10. Corsaro, Stefania & De Simone, Valentina & Marino, Zelda, 2021. "Split Bregman iteration for multi-period mean variance portfolio optimization," Applied Mathematics and Computation, Elsevier, vol. 392(C).
    11. Shi, Zhenjun & Wang, Shengquan, 2011. "Nonmonotone adaptive trust region method," European Journal of Operational Research, Elsevier, vol. 208(1), pages 28-36, January.
    12. Juliano B. Francisco & Douglas S. Gonçalves & Fermín S. V. Bazán & Lila L. T. Paredes, 2020. "Non-monotone inexact restoration method for nonlinear programming," Computational Optimization and Applications, Springer, vol. 76(3), pages 867-888, July.
    13. Yakui Huang & Yu-Hong Dai & Xin-Wei Liu & Hongchao Zhang, 2022. "On the acceleration of the Barzilai–Borwein method," Computational Optimization and Applications, Springer, vol. 81(3), pages 717-740, April.
    14. Xiaopeng Zhao & Jen-Chih Yao, 2022. "Linear convergence of a nonmonotone projected gradient method for multiobjective optimization," Journal of Global Optimization, Springer, vol. 82(3), pages 577-594, March.
    15. Lei Yang, 2024. "Proximal Gradient Method with Extrapolation and Line Search for a Class of Non-convex and Non-smooth Problems," Journal of Optimization Theory and Applications, Springer, vol. 200(1), pages 68-103, January.
    16. di Serafino, Daniela & Ruggiero, Valeria & Toraldo, Gerardo & Zanni, Luca, 2018. "On the steplength selection in gradient methods for unconstrained optimization," Applied Mathematics and Computation, Elsevier, vol. 318(C), pages 176-195.
    17. Livieris, Ioannis E. & Pintelas, Panagiotis, 2015. "A new class of nonmonotone conjugate gradient training algorithms," Applied Mathematics and Computation, Elsevier, vol. 266(C), pages 404-413.
    18. R. Díaz Millán & O. P. Ferreira & J. Ugon, 2024. "Extragradient method with feasible inexact projection to variational inequality problem," Computational Optimization and Applications, Springer, vol. 89(2), pages 459-484, November.
    19. Jiao Li & Yu-Fei Yang & Bo Yu, 2012. "A nonmonotone PSB algorithm for solving unconstrained optimization," Computational Optimization and Applications, Springer, vol. 52(1), pages 267-280, May.
    20. E. Loli Piccolomini & V. L. Coli & E. Morotti & L. Zanni, 2018. "Reconstruction of 3D X-ray CT images from reduced sampling by a scaled gradient projection algorithm," Computational Optimization and Applications, Springer, vol. 71(1), pages 171-191, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:coopap:v:91:y:2025:i:2:d:10.1007_s10589-025-00664-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.