IDEAS home Printed from https://ideas.repec.org/a/spr/joptap/v190y2021i1d10.1007_s10957-021-01869-0.html
   My bibliography  Save this article

Analysis of Optimization Algorithms via Sum-of-Squares

Author

Listed:
  • Sandra S. Y. Tan

    (National University of Singapore)

  • Antonios Varvitsiotis

    (National University of Singapore)

  • Vincent Y. F. Tan

    (National University of Singapore)

Abstract

We introduce a new framework for unifying and systematizing the performance analysis of first-order black-box optimization algorithms for unconstrained convex minimization. The low-cost iteration complexity enjoyed by first-order algorithms renders them particularly relevant for applications in machine learning and large-scale data analysis. Relying on sum-of-squares (SOS) optimization, we introduce a hierarchy of semidefinite programs that give increasingly better convergence bounds for higher levels of the hierarchy. Alluding to the power of the SOS hierarchy, we show that the (dual of the) first level corresponds to the performance estimation problem (PEP) introduced by Drori and Teboulle (Math Program 145(1):451–482, 2014), a powerful framework for determining convergence rates of first-order optimization algorithms. Consequently, many results obtained within the PEP framework can be reinterpreted as degree-1 SOS proofs, and thus, the SOS framework provides a promising new approach for certifying improved rates of convergence by means of higher-order SOS certificates. To determine analytical rate bounds, in this work, we use the first level of the SOS hierarchy and derive new results for noisy gradient descent with inexact line search methods (Armijo, Wolfe, and Goldstein).

Suggested Citation

  • Sandra S. Y. Tan & Antonios Varvitsiotis & Vincent Y. F. Tan, 2021. "Analysis of Optimization Algorithms via Sum-of-Squares," Journal of Optimization Theory and Applications, Springer, vol. 190(1), pages 56-81, July.
    • Handle: RePEc:spr:joptap:v:190:y:2021:i:1:d:10.1007_s10957-021-01869-0
    DOI: 10.1007/s10957-021-01869-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10957-021-01869-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10957-021-01869-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. TAYLOR, Adrien B. & HENDRICKX, Julien M. & François GLINEUR, 2016. "Exact worst-case performance of first-order methods for composite convex optimization," LIDAM Discussion Papers CORE 2016052, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    2. DE KLERK, Etienne & GLINEUR, François & TAYLOR, Adrien B., 2016. "On the Worst-case Complexity of the Gradient Method with Exact Line Search for Smooth Strongly Convex Functions," LIDAM Discussion Papers CORE 2016027, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    3. Laurent, M., 2009. "Sums of squares, moment matrices and optimization over polynomials," Other publications TiSEM 9fef820b-69d2-43f2-a501-e, Tilburg University, School of Economics and Management.
    4. Taylor, A. & Hendrickx, J. & Glineur, F., 2015. "Smooth Strongly Convex Interpolation and Exact Worst-case Performance of First-order Methods," LIDAM Discussion Papers CORE 2015013, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    5. Adrien B. Taylor & Julien M. Hendrickx & François Glineur, 2018. "Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization," Journal of Optimization Theory and Applications, Springer, vol. 178(2), pages 455-476, August.
    6. David G. Luenberger & Yinyu Ye, 2016. "Linear and Nonlinear Programming," International Series in Operations Research and Management Science, Springer, edition 4, number 978-3-319-18842-3, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abbaszadehpeivasti, Hadi & de Klerk, Etienne & Zamani, Moslem, 2022. "The exact worst-case convergence rate of the gradient method with fixed step lengths for L-smooth functions," Other publications TiSEM 061688c6-f97c-4024-bb5b-1, Tilburg University, School of Economics and Management.
    2. Adrien B. Taylor & Julien M. Hendrickx & François Glineur, 2018. "Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization," Journal of Optimization Theory and Applications, Springer, vol. 178(2), pages 455-476, August.
    3. Roland Hildebrand, 2021. "Optimal step length for the Newton method: case of self-concordant functions," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 94(2), pages 253-279, October.
    4. André Uschmajew & Bart Vandereycken, 2022. "A Note on the Optimal Convergence Rate of Descent Methods with Fixed Step Sizes for Smooth Strongly Convex Functions," Journal of Optimization Theory and Applications, Springer, vol. 194(1), pages 364-373, July.
    5. Donghwan Kim & Jeffrey A. Fessler, 2021. "Optimizing the Efficiency of First-Order Methods for Decreasing the Gradient of Smooth Convex Functions," Journal of Optimization Theory and Applications, Springer, vol. 188(1), pages 192-219, January.
    6. Xiao Wang & Xinzhen Zhang & Guangming Zhou, 2020. "SDP relaxation algorithms for $$\mathbf {P}(\mathbf {P}_0)$$P(P0)-tensor detection," Computational Optimization and Applications, Springer, vol. 75(3), pages 739-752, April.
    7. Laurent, Monique & Vargas, Luis Felipe, 2022. "Finite convergence of sum-of-squares hierarchies for the stability number of a graph," Other publications TiSEM 3998b864-7504-4cf4-bc1d-f, Tilburg University, School of Economics and Management.
    8. Laurent, M. & Rostalski, P., 2012. "The approach of moments for polynomial equations," Other publications TiSEM f08f3cd2-b83e-4bf1-9322-a, Tilburg University, School of Economics and Management.
    9. Tomohiko Mizutani & Makoto Yamashita, 2013. "Correlative sparsity structures and semidefinite relaxations for concave cost transportation problems with change of variables," Journal of Global Optimization, Springer, vol. 56(3), pages 1073-1100, July.
    10. Fook Wai Kong & Polyxeni-Margarita Kleniati & Berç Rustem, 2012. "Computation of Correlated Equilibrium with Global-Optimal Expected Social Welfare," Journal of Optimization Theory and Applications, Springer, vol. 153(1), pages 237-261, April.
    11. TAYLOR, Adrien B. & HENDRICKX, Julien M. & François GLINEUR, 2016. "Exact worst-case performance of first-order methods for composite convex optimization," LIDAM Discussion Papers CORE 2016052, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    12. Michael K. McWilliam & Antariksh C. Dicholkar & Frederik Zahle & Taeseong Kim, 2022. "Post-Optimum Sensitivity Analysis with Automatically Tuned Numerical Gradients Applied to Swept Wind Turbine Blades," Energies, MDPI, vol. 15(9), pages 1-19, April.
    13. Hao Hu & Renata Sotirov, 2021. "The linearization problem of a binary quadratic problem and its applications," Annals of Operations Research, Springer, vol. 307(1), pages 229-249, December.
    14. repec:hal:wpspec:info:hdl:2441/4lhe3u3c38ojohjlcbfaupcjr is not listed on IDEAS
    15. Hedlund, Jonas, 2017. "Bayesian persuasion by a privately informed sender," Journal of Economic Theory, Elsevier, vol. 167(C), pages 229-268.
    16. Shenglong Hu & Guoyin Li & Liqun Qi, 2016. "A Tensor Analogy of Yuan’s Theorem of the Alternative and Polynomial Optimization with Sign structure," Journal of Optimization Theory and Applications, Springer, vol. 168(2), pages 446-474, February.
    17. P. M. Kleniati & P. Parpas & B. Rustem, 2010. "Decomposition-based Method for Sparse Semidefinite Relaxations of Polynomial Optimization Problems," Journal of Optimization Theory and Applications, Springer, vol. 145(2), pages 289-310, May.
    18. T. D. Chuong & V. Jeyakumar & G. Li, 2019. "A new bounded degree hierarchy with SOCP relaxations for global polynomial optimization and conic convex semi-algebraic programs," Journal of Global Optimization, Springer, vol. 75(4), pages 885-919, December.
    19. O. P. Ferreira & S. Z. Németh, 2019. "On the spherical convexity of quadratic functions," Journal of Global Optimization, Springer, vol. 73(3), pages 537-545, March.
    20. Papp, Dávid & Regős, Krisztina & Domokos, Gábor & Bozóki, Sándor, 2023. "The smallest mono-unstable convex polyhedron with point masses has 8 faces and 11 vertices," European Journal of Operational Research, Elsevier, vol. 310(2), pages 511-517.
    21. Jiawang Nie & Suhan Zhong, 2023. "Loss functions for finite sets," Computational Optimization and Applications, Springer, vol. 84(2), pages 421-447, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:joptap:v:190:y:2021:i:1:d:10.1007_s10957-021-01869-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.