IDEAS home Printed from https://ideas.repec.org/a/inm/ormoor/v46y2021i3p912-945.html

Statistical Query Algorithms for Mean Vector Estimation and Stochastic Convex Optimization

Author

Listed:
  • Vitaly Feldman

    (Apple, Inc., Cupertino, California 95014)

  • Cristóbal Guzmán

    (Institute for Mathematical and Computational Engineering, Facultad de Matemáticas & Escuela de Ingeniería, Pontificia Universidad Católica de Chile, Region Metropolitana, 3580000 Santiago, Chile)

  • Santosh Vempala

    (ANID – Millennium Science Initiative Program – Millennium Nucleus Center for the Discovery of Structures in Complex Data, 7820244 Santiago, Chile; School of Computer Science, Georgia Institute of Technology, Atlanta, Georgia 30332)

Abstract

Stochastic convex optimization, by which the objective is the expectation of a random convex function, is an important and widely used method with numerous applications in machine learning, statistics, operations research, and other areas. We study the complexity of stochastic convex optimization given only statistical query (SQ) access to the objective function. We show that well-known and popular first-order iterative methods can be implemented using only statistical queries. For many cases of interest, we derive nearly matching upper and lower bounds on the estimation (sample) complexity, including linear optimization in the most general setting. We then present several consequences for machine learning, differential privacy, and proving concrete lower bounds on the power of convex optimization–based methods. The key ingredient of our work is SQ algorithms and lower bounds for estimating the mean vector of a distribution over vectors supported on a convex body in R d . This natural problem has not been previously studied, and we show that our solutions can be used to get substantially improved SQ versions of Perceptron and other online algorithms for learning halfspaces.

Suggested Citation

  • Vitaly Feldman & Cristóbal Guzmán & Santosh Vempala, 2021. "Statistical Query Algorithms for Mean Vector Estimation and Stochastic Convex Optimization," Mathematics of Operations Research, INFORMS, vol. 46(3), pages 912-945, August.
    • Handle: RePEc:inm:ormoor:v:46:y:2021:i:3:p:912-945
    DOI: 10.1287/moor.2020.1111
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/moor.2020.1111
    Download Restriction: no
    File URL: https://libkey.io/10.1287/moor.2020.1111?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Leonid G. Khachiyan, 1996. "Rounding of Polytopes in the Real Number Model of Computation," Mathematics of Operations Research, INFORMS, vol. 21(2), pages 307-320, May.
    2. DEVOLDER, Olivier & GLINEUR, François & NESTEROV, Yurii, 2011. "First-order methods of smooth convex optimization with inexact oracle," LIDAM Discussion Papers CORE 2011002, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    3. Adam Tauman Kalai & Santosh Vempala, 2006. "Simulated Annealing for Convex Optimization," Mathematics of Operations Research, INFORMS, vol. 31(2), pages 253-266, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stephen Baumert & Archis Ghate & Seksan Kiatsupaibul & Yanfang Shen & Robert L. Smith & Zelda B. Zabinsky, 2009. "Discrete Hit-and-Run for Sampling Points from Arbitrary Distributions Over Subsets of Integer Hyperrectangles," Operations Research, INFORMS, vol. 57(3), pages 727-739, June.
    2. Mohit Singh & Weijun Xie, 2020. "Approximation Algorithms for D -optimal Design," Mathematics of Operations Research, INFORMS, vol. 45(4), pages 1512-1534, November.
    3. Nikita Kornilov & Mohammad Alkousa & Eduard Gorbunov & Fedor Stonyakin & Pavel Dvurechensky & Alexander Gasnikov, 2025. "Intermediate Gradient Methods with Relative Inexactness," Journal of Optimization Theory and Applications, Springer, vol. 207(3), pages 1-42, December.
    4. Jueyou Li & Zhiyou Wu & Changzhi Wu & Qiang Long & Xiangyu Wang, 2016. "An Inexact Dual Fast Gradient-Projection Method for Separable Convex Optimization with Linear Coupled Constraints," Journal of Optimization Theory and Applications, Springer, vol. 168(1), pages 153-171, January.
    5. Masaru Ito, 2016. "New results on subgradient methods for strongly convex optimization problems with a unified analysis," Computational Optimization and Applications, Springer, vol. 65(1), pages 127-172, September.
    6. TAYLOR, Adrien B. & HENDRICKX, Julien M. & François GLINEUR, 2016. "Exact worst-case performance of first-order methods for composite convex optimization," LIDAM Discussion Papers CORE 2016052, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    7. Xuexue Zhang & Sanyang Liu & Nannan Zhao, 2023. "An Extended Gradient Method for Smooth and Strongly Convex Functions," Mathematics, MDPI, vol. 11(23), pages 1-14, November.
    8. Masoud Ahookhosh, 2019. "Accelerated first-order methods for large-scale convex optimization: nearly optimal complexity under strong convexity," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 89(3), pages 319-353, June.
    9. Haixiang Zhang & Zeyu Zheng & Javad Lavaei, 2023. "Gradient-Based Algorithms for Convex Discrete Optimization via Simulation," Operations Research, INFORMS, vol. 71(5), pages 1815-1834, September.
    10. Julian Rasch & Antonin Chambolle, 2020. "Inexact first-order primal–dual algorithms," Computational Optimization and Applications, Springer, vol. 76(2), pages 381-430, June.
    11. Selin Ahipaşaoğlu, 2015. "Fast algorithms for the minimum volume estimator," Journal of Global Optimization, Springer, vol. 62(2), pages 351-370, June.
    12. Renbo Zhao, 2026. "New Analysis of an Away-Step Frank–Wolfe Method for Minimizing Log-Homogeneous Barriers," Mathematics of Operations Research, INFORMS, vol. 51(1), pages 35-59, January.
    13. Liam Madden & Stephen Becker & Emiliano Dall’Anese, 2021. "Bounds for the Tracking Error of First-Order Online Optimization Methods," Journal of Optimization Theory and Applications, Springer, vol. 189(2), pages 437-457, May.
    14. Etienne de Klerk & Monique Laurent, 2018. "Comparison of Lasserre’s Measure-Based Bounds for Polynomial Optimization to Bounds Obtained by Simulated Annealing," Mathematics of Operations Research, INFORMS, vol. 43(4), pages 1317-1325, November.
    15. DEVOLDER, Olivier & GLINEUR, François & NESTEROV, Yurii, 2013. "First-order methods with inexact oracle: the strongly convex case," LIDAM Discussion Papers CORE 2013016, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    16. Lovász, L. & Deák, I., 2012. "Computational results of an O∗(n4) volume algorithm," European Journal of Operational Research, Elsevier, vol. 216(1), pages 152-161.
    17. NESTEROV, Yurii, 2013. "Universal gradient methods for convex optimization problems," LIDAM Discussion Papers CORE 2013026, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    18. Le Thi Khanh Hien & Cuong V. Nguyen & Huan Xu & Canyi Lu & Jiashi Feng, 2019. "Accelerated Randomized Mirror Descent Algorithms for Composite Non-strongly Convex Optimization," Journal of Optimization Theory and Applications, Springer, vol. 181(2), pages 541-566, May.
    19. DEVOLDER, Olivier, 2011. "Stochastic first order methods in smooth convex optimization," LIDAM Discussion Papers CORE 2011070, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    20. Kimon Fountoulakis & Rachael Tappenden, 2018. "A flexible coordinate descent method," Computational Optimization and Applications, Springer, vol. 70(2), pages 351-394, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormoor:v:46:y:2021:i:3:p:912-945. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.