IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v13y2025i4p665-d1593909.html
   My bibliography  Save this article

Selective Reviews of Bandit Problems in AI via a Statistical View

Author

Listed:
  • Pengjie Zhou

    (Institute of Artificial Intelligence, Beihang University, Beijing 100191, China)

  • Haoyu Wei

    (Department of Economics, University of California San Diego, La Jolla, CA 92093, USA)

  • Huiming Zhang

    (Institute of Artificial Intelligence, Beihang University, Beijing 100191, China)

Abstract

Reinforcement Learning (RL) is a widely researched area in artificial intelligence that focuses on teaching agents decision-making through interactions with their environment. A key subset includes multi-armed bandit (MAB) and stochastic continuum-armed bandit (SCAB) problems, which model sequential decision-making under uncertainty. This review outlines the foundational models and assumptions of bandit problems, explores non-asymptotic theoretical tools like concentration inequalities and minimax regret bounds, and compares frequentist and Bayesian algorithms for managing exploration–exploitation trade-offs. Additionally, we explore K -armed contextual bandits and SCAB, focusing on their methodologies and regret analyses. We also examine the connections between SCAB problems and functional data analysis. Finally, we highlight recent advances and ongoing challenges in the field.

Suggested Citation

  • Pengjie Zhou & Haoyu Wei & Huiming Zhang, 2025. "Selective Reviews of Bandit Problems in AI via a Statistical View," Mathematics, MDPI, vol. 13(4), pages 1-53, February.
    • Handle: RePEc:gam:jmathe:v:13:y:2025:i:4:p:665-:d:1593909
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/13/4/665/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/13/4/665/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xueru Zhang & Dennis K. J. Lin & Lin Wang, 2023. "Digital Triplet: A Sequential Methodology for Digital Twin Learning," Mathematics, MDPI, vol. 11(12), pages 1-16, June.
    2. Lalit Jain & Zhaoqi Li & Erfan Loghmani & Blake Mason & Hema Yoganarasimhan, 2024. "Effective Adaptive Exploration of Prices and Promotions in Choice-Based Demand Models," Marketing Science, INFORMS, vol. 43(5), pages 1002-1030, September.
    3. Yi Cheng & Donald A. Berry, 2007. "Optimal adaptive randomized designs for clinical trials," Biometrika, Biometrika Trust, vol. 94(3), pages 673-689.
    4. Hang Zhou & Fang Yao & Huiming Zhang, 2023. "Functional linear regression for discretely observed data: from ideal to reality," Biometrika, Biometrika Trust, vol. 110(2), pages 381-393.
    5. Jianqing Fan & Yongyi Guo & Mengxin Yu, 2024. "Policy Optimization Using Semiparametric Models for Dynamic Pricing," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 552-564, January.
    6. Hao Ji & Hans-Georg Müller, 2017. "Optimal designs for longitudinal and functional data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(3), pages 859-876, June.
    7. Yi Liu & Veronika Ročková, 2023. "Variable Selection Via Thompson Sampling," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 118(541), pages 287-304, January.
    8. Marco Battiston & Stefano Favaro & Yee Whye Teh, 2018. "Multi-Armed Bandit for Species Discovery: A Bayesian Nonparametric Approach," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 455-466, January.
    9. Mengjia Yu & Xiaohui Chen, 2021. "Finite sample change point inference and identification for high‐dimensional mean vectors," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(2), pages 247-270, April.
    10. Yao Zheng & Guang Cheng, 2021. "Finite-time analysis of vector autoregressive models under linear restrictions [Nested reduced-rank autogressive models for multiple time series]," Biometrika, Biometrika Trust, vol. 108(2), pages 469-489.
    11. Ruodu Wang & Aaditya Ramdas, 2022. "False discovery rate control with e‐values," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 822-852, July.
    12. Huiming Zhang & Haoyu Wei, 2022. "Sharper Sub-Weibull Concentrations," Mathematics, MDPI, vol. 10(13), pages 1-29, June.
    13. Yao, Fang & Muller, Hans-Georg & Wang, Jane-Ling, 2005. "Functional Data Analysis for Sparse Longitudinal Data," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 577-590, June.
    14. Sungwook Kim & Michael P. Fay & Michael A. Proschan, 2021. "Valid and approximately valid confidence intervals for current status data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(3), pages 438-452, July.
    15. Wei Qian & Ching-Kang Ing & Ji Liu, 2024. "Adaptive Algorithm for Multi-Armed Bandit Problem with High-Dimensional Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(546), pages 970-982, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pengjie Zhou & Haoyu Wei & Huiming Zhang, 2024. "Selective Reviews of Bandit Problems in AI via a Statistical View," Papers 2412.02251, arXiv.org, revised Feb 2025.
    2. Park, So Young & Xiao, Luo & Willbur, Jayson D. & Staicu, Ana-Maria & Jumbe, N. L’ntshotsholé, 2018. "A joint design for functional data with application to scheduling ultrasound scans," Computational Statistics & Data Analysis, Elsevier, vol. 122(C), pages 101-114.
    3. Rha, Hyungmin & Kao, Ming-Hung & Pan, Rong, 2020. "Design optimal sampling plans for functional regression models," Computational Statistics & Data Analysis, Elsevier, vol. 146(C).
    4. Zhong, Rou & Liu, Shishi & Li, Haocheng & Zhang, Jingxiao, 2022. "Robust functional principal component analysis for non-Gaussian longitudinal data," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    5. Kao, Ming-Hung & Huang, Ping-Han, 2024. "Hybrid exact-approximate design approach for sparse functional data," Computational Statistics & Data Analysis, Elsevier, vol. 190(C).
    6. Xu, Wenchao & Zhang, Xinyu & Liang, Hua, 2024. "Linearized maximum rank correlation estimation when covariates are functional," Journal of Multivariate Analysis, Elsevier, vol. 202(C).
    7. Ana-Maria Staicu & Yingxing Li & Ciprian M. Crainiceanu & David Ruppert, 2014. "Likelihood Ratio Tests for Dependent Data with Applications to Longitudinal and Functional Data Analysis," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 932-949, December.
    8. Jianyu Xu & Yining Wang & Xi Chen & Yu-Xiang Wang, 2025. "Dynamic Pricing with Adversarially-Censored Demands," Papers 2502.06168, arXiv.org.
    9. Şentürk, Damla & Ghosh, Samiran & Nguyen, Danh V., 2014. "Exploratory time varying lagged regression: Modeling association of cognitive and functional trajectories with expected clinic visits in older adults," Computational Statistics & Data Analysis, Elsevier, vol. 73(C), pages 1-15.
    10. Álvarez-Liébana, J. & López-Pérez, A. & González-Manteiga, W. & Febrero-Bande, M., 2025. "A goodness-of-fit test for functional time series with applications to Ornstein-Uhlenbeck processes," Computational Statistics & Data Analysis, Elsevier, vol. 203(C).
    11. Wang, Jingxing & Chung, Seokhyun & AlShelahi, Abdullah & Kontar, Raed & Byon, Eunshin & Saigal, Romesh, 2021. "Look-ahead decision making for renewable energy: A dynamic “predict and store” approach," Applied Energy, Elsevier, vol. 296(C).
    12. Heredia, María Belén & Prieur, Clémentine & Eckert, Nicolas, 2022. "Global sensitivity analysis with aggregated Shapley effects, application to avalanche hazard assessment," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    13. Huiming Zhang & Haoyu Wei & Guang Cheng, 2023. "Tight Non-asymptotic Inference via Sub-Gaussian Intrinsic Moment Norm," Papers 2303.07287, arXiv.org, revised Jan 2024.
    14. Febrero-Bande, Manuel & González-Manteiga, Wenceslao & Prallon, Brenda & Saporito, Yuri F., 2023. "Functional classification of bitcoin addresses," Computational Statistics & Data Analysis, Elsevier, vol. 181(C).
    15. Li, Pai-Ling & Chiou, Jeng-Min, 2011. "Identifying cluster number for subspace projected functional data clustering," Computational Statistics & Data Analysis, Elsevier, vol. 55(6), pages 2090-2103, June.
    16. Shuyu Meng & Zhensheng Huang, 2024. "Variable Selection in Semi-Functional Partially Linear Regression Models with Time Series Data," Mathematics, MDPI, vol. 12(17), pages 1-23, September.
    17. Xiuli Du & Xiaohu Jiang & Jinguan Lin, 2023. "Multinomial Logistic Factor Regression for Multi-source Functional Block-wise Missing Data," Psychometrika, Springer;The Psychometric Society, vol. 88(3), pages 975-1001, September.
    18. Guangxing Wang & Sisheng Liu & Fang Han & Chong‐Zhi Di, 2023. "Robust functional principal component analysis via a functional pairwise spatial sign operator," Biometrics, The International Biometric Society, vol. 79(2), pages 1239-1253, June.
    19. Jiménez Recaredo, Raúl José & Elías Fernández, Antonio, 2017. "Prediction Bands for Functional Data Based on Depth Measures," DES - Working Papers. Statistics and Econometrics. WS 24606, Universidad Carlos III de Madrid. Departamento de Estadística.
    20. Zhang, Tao & Zhang, Qingzhao & Wang, Qihua, 2014. "Model detection for functional polynomial regression," Computational Statistics & Data Analysis, Elsevier, vol. 70(C), pages 183-197.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:13:y:2025:i:4:p:665-:d:1593909. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.