IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/127940.html

A review of off-policy evaluation in reinforcement learning

Author

Listed:
  • Uehara, Masatoshi
  • Shi, Chengchun
  • Kallus, Nathan

Abstract

Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems. In this paper, we primarily focus on off-policy evaluation (OPE), one of the most fundamental topics in RL. In recent years, a number of OPE methods have been developed in the statistics and computer science literature. We provide a discussion on the efficiency bound of OPE, some of the existing state-of-the-art OPE methods, their statistical properties and some other related research directions that are currently actively explored.

Suggested Citation

  • Uehara, Masatoshi & Shi, Chengchun & Kallus, Nathan, 2026. "A review of off-policy evaluation in reinforcement learning," LSE Research Online Documents on Economics 127940, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:127940
    as

    Download full text from publisher

    File URL: https://researchonline.lse.ac.uk/id/eprint/127940/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Baqun Zhang & Anastasios A. Tsiatis & Eric B. Laber & Marie Davidian, 2012. "A Robust Method for Estimating Optimal Treatment Regimes," Biometrics, The International Biometric Society, vol. 68(4), pages 1010-1018, December.
    2. Anastasios A. Tsiatis & Marie Davidian & Weihua Cao, 2011. "Improved Doubly Robust Estimation When Data Are Monotonely Coarsened, with Application to Longitudinal Studies with Dropout," Biometrics, The International Biometric Society, vol. 67(2), pages 536-545, June.
    3. Rajarshi Mukherjee & Whitney K. Newey & James Robins, 2017. "Semiparametric efficient empirical higher order influence function estimators," CeMMAP working papers CWP30/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    4. Rajarshi Mukherjee & Whitney K. Newey & James Robins, 2017. "Semiparametric efficient empirical higher order influence function estimators," CeMMAP working papers 30/17, Institute for Fiscal Studies.
    5. Chengchun Shi & Rui Song & Wenbin Lu & Bo Fu, 2018. "Maximin projection learning for optimal treatment decision with heterogeneous individualized treatment effects," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(4), pages 681-702, September.
    6. Lu Wang & Andrea Rotnitzky & Xihong Lin & Randall E. Millikan & Peter F. Thall, 2012. "Evaluation of Viable Dynamic Treatment Regimes in a Sequentially Randomized Trial of Advanced Prostate Cancer," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 493-508, June.
    7. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    8. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    9. Audrey Boruvka & Daniel Almirall & Katie Witkiewitz & Susan A. Murphy, 2018. "Assessing Time-Varying Causal Effect Moderation in Mobile Health," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1112-1121, July.
    10. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    11. Weibin Mo & Zhengling Qi & Yufeng Liu, 2021. "Rejoinder: Learning Optimal Distributionally Robust Individualized Treatment Rules," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 699-707, April.
    12. Yingqi Zhao & Donglin Zeng & A. John Rush & Michael R. Kosorok, 2012. "Estimating Individualized Treatment Rules Using Outcome Weighted Learning," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(499), pages 1106-1118, September.
    13. E Smucler & F Sapienza & A Rotnitzky, 2022. "Efficient adjustment sets in causal graphical models with hidden variables [Double/debiased machine learning for treatment and structural parameters]," Biometrika, Biometrika Trust, vol. 109(1), pages 49-65.
    14. Martin Huber, 2019. "An introduction to flexible methods for policy evaluation," Papers 1910.00641, arXiv.org.
    15. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    16. Edward H. Kennedy, 2019. "Nonparametric Causal Effects Based on Incremental Propensity Score Interventions," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(526), pages 645-656, April.
    17. Shi, Chengchun & Song, Rui & Lu, Wenbin & Fu, Bo, 2018. "Maximin projection learning for optimal treatment decision with heterogeneous individualized treatment effects," LSE Research Online Documents on Economics 102112, London School of Economics and Political Science, LSE Library.
    18. Shi, Chengchun & Wang, Xiaoyu & Luo, Shikai & Zhu, Hongtu & Ye, Jieping & Song, Rui, 2022. "Dynamic causal effects evaluation in A/B testing with a reinforcement learning framework," LSE Research Online Documents on Economics 113310, London School of Economics and Political Science, LSE Library.
    19. Yixin Wang & Jose R Zubizarreta, 2020. "Minimal dispersion approximately balancing weights: asymptotic properties and practical considerations," Biometrika, Biometrika Trust, vol. 107(1), pages 93-105.
    20. Yang Ning & Peng Sida & Kosuke Imai, 2020. "Robust estimation of causal effects via a high-dimensional covariate balancing propensity score," Biometrika, Biometrika Trust, vol. 107(3), pages 533-554.
    21. Elynn Y. Chen & Rui Song & Michael I. Jordan, 2024. "Reinforcement Learning in Latent Heterogeneous Environments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(548), pages 3113-3126, October.
    22. Runchao Jiang & Wenbin Lu & Rui Song & Marie Davidian, 2017. "On estimation of optimal treatment regimes for maximizing t-year survival probability," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(4), pages 1165-1185, September.
    23. Yunan Wu & Lan Wang, 2021. "Resampling‐based confidence intervals for model‐free robust inference on optimal treatment regimes," Biometrics, The International Biometric Society, vol. 77(2), pages 465-476, June.
    24. Wang Miao & Zhi Geng & Eric J Tchetgen Tchetgen, 2018. "Identifying causal effects with proxy variables of an unmeasured confounder," Biometrika, Biometrika Trust, vol. 105(4), pages 987-993.
    25. Ruoqing Zhu & Ying-Qi Zhao & Guanhua Chen & Shuangge Ma & Hongyu Zhao, 2017. "Greedy outcome weighted tree learning of optimal personalized treatment rules," Biometrics, The International Biometric Society, vol. 73(2), pages 391-400, June.
    26. Ashkan Ertefaie & Robert L Strawderman, 2018. "Constructing dynamic treatment regimes over indefinite time horizons," Biometrika, Biometrika Trust, vol. 105(4), pages 963-977.
    27. Edward H. Kennedy & Zongming Ma & Matthew D. McHugh & Dylan S. Small, 2017. "Non-parametric methods for doubly robust estimation of continuous treatment effects," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(4), pages 1229-1245, September.
    28. Meinshausen, Nicolai & Meier, Lukas & Bühlmann, Peter, 2009. "p-Values for High-Dimensional Regression," Journal of the American Statistical Association, American Statistical Association, vol. 104(488), pages 1671-1681.
    29. Yichi Zhang & Eric B. Laber & Marie Davidian & Anastasios A. Tsiatis, 2018. "Interpretable Dynamic Treatment Regimes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1541-1549, October.
    30. Weihua Cao & Anastasios A. Tsiatis & Marie Davidian, 2009. "Improving efficiency and robustness of the doubly robust estimator for a population mean with incomplete data," Biometrika, Biometrika Trust, vol. 96(3), pages 723-734.
    31. Daniel J. Luckett & Eric B. Laber & Anna R. Kahkoska & David M. Maahs & Elizabeth Mayer-Davis & Michael R. Kosorok, 2020. "Estimating Dynamic Treatment Regimes in Mobile Health Using V-Learning," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(530), pages 692-706, April.
    32. S. A. Murphy, 2003. "Optimal dynamic treatment regimes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 65(2), pages 331-355, May.
    33. Baqun Zhang & Anastasios A. Tsiatis & Eric B. Laber & Marie Davidian, 2013. "Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions," Biometrika, Biometrika Trust, vol. 100(3), pages 681-694.
    34. Lan Wang & Yu Zhou & Rui Song & Ben Sherwood, 2018. "Quantile-Optimal Treatment Regimes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1243-1254, July.
    35. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    36. Caiyun Fan & Wenbin Lu & Rui Song & Yong Zhou, 2017. "Concordance-assisted learning for estimating optimal individualized treatment regimes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(5), pages 1565-1582, November.
    37. Smucler Ezequiel & Rotnitzky Andrea, 2022. "A note on efficient minimum cost adjustment sets in causal graphical models," Journal of Causal Inference, De Gruyter, vol. 10(1), pages 174-189, January.
    38. Chunrong Ai & Xiaohong Chen, 2003. "Efficient Estimation of Models with Conditional Moment Restrictions Containing Unknown Functions," Econometrica, Econometric Society, vol. 71(6), pages 1795-1843, November.
    39. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    40. Karel Vermeulen & Stijn Vansteelandt, 2015. "Bias-Reduced Doubly Robust Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 1024-1036, September.
    41. Zhiqiang Tan, 2010. "Bounded, efficient and doubly robust estimation with inverse weighting," Biometrika, Biometrika Trust, vol. 97(3), pages 661-682.
    42. repec:bla:biomet:v:71:y:2015:i:4:p:895-904 is not listed on IDEAS
    43. Wensheng Zhu & Donglin Zeng & Rui Song, 2019. "Proper Inference for Value Function in High-Dimensional Q-Learning for Dynamic Treatment Regimes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(527), pages 1404-1417, July.
    44. Peng Liao & Predrag Klasnja & Susan Murphy, 2021. "Off-Policy Estimation of Long-Term Average Outcomes With Applications to Mobile Health," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(533), pages 382-391, March.
    45. Chengchun Shi & Rui Song & Wenbin Lu & Runze Li, 2021. "Statistical Inference for High-Dimensional Models via Recursive Online-Score Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(535), pages 1307-1318, July.
    46. Weibin Mo & Zhengling Qi & Yufeng Liu, 2021. "Learning Optimal Distributionally Robust Individualized Treatment Rules," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 659-674, April.
    47. Roland A. Matsouaka & Junlong Li & Tianxi Cai, 2014. "Evaluating marker-guided treatment selection strategies," Biometrics, The International Biometric Society, vol. 70(3), pages 489-499, September.
    48. Wenzhuo Zhou & Ruoqing Zhu & Annie Qu, 2024. "Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 625-638, January.
    49. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    50. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    51. Shi, Chengchun & Zhang, Shengxing & Lu, Wenbin & Song, Rui, 2022. "Statistical inference of the value function for reinforcement learning in infinite-horizon settings," LSE Research Online Documents on Economics 110882, London School of Economics and Political Science, LSE Library.
    52. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    53. Ai, Chunrong & Chen, Xiaohong, 2012. "The semiparametric efficiency bound for models of sequential moment restrictions containing unknown functions," Journal of Econometrics, Elsevier, vol. 170(2), pages 442-457.
    54. Whitney K. Newey, 2013. "Nonparametric Instrumental Variables Estimation," American Economic Review, American Economic Association, vol. 103(3), pages 550-556, May.
    55. Chamberlain, Gary, 1992. "Sequential Moment Restrictions in Panel Data: Comment," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(1), pages 20-26, January.
    56. Isabel R. Fulcher & Ilya Shpitser & Stella Marealle & Eric J. Tchetgen Tchetgen, 2020. "Robust inference on population indirect causal effects: the generalized front door criterion," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(1), pages 199-214, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shi, Chengchun & Wan, Runzhe & Song, Ge & Luo, Shikai & Zhu, Hongtu & Song, Rui, 2023. "A multiagent reinforcement learning framework for off-policy evaluation in two-sided markets," LSE Research Online Documents on Economics 117174, London School of Economics and Political Science, LSE Library.
    2. Gao, Yuhe & Shi, Chengchun & Song, Rui, 2023. "Deep spectral Q-learning with application to mobile health," LSE Research Online Documents on Economics 119445, London School of Economics and Political Science, LSE Library.
    3. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    4. Shi, Chengchun & Luo, Shikai & Le, Yuan & Zhu, Hongtu & Song, Rui, 2024. "Statistically efficient advantage learning for offline reinforcement learning in infinite horizons," LSE Research Online Documents on Economics 115598, London School of Economics and Political Science, LSE Library.
    5. Zhen Li & Jie Chen & Eric Laber & Fang Liu & Richard Baumgartner, 2023. "Optimal Treatment Regimes: A Review and Empirical Comparison," International Statistical Review, International Statistical Institute, vol. 91(3), pages 427-463, December.
    6. Li, Ting & Shi, Chengchun & Wen, Qianglin & Sui, Yang & Qin, Yongli & Lai, Chunbo & Zhu, Hongtu, 2024. "Combining experimental and historical data for policy evaluation," LSE Research Online Documents on Economics 125588, London School of Economics and Political Science, LSE Library.
    7. Zhang, Yingying & Shi, Chengchun & Luo, Shikai, 2023. "Conformal off-policy prediction," LSE Research Online Documents on Economics 118250, London School of Economics and Political Science, LSE Library.
    8. Masahiro Kato, 2021. "Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability," Papers 2102.08975, arXiv.org, revised Mar 2021.
    9. Muxuan Liang & Menggang Yu, 2023. "Relative contrast estimation and inference for treatment recommendation," Biometrics, The International Biometric Society, vol. 79(4), pages 2920-2932, December.
    10. Shixiao Zhang & Peisong Han & Changbao Wu, 2023. "Calibration Techniques Encompassing Survey Sampling, Missing Data Analysis and Causal Inference," International Statistical Review, International Statistical Institute, vol. 91(2), pages 165-192, August.
    11. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    12. Jianxuan Liu & Yanyuan Ma & Lan Wang, 2018. "An alternative robust estimator of average treatment effect in causal inference," Biometrics, The International Biometric Society, vol. 74(3), pages 910-923, September.
    13. Liu, Lin & Mukherjee, Rajarshi & Robins, James M., 2024. "Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators," Journal of Econometrics, Elsevier, vol. 240(2).
    14. Zhishuai Liu & Jesse Clifton & Eric B. Laber & John Drake & Ethan X. Fang, 2023. "Deep Spatial Q-Learning for Infectious Disease Control," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 28(4), pages 749-773, December.
    15. Weibin Mo & Yufeng Liu, 2022. "Efficient learning of optimal individualized treatment rules for heteroscedastic or misspecified treatment‐free effect models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 440-472, April.
    16. Rahul Singh & Liyuan Xu & Arthur Gretton, 2020. "Kernel Methods for Causal Functions: Dose, Heterogeneous, and Incremental Response Curves," Papers 2010.04855, arXiv.org, revised Oct 2022.
    17. Difang Huang & Jiti Gao & Tatsushi Oka, 2025. "Semiparametric single-index estimation for average treatment effects," Econometric Reviews, Taylor & Francis Journals, vol. 44(6), pages 843-885, July.
    18. Jiaming Mao & Jingzhi Xu, 2020. "Ensemble Learning with Statistical and Structural Models," Papers 2006.05308, arXiv.org.
    19. Nina Deliu & Joseph Jay Williams & Bibhas Chakraborty, 2025. "Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions," International Statistical Review, International Statistical Institute, vol. 93(3), pages 385-424, December.
    20. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:127940. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.