IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2012.00219.html
   My bibliography  Save this paper

Unbounded Dynamic Programming via the Q-Transform

Author

Listed:
  • Qingyin Ma
  • John Stachurski
  • Alexis Akira Toda

Abstract

We propose a new approach to solving dynamic decision problems with unbounded rewards based on the transformations used in Q-learning. In our case, the objective of the transform is to convert an unbounded dynamic program into a bounded one. The approach is general enough to handle problems for which existing methods struggle, and yet simple relative to other techniques and accessible for applied work. We show by example that many common decision problems satisfy our conditions.

Suggested Citation

  • Qingyin Ma & John Stachurski & Alexis Akira Toda, 2020. "Unbounded Dynamic Programming via the Q-Transform," Papers 2012.00219, arXiv.org, revised Mar 2021.
    • Handle: RePEc:arx:papers:2012.00219
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2012.00219
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Mark Aguiar & Manuel Amador & Hugo Hopenhayn & Iván Werning, 2019. "Take the Short Route: Equilibrium Default and Debt Maturity," Econometrica, Econometric Society, vol. 87(2), pages 423-462, March.
    2. Rust, John, 1987. "Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher," Econometrica, Econometric Society, vol. 55(5), pages 999-1033, September.
    3. Juan Carlos Hatchondo & Leonardo Martinez & César Sosa-Padilla, 2016. "Debt Dilution and Sovereign Default Risk," Journal of Political Economy, University of Chicago Press, vol. 124(5), pages 1383-1422.
    4. Takashi Kamihigashi, 2014. "Elementary results on solutions to the bellman equation of dynamic programming: existence, uniqueness, and convergence," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 56(2), pages 251-273, June.
    5. Le Van, Cuong & Vailakis, Yiannis, 2005. "Recursive utility and optimal growth with bounded or unbounded returns," Journal of Economic Theory, Elsevier, vol. 123(2), pages 187-209, August.
    6. Ma, Qingyin & Toda, Alexis Akira, 2021. "A theory of the saving rate of the rich," Journal of Economic Theory, Elsevier, vol. 192(C).
    7. Jonathan Heathcote & Kjetil Storesletten & Giovanni L. Violante, 2010. "The Macroeconomic Implications of Rising Wage Inequality in the United States," Journal of Political Economy, University of Chicago Press, vol. 118(4), pages 681-722, August.
    8. Andreas Fagereng & Luigi Guiso & Davide Malacrino & Luigi Pistaferri, 2020. "Heterogeneity and Persistence in Returns to Wealth," Econometrica, Econometric Society, vol. 88(1), pages 115-170, January.
    9. Paul A. Samuelson, 2011. "Lifetime Portfolio Selection by Dynamic Stochastic Programming," World Scientific Book Chapters, in: Leonard C MacLean & Edward O Thorp & William T Ziemba (ed.), THE KELLY CAPITAL GROWTH INVESTMENT CRITERION THEORY and PRACTICE, chapter 31, pages 465-472, World Scientific Publishing Co. Pte. Ltd..
    10. He, Hua & Pearson, Neil D., 1991. "Consumption and portfolio policies with incomplete markets and short-sale constraints: The infinite dimensional case," Journal of Economic Theory, Elsevier, vol. 54(2), pages 259-304, August.
    11. Alvarez, Fernando & Stokey, Nancy L., 1998. "Dynamic Programming with Homogeneous Functions," Journal of Economic Theory, Elsevier, vol. 82(1), pages 167-189, September.
    12. Shenghao Zhu, 2020. "Existence Of Stationary Equilibrium In An Incomplete‐Market Model With Endogenous Labor Supply," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 61(3), pages 1115-1138, August.
    13. Hua He & Neil D. Pearson, 1991. "Consumption and Portfolio Policies With Incomplete Markets and Short‐Sale Constraints: the Finite‐Dimensional Case1," Mathematical Finance, Wiley Blackwell, vol. 1(3), pages 1-10, July.
    14. J. J. McCall, 1970. "Economics of Information and Job Search," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 84(1), pages 113-126.
    15. Moritz Kuhn, 2013. "Recursive Equilibria In An Aiyagari‐Style Economy With Permanent Income Shocks," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 54(3), pages 807-835, August.
    16. Dan Cao & Wenlan Luo, 2017. "Persistent Heterogeneous Returns and Top End Wealth Inequality," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 26, pages 301-326, October.
    17. Jaap H. Abbring & Jeffrey R. Campbell & Jan Tilly & Nan Yang, 2018. "Very Simple Markov‐Perfect Industry Dynamics: Theory," Econometrica, Econometric Society, vol. 86(2), pages 721-735, March.
    18. S. Rao Aiyagari, 1994. "Uninsured Idiosyncratic Risk and Aggregate Saving," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 109(3), pages 659-684.
    19. Stachurski, John & Toda, Alexis Akira, 2019. "An impossibility theorem for wealth in heterogeneous-agent models with limited heterogeneity," Journal of Economic Theory, Elsevier, vol. 182(C), pages 1-24.
    20. Boud, John III, 1990. "Recursive utility and the Ramsey problem," Journal of Economic Theory, Elsevier, vol. 50(2), pages 326-345, April.
    21. Ma, Qingyin & Stachurski, John & Toda, Alexis Akira, 2020. "The income fluctuation problem and the evolution of wealth," Journal of Economic Theory, Elsevier, vol. 187(C).
    22. Aguiar, Mark & Amador, Manuel, 2019. "A contraction for sovereign debt models," Journal of Economic Theory, Elsevier, vol. 183(C), pages 842-875.
    23. Cristina Arellano, 2008. "Default Risk and Income Fluctuations in Emerging Economies," American Economic Review, American Economic Association, vol. 98(3), pages 690-712, June.
    24. Eugene A. Feinberg & Pavlo O. Kasyanov & Nina V. Zadoianchuk, 2012. "Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities," Mathematics of Operations Research, INFORMS, vol. 37(4), pages 591-607, November.
    25. V. Filipe Martins-da-Rocha & Yiannis Vailakis, 2010. "Existence and Uniqueness of a Fixed Point for Local Contractions," Econometrica, Econometric Society, vol. 78(3), pages 1127-1141, May.
    26. Jaap H. Abbring & Jeffrey R. Campbell & Jan Tilly & Nan Yang, 2018. "Very Simple Markov-Perfect Industry Dynamics: Empirics," Working Paper Series WP-2018-17, Federal Reserve Bank of Chicago.
    27. Mette Ejrnæs & Martin Browning, 2014. "The persistent–transitory representation for earnings processes," Quantitative Economics, Econometric Society, vol. 5(3), pages 555-581, November.
    28. Toda, Alexis Akira, 2014. "Incomplete market dynamics and cross-sectional distributions," Journal of Economic Theory, Elsevier, vol. 154(C), pages 310-348.
    29. Jovanovic, Boyan, 1982. "Selection and the Evolution of Industry," Econometrica, Econometric Society, vol. 50(3), pages 649-670, May.
    30. Bäuerle, Nicole & Jaśkiewicz, Anna, 2018. "Stochastic optimal growth model with risk sensitive preferences," Journal of Economic Theory, Elsevier, vol. 173(C), pages 181-200.
    31. Janusz Matkowski & Andrzej Nowak, 2011. "On discounted dynamic programming with unbounded returns," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 46(3), pages 455-474, April.
    32. Juan Carlos Hatchondo & Leonardo Martinez & Horacio Sapriza, 2009. "Heterogeneous Borrowers In Quantitative Models Of Sovereign Default," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 50(4), pages 1129-1151, November.
    33. Li, Huiyu & Stachurski, John, 2014. "Solving the income fluctuation problem with unbounded rewards," Journal of Economic Dynamics and Control, Elsevier, vol. 45(C), pages 353-365.
    34. Abbring, Jaap & Campbell, J.R. & Tilly, J. & Yang, N., 2018. "Very Simple Markov-Perfect Industry Dynamics (revision of 2017-021) : Empirics," Other publications TiSEM 3a12f099-900b-44ac-b692-a, Tilburg University, School of Economics and Management.
    35. Benhabib, Jess & Bisin, Alberto & Zhu, Shenghao, 2015. "The wealth distribution in Bewley economies with capital income risk," Journal of Economic Theory, Elsevier, vol. 159(PA), pages 489-515.
    36. Cao, Dan, 2020. "Recursive equilibrium in Krusell and Smith (1998)," Journal of Economic Theory, Elsevier, vol. 186(C).
    37. Charalambos D. Aliprantis & Kim C. Border, 2006. "Infinite Dimensional Analysis," Springer Books, Springer, edition 0, number 978-3-540-29587-7, December.
    38. Ma, Qingyin & Stachurski, John, 2019. "Optimal timing of decisions: A general theory based on continuation values," Journal of Economic Dynamics and Control, Elsevier, vol. 101(C), pages 62-81.
    39. Joachim Hubmer & Per Krusell & Anthony A. Smith Jr., 2020. "Sources of US Wealth Inequality: Past, Present, and Future," NBER Chapters, in: NBER Macroeconomics Annual 2020, volume 35, pages 391-455, National Bureau of Economic Research, Inc.
    40. Moritz Kuhn, 2013. "Recursive Equilibria In An Aiyagari‐Style Economy With Permanent Income Shocks," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 54, pages 807-835, August.
    41. Cuong Le Van & Yiannis Vailakis, 2005. "Recursive utility and optimal growth with bounded or unbounded returns," Post-Print halshs-00101201, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wang, Hanchen & Arjmandzadeh, Ziba & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2024. "FlexNet: A warm start method for deep reinforcement learning in hybrid electric vehicle energy management applications," Energy, Elsevier, vol. 288(C).
    2. Augeraud-Veron, Emmanuelle & Boucekkine, Raouf & Gozzi, Fausto & Venditti, Alain & Zou, Benteng, 2024. "Fifty years of mathematical growth theory: Classical topics and new trends," Journal of Mathematical Economics, Elsevier, vol. 111(C).
    3. Bloise, G. & Van, C. Le & Vailakis, Y., 2024. "An approximation approach to dynamic programming with unbounded returns," Journal of Mathematical Economics, Elsevier, vol. 111(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas J. Sargent & John Stachurski, 2024. "Dynamic Programming: Finite States," Papers 2401.10473, arXiv.org.
    2. Qingyin Ma & John Stachurski, 2019. "Dynamic Optimal Choice When Rewards are Unbounded Below," Papers 1911.13025, arXiv.org.
    3. Ma, Qingyin & Stachurski, John & Toda, Alexis Akira, 2020. "The income fluctuation problem and the evolution of wealth," Journal of Economic Theory, Elsevier, vol. 187(C).
    4. Ma, Qingyin & Stachurski, John, 2019. "Optimal timing of decisions: A general theory based on continuation values," Journal of Economic Dynamics and Control, Elsevier, vol. 101(C), pages 62-81.
    5. Marcello D’Amato & Christian Di Pietro & Marco M. Sorge, 2024. "Left and right: a tale of two tails of the wealth distribution," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 78(4), pages 1389-1433, December.
    6. Philippe Bich & Jean-Pierre Drugeon & Lisa Morhaim, 2015. "On Aggregators and Dynamic Programming," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01169552, HAL.
    7. Bloise, G. & Van, C. Le & Vailakis, Y., 2024. "An approximation approach to dynamic programming with unbounded returns," Journal of Mathematical Economics, Elsevier, vol. 111(C).
    8. Philippe Bich & Jean-Pierre Drugeon & Lisa Morhaim, 2018. "On temporal aggregators and dynamic programming," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 66(3), pages 787-817, October.
    9. Philippe Bich & Jean-Pierre Drugeon & Lisa Morhaim, 2015. "On Aggregators and Dynamic Programming," Post-Print halshs-01169552, HAL.
    10. Ma, Qingyin & Toda, Alexis Akira, 2022. "Asymptotic linearity of consumption functions and computational efficiency," Journal of Mathematical Economics, Elsevier, vol. 98(C).
    11. Philippe Bich & Jean-Pierre Drugeon & Lisa Morhaim, 2018. "On Temporal Aggregators and Dynamic Programming," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01437496, HAL.
    12. Philippe Bich & Jean-Pierre Drugeon & Lisa Morhaim, 2015. "On Aggregators and Dynamic Programming," Documents de travail du Centre d'Economie de la Sorbonne 15053, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    13. Bloise, Gaetano & Vailakis, Yiannis, 2018. "Convex dynamic programming with (bounded) recursive utility," Journal of Economic Theory, Elsevier, vol. 173(C), pages 118-141.
    14. John Stachurski, 2009. "Economic Dynamics: Theory and Computation," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262012774, April.
    15. Toda, Alexis Akira, 2019. "Wealth distribution with random discount factors," Journal of Monetary Economics, Elsevier, vol. 104(C), pages 101-113.
    16. Joachim Hubmer & Per Krusell & Anthony A. Smith Jr., 2020. "Sources of US Wealth Inequality: Past, Present, and Future," NBER Chapters, in: NBER Macroeconomics Annual 2020, volume 35, pages 391-455, National Bureau of Economic Research, Inc.
    17. Stachurski, John & Toda, Alexis Akira, 2019. "An impossibility theorem for wealth in heterogeneous-agent models with limited heterogeneity," Journal of Economic Theory, Elsevier, vol. 182(C), pages 1-24.
    18. Gouin-Bonenfant, Emilien & Toda, Alexis Akira, 2018. "Pareto Extrapolation: Bridging Theoretical and Quantitative Models of Wealth Inequality," University of California at San Diego, Economics Working Paper Series qt90n2h2bb, Department of Economics, UC San Diego.
    19. Robert A. Becker & Juan Pablo Rincón-Zapatero, 2017. "Arbitration and Renegotiation in Trade Agreements," CAEPR Working Papers 2017-007, Center for Applied Economics and Policy Research, Department of Economics, Indiana University Bloomington.
    20. Becker, Robert A. & Rincón-Zapatero, Juan Pablo, 2021. "Thompson aggregators, Scott continuous Koopmans operators, and Least Fixed Point theory," Mathematical Social Sciences, Elsevier, vol. 112(C), pages 84-97.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2012.00219. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.