IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v12y2021i1d10.1038_s41467-021-26731-9.html
   My bibliography  Save this article

A model for learning based on the joint estimation of stochasticity and volatility

Author

Listed:
  • Payam Piray

    (Princeton University)

  • Nathaniel D. Daw

    (Princeton University)

Abstract

Previous research has stressed the importance of uncertainty for controlling the speed of learning, and how such control depends on the learner inferring the noise properties of the environment, especially volatility: the speed of change. However, learning rates are jointly determined by the comparison between volatility and a second factor, moment-to-moment stochasticity. Yet much previous research has focused on simplified cases corresponding to estimation of either factor alone. Here, we introduce a learning model, in which both factors are learned simultaneously from experience, and use the model to simulate human and animal data across many seemingly disparate neuroscientific and behavioral phenomena. By considering the full problem of joint estimation, we highlight a set of previously unappreciated issues, arising from the mutual interdependence of inference about volatility and stochasticity. This interdependence complicates and enriches the interpretation of previous results, such as pathological learning in individuals with anxiety and following amygdala damage.

Suggested Citation

  • Payam Piray & Nathaniel D. Daw, 2021. "A model for learning based on the joint estimation of stochasticity and volatility," Nature Communications, Nature, vol. 12(1), pages 1-16, December.
    • Handle: RePEc:nat:natcom:v:12:y:2021:i:1:d:10.1038_s41467-021-26731-9
    DOI: 10.1038/s41467-021-26731-9
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-021-26731-9
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-021-26731-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Matthew R. Nassar & Rasmus Bruckner & Joshua I. Gold & Shu-Chen Li & Hauke R. Heekeren & Ben Eppinger, 2016. "Age differences in learning emerge from an insufficient representation of uncertainty in older adults," Nature Communications, Nature, vol. 7(1), pages 1-13, September.
    2. Payam Piray & Nathaniel D Daw, 2020. "A simple model for learning in volatile environments," PLOS Computational Biology, Public Library of Science, vol. 16(7), pages 1-26, July.
    3. Jessica Aylward & Vincent Valton & Woo-Young Ahn & Rebecca L. Bond & Peter Dayan & Jonathan P. Roiser & Oliver J. Robinson, 2019. "Altered learning under uncertainty in unmedicated mood and anxiety disorders," Nature Human Behaviour, Nature, vol. 3(10), pages 1116-1123, October.
    4. Andreea O Diaconescu & Christoph Mathys & Lilian A E Weber & Jean Daunizeau & Lars Kasper & Ekaterina I Lomakina & Ernst Fehr & Klaas E Stephan, 2014. "Inferring on the Intentions of Others by Hierarchical Bayesian Learning," PLOS Computational Biology, Public Library of Science, vol. 10(9), pages 1-19, September.
    5. Vincent Moens & Alexandre Zénon, 2019. "Learning and forgetting using reinforced Bayesian change detection," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-41, April.
    6. Robert C Wilson & Matthew R Nassar & Joshua I Gold, 2013. "A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems," PLOS Computational Biology, Public Library of Science, vol. 9(7), pages 1-18, July.
    7. Nathaniel D. Daw & John P. O'Doherty & Peter Dayan & Ben Seymour & Raymond J. Dolan, 2006. "Cortical substrates for exploratory decisions in humans," Nature, Nature, vol. 441(7095), pages 876-879, June.
    8. Archy O. de Berker & Robb B. Rutledge & Christoph Mathys & Louise Marshall & Gemma F. Cross & Raymond J. Dolan & Sven Bestmann, 2016. "Computations of uncertainty mediate acute stress responses in humans," Nature Communications, Nature, vol. 7(1), pages 1-11, April.
    9. Jean Daunizeau & Hanneke E M den Ouden & Matthias Pessiglione & Stefan J Kiebel & Klaas E Stephan & Karl J Friston, 2010. "Observing the Observer (I): Meta-Bayesian Models of Learning and Decision-Making," PLOS ONE, Public Library of Science, vol. 5(12), pages 1-10, December.
    10. Dani Gamerman & Thiago Rezende Santos & Glaura C. Franco, 2013. "A Non-Gaussian Family Of State-Space Models With Exact Marginal Likelihood," Journal of Time Series Analysis, Wiley Blackwell, vol. 34(6), pages 625-645, November.
    11. Peyman Khorsand & Alireza Soltani, 2017. "Optimal structure of metaplasticity for adaptive learning," PLOS Computational Biology, Public Library of Science, vol. 13(6), pages 1-22, June.
    12. Timothy E. J. Behrens & Laurence T. Hunt & Mark W. Woolrich & Matthew F. S. Rushworth, 2008. "Associative learning of social value," Nature, Nature, vol. 456(7219), pages 245-249, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ondrej Zika & Katja Wiech & Andrea Reinecke & Michael Browning & Nicolas W. Schuck, 2023. "Trait anxiety is associated with hidden state inference during aversive reversal learning," Nature Communications, Nature, vol. 14(1), pages 1-16, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Payam Piray & Nathaniel D Daw, 2020. "A simple model for learning in volatile environments," PLOS Computational Biology, Public Library of Science, vol. 16(7), pages 1-26, July.
    2. Andreea O Diaconescu & Christoph Mathys & Lilian A E Weber & Jean Daunizeau & Lars Kasper & Ekaterina I Lomakina & Ernst Fehr & Klaas E Stephan, 2014. "Inferring on the Intentions of Others by Hierarchical Bayesian Learning," PLOS Computational Biology, Public Library of Science, vol. 10(9), pages 1-19, September.
    3. Giovanni Leone & Charlotte Postel & Alison Mary & Florence Fraisse & Thomas Vallée & Fausto Viader & Vincent Sayette & Denis Peschanski & Jaques Dayan & Francis Eustache & Pierre Gagnepain, 2022. "Altered predictive control during memory suppression in PTSD," Nature Communications, Nature, vol. 13(1), pages 1-16, December.
    4. Benjamin Skerritt-Davis & Mounya Elhilali, 2018. "Detecting change in stochastic sound sequences," PLOS Computational Biology, Public Library of Science, vol. 14(5), pages 1-24, May.
    5. Robert C Wilson & Yael Niv, 2015. "Is Model Fitting Necessary for Model-Based fMRI?," PLOS Computational Biology, Public Library of Science, vol. 11(6), pages 1-21, June.
    6. Marie Devaine & Guillaume Hollard & Jean Daunizeau, 2014. "The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn?," PLOS Computational Biology, Public Library of Science, vol. 10(12), pages 1-14, December.
    7. Yongping Bao & Ludwig Danwitz & Fabian Dvorak & Sebastian Fehrler & Lars Hornuf & Hsuan Yu Lin & Bettina von Helversen, 2022. "Similarity and Consistency in Algorithm-Guided Exploration," CESifo Working Paper Series 10188, CESifo.
    8. Corgnet, Brice & Hernán-González, Roberto & Kujal, Praveen, 2020. "On booms that never bust: Ambiguity in experimental asset markets with bubbles," Journal of Economic Dynamics and Control, Elsevier, vol. 110(C).
    9. Jacqueline Scholl & Nils Kolling & Natalie Nelissen & Michael Browning & Matthew F S Rushworth & Catherine J Harmer, 2017. "Beyond negative valence: 2-week administration of a serotonergic antidepressant enhances both reward and effort learning signals," PLOS Biology, Public Library of Science, vol. 15(2), pages 1-30, February.
    10. Shi, Yuwei & Herniman, John, 2023. "The role of expectation in innovation evolution: Exploring hype cycles," Technovation, Elsevier, vol. 119(C).
    11. Sashittal, Hemant C. & Sriramachandramurthy, Rajendran & Hodis, Monica, 2012. "Targeting college students on Facebook? How to stop wasting your money," Business Horizons, Elsevier, vol. 55(5), pages 495-507.
    12. Micha Heilbron & Florent Meyniel, 2019. "Confidence resets reveal hierarchical adaptive learning in humans," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-24, April.
    13. repec:cup:judgdm:v:16:y:2021:i:6:p:1413-1438 is not listed on IDEAS
    14. Peter S. Riefer & Bradley C. Love, 2015. "Unfazed by Both the Bull and Bear: Strategic Exploration in Dynamic Environments," Games, MDPI, vol. 6(3), pages 1-11, August.
    15. Makoto Naruse & Eiji Yamamoto & Takashi Nakao & Takuma Akimoto & Hayato Saigo & Kazuya Okamura & Izumi Ojima & Georg Northoff & Hirokazu Hori, 2018. "Why is the environment important for decision making? Local reservoir model for choice-based learning," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
    16. Jacqueline N. Zadelaar & Joost A. Agelink van Rentergem & Jessica V. Schaaf & Tycho J. Dekkers & Nathalie de Vent & Laura M. S. Dekkers & Maria C. Olthof & Brenda R. J. Jansen & Hilde M. Huizenga, 2021. "Development of decision making based on internal and external information: A hierarchical Bayesian approach," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 16(6), pages 1413-1438, November.
    17. Holger Mohr & Katharina Zwosta & Dimitrije Markovic & Sebastian Bitzer & Uta Wolfensteller & Hannes Ruge, 2018. "Deterministic response strategies in a trial-and-error learning task," PLOS Computational Biology, Public Library of Science, vol. 14(11), pages 1-19, November.
    18. Mel W Khaw & Luminita Stevens & Michael Woodford, 2021. "Individual differences in the perception of probability," PLOS Computational Biology, Public Library of Science, vol. 17(4), pages 1-25, April.
    19. Dimitrije Marković & Jan Gläscher & Peter Bossaerts & John O’Doherty & Stefan J Kiebel, 2015. "Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism," PLOS Computational Biology, Public Library of Science, vol. 11(10), pages 1-34, October.
    20. Christina Fang & Daniel Levinthal, 2009. "Near-Term Liability of Exploitation: Exploration and Exploitation in Multistage Problems," Organization Science, INFORMS, vol. 20(3), pages 538-551, June.
    21. Carolyn Yoon & Richard Gonzalez & Antoine Bechara & Gregory Berns & Alain Dagher & Laurette Dubé & Scott Huettel & Joseph Kable & Israel Liberzon & Hilke Plassmann & Ale Smidts & Charles Spence, 2012. "Decision neuroscience and consumer decision making," Marketing Letters, Springer, vol. 23(2), pages 473-485, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:12:y:2021:i:1:d:10.1038_s41467-021-26731-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.