IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1007475.html
   My bibliography  Save this article

Tonic exploration governs both flexibility and lapses

Author

Listed:
  • R Becket Ebitz
  • Brianna J Sleezer
  • Hank P Jedema
  • Charles W Bradberry
  • Benjamin Y Hayden

Abstract

In many cognitive tasks, lapses (spontaneous errors) are tacitly dismissed as the result of nuisance processes like sensorimotor noise, fatigue, or disengagement. However, some lapses could also be caused by exploratory noise: randomness in behavior that facilitates learning in changing environments. If so, then strategic processes would need only up-regulate (rather than generate) exploration to adapt to a changing environment. This view predicts that more frequent lapses should be associated with greater flexibility because these behaviors share a common cause. Here, we report that when rhesus macaques performed a set-shifting task, lapse rates were negatively correlated with perseverative error frequency across sessions, consistent with a common basis in exploration. The results could not be explained by local failures to learn. Furthermore, chronic exposure to cocaine, which is known to impair cognitive flexibility, did increase perseverative errors, but, surprisingly, also improved overall set-shifting task performance by reducing lapse rates. We reconcile these results with a state-switching model in which cocaine decreases exploration by deepening attractor basins corresponding to rule states. These results support the idea that exploratory noise contributes to lapses, affecting rule-based decision-making even when it has no strategic value, and suggest that one key mechanism for regulating exploration may be the depth of rule states.Author summary: Why do we make mistakes? We seem to have the capacity to identify the best course of action, but we do not always choose it. Here, we report that at least some mistakes are due to exploration—a type of decision-making that is focused on discovery and learning, rather than on choosing the best option. This is surprising because many views of exploration assume that exploration only happens phasically—when the circumstances suggest that you should abandon your previous course of action and make a new plan. However, here, we find evidence that exploration drives decisions to change your behavior both when change is helpful and when it is a mistake. More work is needed to understand why we explore tonically, but it is possible that tonic exploration may been so useful over evolutionary time that our brains evolved to continue to explore today, even when it has no strategic benefit in the moment. For example, a tonic algorithm for exploration could reduce the effort required to make decisions or prepare us to take advantage of unexpected opportunities.

Suggested Citation

  • R Becket Ebitz & Brianna J Sleezer & Hank P Jedema & Charles W Bradberry & Benjamin Y Hayden, 2019. "Tonic exploration governs both flexibility and lapses," PLOS Computational Biology, Public Library of Science, vol. 15(11), pages 1-37, November.
    • Handle: RePEc:plo:pcbi00:1007475
    DOI: 10.1371/journal.pcbi.1007475
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007475
    Download Restriction: no
    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1007475&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pcbi.1007475?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Shiva Farashahi & Katherine Rowe & Zohra Aslami & Daeyeol Lee & Alireza Soltani, 2017. "Feature-based learning improves adaptability without compromising precision," Nature Communications, Nature, vol. 8(1), pages 1-16, December.
    2. Trevor W. Robbins & Barry J. Everitt, 1999. "Drug addiction: bad habits add up," Nature, Nature, vol. 398(6728), pages 567-570, April.
    3. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    4. Jonathan D. Wallis & Kathleen C. Anderson & Earl K. Miller, 2001. "Single neurons in prefrontal cortex encode abstract rules," Nature, Nature, vol. 411(6840), pages 953-956, June.
    5. Nathaniel D. Daw & John P. O'Doherty & Peter Dayan & Ben Seymour & Raymond J. Dolan, 2006. "Cortical substrates for exploratory decisions in humans," Nature, Nature, vol. 441(7095), pages 876-879, June.
    6. Bruno B Averbeck, 2015. "Theory of Choice in Bandit, Information Sampling and Foraging Tasks," PLOS Computational Biology, Public Library of Science, vol. 11(3), pages 1-28, March.
    7. Nuo Li & Kayvon Daie & Karel Svoboda & Shaul Druckmann, 2016. "Robust neuronal dynamics in premotor cortex during motor planning," Nature, Nature, vol. 532(7600), pages 459-464, April.
    8. Germain Lefebvre & Maël Lebreton & Florent Meyniel & Sacha Bourgeois-Gironde & Stefano Palminteri, 2017. "Behavioural and neural characterization of optimistic reinforcement learning," Nature Human Behaviour, Nature, vol. 1(4), pages 1-9, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ayaka Kato & Kenji Morita, 2016. "Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation," PLOS Computational Biology, Public Library of Science, vol. 12(10), pages 1-41, October.
    2. Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
    3. Hu Sun & Yun Wang, 2019. "Do On-lookers See Most of the Game? Evaluating Job-seekers' Competitiveness of Oneself versus of Others in a Labor Market Experiment," Working Papers 2019-07-11, Wang Yanan Institute for Studies in Economics (WISE), Xiamen University.
    4. Daniel Bennett & Stefan Bode & Maja Brydevall & Hayley Warren & Carsten Murawski, 2016. "Intrinsic Valuation of Information in Decision Making under Uncertainty," PLOS Computational Biology, Public Library of Science, vol. 12(7), pages 1-21, July.
    5. Tal Neiman & Yonatan Loewenstein, 2011. "Reinforcement learning in professional basketball players," Discussion Paper Series dp593, The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem.
    6. Phanish Puranam & Murali Swamy, 2016. "How Initial Representations Shape Coupled Learning Processes," Organization Science, INFORMS, vol. 27(2), pages 323-335, April.
    7. Aurélien Nioche & Basile Garcia & Germain Lefebvre & Thomas Boraud & Nicolas P. Rougier & Sacha Bourgeois-Gironde, 2019. "Coordination over a unique medium of exchange under information scarcity," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-11, December.
    8. Lieke L F van Lieshout & Iris J Traast & Floris P de Lange & Roshan Cools, 2021. "Curiosity or savouring? Information seeking is modulated by both uncertainty and valence," PLOS ONE, Public Library of Science, vol. 16(9), pages 1-19, September.
    9. Daniel E Acuña & Paul Schrater, 2010. "Structure Learning in Human Sequential Decision-Making," PLOS Computational Biology, Public Library of Science, vol. 6(12), pages 1-12, December.
    10. Alina Ferecatu & Arnaud De Bruyn, 2022. "Understanding Managers’ Trade-Offs Between Exploration and Exploitation," Marketing Science, INFORMS, vol. 41(1), pages 139-165, January.
    11. Noah Gans & George Knox & Rachel Croson, 2007. "Simple Models of Discrete Choice and Their Performance in Bandit Experiments," Manufacturing & Service Operations Management, INFORMS, vol. 9(4), pages 383-408, December.
    12. Terry E. Daniel & Eyran J. Gisches & Amnon Rapoport, 2009. "Departure Times in Y-Shaped Traffic Networks with Multiple Bottlenecks," American Economic Review, American Economic Association, vol. 99(5), pages 2149-2176, December.
    13. Iftekhar, M. S. & Tisdell, J. G., 2018. "Learning in repeated multiple unit combinatorial auctions: An experimental study," Working Papers 267301, University of Western Australia, School of Agricultural and Resource Economics.
    14. Ianni, A., 2002. "Reinforcement learning and the power law of practice: some analytical results," Discussion Paper Series In Economics And Econometrics 203, Economics Division, School of Social Sciences, University of Southampton.
    15. Yongping Bao & Ludwig Danwitz & Fabian Dvorak & Sebastian Fehrler & Lars Hornuf & Hsuan Yu Lin & Bettina von Helversen, 2022. "Similarity and Consistency in Algorithm-Guided Exploration," CESifo Working Paper Series 10188, CESifo.
    16. Chih-Chung Ting & Nahuel Salem-Garcia & Stefano Palminteri & Jan B. Engelmann & Maël Lebreton, 2023. "Neural and computational underpinnings of biased confidence in human reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    17. Benaïm, Michel & Hofbauer, Josef & Hopkins, Ed, 2009. "Learning in games with unstable equilibria," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1694-1709, July.
    18. Oechssler, Jorg & Schipper, Burkhard, 2003. "Can you guess the game you are playing?," Games and Economic Behavior, Elsevier, vol. 43(1), pages 137-152, April.
    19. Erhao Xie, 2019. "Monetary Payoff and Utility Function in Adaptive Learning Models," Staff Working Papers 19-50, Bank of Canada.
    20. B Kelsey Jack, 2009. "Auctioning Conservation Contracts in Indonesia - Participant Learning in Multiple Trial Rounds," CID Working Papers 35, Center for International Development at Harvard University.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1007475. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.