IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1010047.html
   My bibliography  Save this article

Conflict and competition between model-based and model-free control

Author

Listed:
  • Yuqing Lei
  • Alec Solway

Abstract

A large literature has accumulated suggesting that human and animal decision making is driven by at least two systems, and that important functions of these systems can be captured by reinforcement learning algorithms. The “model-free” system caches and uses stimulus–value or stimulus–response associations, and the “model-based” system implements more flexible planning using a model of the world. However, it is not clear how the two systems interact during deliberation and how a single decision emerges from this process, especially when they disagree. Most previous work has assumed that while the systems operate in parallel, they do so independently, and they combine linearly to influence decisions. Using an integrated reinforcement learning/drift-diffusion model, we tested the hypothesis that the two systems interact in a non-linear fashion similar to other situations with cognitive conflict. We differentiated two forms of conflict: action conflict, a binary state representing whether the systems disagreed on the best action, and value conflict, a continuous measure of the extent to which the two systems disagreed on the difference in value between the available options. We found that decisions with greater value conflict were characterized by reduced model-based control and increased caution both with and without action conflict. Action conflict itself (the binary state) acted in the opposite direction, although its effects were less prominent. We also found that between-system conflict was highly correlated with within-system conflict, and although it is less clear a priori why the latter might influence the strength of each system above its standard linear contribution, we could not rule it out. Our work highlights the importance of non-linear conflict effects, and provides new constraints for more detailed process models of decision making. It also presents new avenues to explore with relation to disorders of compulsivity, where an imbalance between systems has been implicated.Author summary: A number of studies have framed goal-directed and habitual decision making from the perspective of different reinforcement learning algorithms (“model-based” and “model-free”), and further suggested that they are supported by separate though potentially overlapping systems. However, there has been little work to understand how the different systems work together. By design, they will sometimes disagree on the identity of the best action, and even when they agree, they will assign different values to the actions. Despite this, the end result is a single behavioral output. The issue of how the two systems interact and compete draws parallels to the existing literature on cognitive control, where a central question has been how more ‘costly’ cognitive resources should be deployed in the presence of decision conflict (here, the goal-directed system is more computationally ‘expensive’). Across four datasets, we found that the influence of the goal-directed system was reduced as a function of conflict between systems, and in addition, responses overall were more cautious. Our results provide new constraints for process models of decision making, and suggest new research directions for questions related to psychopathology and disorders of compulsivity in particular, where an imbalance between the two systems has previously been implicated.

Suggested Citation

  • Yuqing Lei & Alec Solway, 2022. "Conflict and competition between model-based and model-free control," PLOS Computational Biology, Public Library of Science, vol. 18(5), pages 1-22, May.
  • Handle: RePEc:plo:pcbi00:1010047
    DOI: 10.1371/journal.pcbi.1010047
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010047
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1010047&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1010047?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sang Wan Lee & John P O’Doherty & Shinsuke Shimojo, 2015. "Neural Computations Mediating One-Shot Learning in the Human Brain," PLOS Biology, Public Library of Science, vol. 13(4), pages 1-36, April.
    2. Milica Milosavljevic & Jonathan Malmaud & Alexander Huth & Christof Koch & Antonio Rangel, 2010. "The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(6), pages 437-449, October.
    3. Ernst Fehr & Antonio Rangel, 2011. "Neuroeconomic Foundations of Economic Choice--Recent Advances," Journal of Economic Perspectives, American Economic Association, vol. 25(4), pages 3-30, Fall.
    4. repec:cup:judgdm:v:5:y:2010:i:6:p:437-449 is not listed on IDEAS
    5. Milosavljevic, Milica & Malmaud, Jonathan & Huth, Alexander & Koch, Christof & Rangel, Antonio, 2010. "The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure," Judgment and Decision Making, Cambridge University Press, vol. 5(6), pages 437-449, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. Silvestre, & T. Araújo & M. St. Aubyn, 2016. "Economic growth and individual satisfaction in an agent-based economy," Working Papers Department of Economics 2016/19, ISEG - Lisbon School of Economics and Management, Department of Economics, Universidade de Lisboa.
    2. José F. Tudón M., 2019. "Perception, utility, and evolution," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 7(2), pages 191-208, December.
    3. Cary Frydman & Gideon Nave, 2017. "Extrapolative Beliefs in Perceptual and Economic Decisions: Evidence of a Common Mechanism," Management Science, INFORMS, vol. 63(7), pages 2340-2352, July.
    4. Daniel Serra, 2019. "La neuroéconomie en question : débats et controverses," CEE-M Working Papers halshs-02160911, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    5. Holger Mohr & Katharina Zwosta & Dimitrije Markovic & Sebastian Bitzer & Uta Wolfensteller & Hannes Ruge, 2018. "Deterministic response strategies in a trial-and-error learning task," PLOS Computational Biology, Public Library of Science, vol. 14(11), pages 1-19, November.
    6. Yildiz, Özgür & Rommel, Jens & Debor, Sarah & Holstenkamp, Lars & Mey, Franziska & Müller, Jakob R. & Radtke, Jörg & Rognli, Judith, 2014. "Research Perspectives on Renewable Energy Cooperatives in Germany: Empirical Insights and Theoretical Lenses," MPRA Paper 55931, University Library of Munich, Germany.
    7. Claus Dierksmeier, 2020. "From Jensen to Jensen: Mechanistic Management Education or Humanistic Management Learning?," Journal of Business Ethics, Springer, vol. 166(1), pages 73-87, September.
    8. Barberis, Nicholas & Greenwood, Robin & Jin, Lawrence & Shleifer, Andrei, 2018. "Extrapolation and bubbles," Journal of Financial Economics, Elsevier, vol. 129(2), pages 203-227.
    9. Susann Fiedler & Adrian Hillenbrand, 2018. "Gain-Loss Framing in Interdependent Choice," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2018_15, Max Planck Institute for Research on Collective Goods.
    10. Hébert, Benjamin & Woodford, Michael, 2023. "Rational inattention when decisions take time," Journal of Economic Theory, Elsevier, vol. 208(C).
    11. Grossman, Zachary & van der Weele, Joël & Andrijevik, Ana, 2014. "A Test of Dual-Process Reasoning in Charitable Giving," University of California at Santa Barbara, Economics Working Paper Series qt4tm617f7, Department of Economics, UC Santa Barbara.
    12. Stephanie M. Smith & Ian Krajbich & Ryan Webb, 2019. "Estimating the dynamic role of attention via random utility," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 5(1), pages 97-111, August.
    13. Jeffrey E. Harris & Mariana Gerstenblüth & Patricia Triunfo, 2018. "Smokers’ Rational Lexicographic Preferences for Cigarette Package Warnings: A Discrete Choice Experiment with Eye Tracking," Documentos de Trabajo (working papers) 0218, Department of Economics - dECON.
    14. Gebhard Kirchgässner, 2013. "The Weak Rationality Principle in Economics," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 149(I), pages 1-26, March.
    15. Köke, Sonja & Perino, Grischa, 2017. "For "better" or "worse": a direct approach to elicit preference rankings from life-satisfaction data," WiSo-HH Working Paper Series 43, University of Hamburg, Faculty of Business, Economics and Social Sciences, WISO Research Laboratory.
    16. Ryan Webb & Paul W. Glimcher & Kenway Louie, 2021. "The Normalization of Consumer Valuations: Context-Dependent Preferences from Neurobiological Constraints," Management Science, INFORMS, vol. 67(1), pages 93-125, January.
    17. Grzegorz Drozdowski & Joanna Rogozińska-Mitrut & Jacek Stasiak, 2021. "The Empirical Analysis of the Core Competencies of the Company’s Resource Management Risk. Preliminary Study," Risks, MDPI, vol. 9(6), pages 1-12, June.
    18. Ahmet Ak & Oner Gumus, 2019. "The Possible Effects of Personal Income Tax and Value Added Tax on Consumer Behaviors," Papers 1910.03141, arXiv.org.
    19. Carlo Baldassi & Simone Cerreia-Vioglio & Fabio Maccheroni & Massimo Marinacci & Marco Pirazzini, 2020. "A Behavioral Characterization of the Drift Diffusion Model and Its Multialternative Extension for Choice Under Time Pressure," Management Science, INFORMS, vol. 66(11), pages 5075-5093, November.
    20. S. Cerreia-Vioglio & F. Maccheroni & M. Marinacci & A. Rustichini, 2017. "Multinomial logit processes and preference discovery: inside and outside the black box," Working Papers 615, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1010047. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.