IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-38674-4.html
   My bibliography  Save this article

Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations

Author

Listed:
  • Ghislain St-Yves

    (University of Minnesota
    University of Minnesota)

  • Emily J. Allen

    (University of Minnesota)

  • Yihan Wu

    (University of Minnesota)

  • Kendrick Kay

    (University of Minnesota
    University of Minnesota)

  • Thomas Naselaris

    (University of Minnesota
    University of Minnesota)

Abstract

Deep neural networks (DNNs) optimized for visual tasks learn representations that align layer depth with the hierarchy of visual areas in the primate brain. One interpretation of this finding is that hierarchical representations are necessary to accurately predict brain activity in the primate visual system. To test this interpretation, we optimized DNNs to directly predict brain activity measured with fMRI in human visual areas V1-V4. We trained a single-branch DNN to predict activity in all four visual areas jointly, and a multi-branch DNN to predict each visual area independently. Although it was possible for the multi-branch DNN to learn hierarchical representations, only the single-branch DNN did so. This result shows that hierarchical representations are not necessary to accurately predict human brain activity in V1-V4, and that DNNs that encode brain-like visual representations may differ widely in their architecture, ranging from strict serial hierarchies to multiple independent branches.

Suggested Citation

  • Ghislain St-Yves & Emily J. Allen & Yihan Wu & Kendrick Kay & Thomas Naselaris, 2023. "Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-38674-4
    DOI: 10.1038/s41467-023-38674-4
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-38674-4
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-38674-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Carsen Stringer & Marius Pachitariu & Nicholas Steinmetz & Matteo Carandini & Kenneth D. Harris, 2019. "High-dimensional geometry of population responses in visual cortex," Nature, Nature, vol. 571(7765), pages 361-365, July.
    2. K Seeliger & L Ambrogioni & Y Güçlütürk & L M van den Bulk & U Güçlü & M A J van Gerven, 2021. "End-to-end neural system identification with neural information flow," PLOS Computational Biology, Public Library of Science, vol. 17(2), pages 1-22, February.
    3. Kendrick N. Kay & Thomas Naselaris & Ryan J. Prenger & Jack L. Gallant, 2008. "Identifying natural images from human brain activity," Nature, Nature, vol. 452(7185), pages 352-355, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Umut Güçlü & Marcel A J van Gerven, 2014. "Unsupervised Feature Learning Improves Prediction of Human Brain Activity in Response to Natural Images," PLOS Computational Biology, Public Library of Science, vol. 10(8), pages 1-12, August.
    2. Guillermo A Cecchi & Lejian Huang & Javeria Ali Hashmi & Marwan Baliki & María V Centeno & Irina Rish & A Vania Apkarian, 2012. "Predictive Dynamics of Human Pain Perception," PLOS Computational Biology, Public Library of Science, vol. 8(10), pages 1-12, October.
    3. Zvi N. Roth & Kendrick Kay & Elisha P. Merriam, 2022. "Natural scene sampling reveals reliable coarse-scale orientation tuning in human V1," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    4. Hamed Nili & Cai Wingfield & Alexander Walther & Li Su & William Marslen-Wilson & Nikolaus Kriegeskorte, 2014. "A Toolbox for Representational Similarity Analysis," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-11, April.
    5. Hamed Nili & Alexander Walther & Arjen Alink & Nikolaus Kriegeskorte, 2020. "Inferring exemplar discriminability in brain representations," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-28, June.
    6. Yuke Yan & James M. Goodman & Dalton D. Moore & Sara A. Solla & Sliman J. Bensmaia, 2020. "Unexpected complexity of everyday manual behaviors," Nature Communications, Nature, vol. 11(1), pages 1-8, December.
    7. Jacob M. Paul & Martijn Ackooij & Tuomas C. Cate & Ben M. Harvey, 2022. "Numerosity tuning in human association cortices and local image contrast representations in early visual cortex," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    8. Kenneth W. Latimer & David J. Freedman, 2023. "Low-dimensional encoding of decisions in parietal cortex reflects long-term training history," Nature Communications, Nature, vol. 14(1), pages 1-24, December.
    9. Mehrabbeik, Mahtab & Shams-Ahmar, Mohammad & Levine, Alexandra T. & Jafari, Sajad & Merrikhi, Yaser, 2022. "Distinctive nonlinear dimensionality of neural spiking activity in extrastriate cortex during spatial working memory; a Higuchi fractal analysis," Chaos, Solitons & Fractals, Elsevier, vol. 158(C).
    10. Agustin Lage-Castellanos & Giancarlo Valente & Elia Formisano & Federico De Martino, 2019. "Methods for computing the maximum performance of computational models of fMRI responses," PLOS Computational Biology, Public Library of Science, vol. 15(3), pages 1-25, March.
    11. Kay H Brodersen & Thomas M Schofield & Alexander P Leff & Cheng Soon Ong & Ekaterina I Lomakina & Joachim M Buhmann & Klaas E Stephan, 2011. "Generative Embedding for Model-Based Classification of fMRI Data," PLOS Computational Biology, Public Library of Science, vol. 7(6), pages 1-19, June.
    12. Shinsuke Koyama & Uri Eden & Emery Brown & Robert Kass, 2010. "Bayesian decoding of neural spike trains," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 62(1), pages 37-59, February.
    13. Kiyohito Iigaya & Sanghyun Yi & Iman A. Wahle & Sandy Tanwisuth & Logan Cross & John P. O’Doherty, 2023. "Neural mechanisms underlying the hierarchical construction of perceived aesthetic value," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    14. Ming Bo Cai & Nicolas W Schuck & Jonathan W Pillow & Yael Niv, 2019. "Representational structure or task structure? Bias in neural representational similarity analysis and a Bayesian method for reducing bias," PLOS Computational Biology, Public Library of Science, vol. 15(5), pages 1-30, May.
    15. Jérémie Sibille & Carolin Gehr & Jonathan I. Benichov & Hymavathy Balasubramanian & Kai Lun Teh & Tatiana Lupashina & Daniela Vallentin & Jens Kremkow, 2022. "High-density electrode recordings reveal strong and specific connections between retinal ganglion cells and midbrain neurons," Nature Communications, Nature, vol. 13(1), pages 1-18, December.
    16. Hanzhong Liu & Bin Yu, 2017. "Comments on: High-dimensional simultaneous inference with the bootstrap," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 26(4), pages 740-750, December.
    17. Raheel Zafar & Sarat C Dass & Aamir Saeed Malik, 2017. "Electroencephalogram-based decoding cognitive states using convolutional neural network and likelihood ratio based score fusion," PLOS ONE, Public Library of Science, vol. 12(5), pages 1-23, May.
    18. Patrice Abry & B. Cooper Boniece & Gustavo Didier & Herwig Wendt, 2023. "Wavelet eigenvalue regression in high dimensions," Statistical Inference for Stochastic Processes, Springer, vol. 26(1), pages 1-32, April.
    19. Rong J. B. Zhu & Xue-Xin Wei, 2023. "Unsupervised approach to decomposing neural tuning variability," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    20. Yargholi, E. & Hossein-Zadeh, G.-A., 2019. "Cross recurrence quantifiers as new connectivity measures for structure learning of Bayesian networks in brain decoding," Chaos, Solitons & Fractals, Elsevier, vol. 123(C), pages 263-274.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-38674-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.