IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0270667.html
   My bibliography  Save this article

The contribution of object identity and configuration to scene representation in convolutional neural networks

Author

Listed:
  • Kevin Tang
  • Matthew Chin
  • Marvin Chun
  • Yaoda Xu

Abstract

Scene perception involves extracting the identities of the objects comprising a scene in conjunction with their configuration (the spatial layout of the objects in the scene). How object identity and configuration information is weighted during scene processing and how this weighting evolves over the course of scene processing however, is not fully understood. Recent developments in convolutional neural networks (CNNs) have demonstrated their aptitude at scene processing tasks and identified correlations between processing in CNNs and in the human brain. Here we examined four CNN architectures (Alexnet, Resnet18, Resnet50, Densenet161) and their sensitivity to changes in object and configuration information over the course of scene processing. Despite differences among the four CNN architectures, across all CNNs, we observed a common pattern in the CNN’s response to object identity and configuration changes. Each CNN demonstrated greater sensitivity to configuration changes in early stages of processing and stronger sensitivity to object identity changes in later stages. This pattern persists regardless of the spatial structure present in the image background, the accuracy of the CNN in classifying the scene, and even the task used to train the CNN. Importantly, CNNs’ sensitivity to a configuration change is not the same as their sensitivity to any type of position change, such as that induced by a uniform translation of the objects without a configuration change. These results provide one of the first documentations of how object identity and configuration information are weighted in CNNs during scene processing.

Suggested Citation

  • Kevin Tang & Matthew Chin & Marvin Chun & Yaoda Xu, 2022. "The contribution of object identity and configuration to scene representation in convolutional neural networks," PLOS ONE, Public Library of Science, vol. 17(6), pages 1-26, June.
  • Handle: RePEc:plo:pone00:0270667
    DOI: 10.1371/journal.pone.0270667
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0270667
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0270667&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0270667?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Thomas P. O’Connell & Marvin M. Chun, 2018. "Predicting eye movement patterns from fMRI responses to natural scenes," Nature Communications, Nature, vol. 9(1), pages 1-15, December.
    2. Yaoda Xu & Maryam Vaziri-Pashkam, 2021. "Publisher Correction: Limits to visual representational correspondence between convolutional neural networks and the human brain," Nature Communications, Nature, vol. 12(1), pages 1-1, December.
    3. Michael F Bonner & Russell A Epstein, 2018. "Computational mechanisms underlying cortical responses to the affordance properties of visual scenes," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-31, April.
    4. Yaoda Xu & Maryam Vaziri-Pashkam, 2021. "Limits to visual representational correspondence between convolutional neural networks and the human brain," Nature Communications, Nature, vol. 12(1), pages 1-16, December.
    5. Russell Epstein & Nancy Kanwisher, 1998. "A cortical representation of the local visual environment," Nature, Nature, vol. 392(6676), pages 598-601, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marisa Nordt & Jesse Gomez & Vaidehi S. Natu & Alex A. Rezai & Dawn Finzi & Holly Kular & Kalanit Grill-Spector, 2023. "Longitudinal development of category representations in ventral temporal cortex predicts word and face recognition," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    2. Vasiliki Bougou & Michaël Vanhoyland & Alexander Bertrand & Wim Paesschen & Hans Op De Beeck & Peter Janssen & Tom Theys, 2024. "Neuronal tuning and population representations of shape and category in human visual cortex," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    3. Ying Wang & Xue Zhang & Chunhui Wang & Weifen Huang & Qian Xu & Dong Liu & Wen Zhou & Shanguang Chen & Yi Jiang, 2022. "Modulation of biological motion perception in humans by gravity," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    4. Joel Z Leibo & Qianli Liao & Fabio Anselmi & Tomaso Poggio, 2015. "The Invariance Hypothesis Implies Domain-Specific Regions in Visual Cortex," PLOS Computational Biology, Public Library of Science, vol. 11(10), pages 1-29, October.
    5. Isabella C. Wagner & Luise P. Graichen & Boryana Todorova & Andre Lüttig & David B. Omer & Matthias Stangl & Claus Lamm, 2023. "Entorhinal grid-like codes and time-locked network dynamics track others navigating through space," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    6. repec:plo:pone00:0058594 is not listed on IDEAS
    7. repec:plo:pone00:0010556 is not listed on IDEAS
    8. Marcelo G Mattar & Michael W Cole & Sharon L Thompson-Schill & Danielle S Bassett, 2015. "A Functional Cartography of Cognitive Systems," PLOS Computational Biology, Public Library of Science, vol. 11(12), pages 1-26, December.
    9. Guohua Shen & Tomoyasu Horikawa & Kei Majima & Yukiyasu Kamitani, 2019. "Deep image reconstruction from human brain activity," PLOS Computational Biology, Public Library of Science, vol. 15(1), pages 1-23, January.
    10. Batrancea Larissa, 2021. "Research Insights From Cognitive Neuroscience For Everyday Economists," Annals - Economy Series, Constantin Brancusi University, Faculty of Economics, vol. 2, pages 35-41, April.
    11. Zhou, Lixing & Takane, Yoshio & Hwang, Heungsun, 2016. "Dynamic GSCANO (Generalized Structured Canonical Correlation Analysis) with applications to the analysis of effective connectivity in functional neuroimaging data," Computational Statistics & Data Analysis, Elsevier, vol. 101(C), pages 93-109.
    12. Mengna Yao & Bincheng Wen & Mingpo Yang & Jiebin Guo & Haozhou Jiang & Chao Feng & Yilei Cao & Huiguang He & Le Chang, 2023. "High-dimensional topographic organization of visual features in the primate temporal lobe," Nature Communications, Nature, vol. 14(1), pages 1-23, December.
    13. Michael F Bonner & Russell A Epstein, 2018. "Computational mechanisms underlying cortical responses to the affordance properties of visual scenes," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-31, April.
    14. repec:plo:pone00:0049537 is not listed on IDEAS
    15. Yujia Zhai & Binbin Fan & Jingyao Yu & Ruoyu Gong & Jie Yin, 2024. "Effects of Spatial Type and Scale of Small Urban Open Spaces on Perceived Restoration: An Online Survey-Based Experiment," Land, MDPI, vol. 13(9), pages 1-17, August.
    16. Samy A. Abdel-Ghaffar & Alexander G. Huth & Mark D. Lescroart & Dustin Stansbury & Jack L. Gallant & Sonia J. Bishop, 2024. "Occipital-temporal cortical tuning to semantic and affective features of natural images predicts associated behavioral responses," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    17. Krisztina Nagy & Mark W Greenlee & Gyula Kovács, 2011. "Sensory Competition in the Face Processing Areas of the Human Brain," PLOS ONE, Public Library of Science, vol. 6(9), pages 1-12, September.
    18. repec:plo:pone00:0214997 is not listed on IDEAS
    19. Kai J Miller & Gerwin Schalk & Dora Hermes & Jeffrey G Ojemann & Rajesh P N Rao, 2016. "Spontaneous Decoding of the Timing and Content of Human Object Perception from Cortical Surface Recordings Reveals Complementary Information in the Event-Related Potential and Broadband Spectral Chang," PLOS Computational Biology, Public Library of Science, vol. 12(1), pages 1-20, January.
    20. Ping‐Shou Zhong & Jun Li & Piotr Kokoszka, 2021. "Multivariate analysis of variance and change points estimation for high‐dimensional longitudinal data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 48(2), pages 375-405, June.
    21. Haider Al-Tahan & Yalda Mohsenzadeh, 2021. "Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder," PLOS Computational Biology, Public Library of Science, vol. 17(3), pages 1-19, March.
    22. István Czigler & Helene Intraub & Gábor Stefanics, 2013. "Prediction Beyond the Borders: ERP Indices of Boundary Extension-Related Error," PLOS ONE, Public Library of Science, vol. 8(9), pages 1-1, September.
    23. Jeongho Park & Edward Soucy & Jennifer Segawa & Ross Mair & Talia Konkle, 2024. "Immersive scene representation in human visual cortex with ultra-wide-angle neuroimaging," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    24. Benjamin Lahner & Kshitij Dwivedi & Polina Iamshchinina & Monika Graumann & Alex Lascelles & Gemma Roig & Alessandro Thomas Gifford & Bowen Pan & SouYoung Jin & N. Apurva Ratan Murty & Kendrick Kay & , 2024. "Modeling short visual events through the BOLD moments video fMRI dataset and metadata," Nature Communications, Nature, vol. 15(1), pages 1-26, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0270667. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.