Shape Similarity, Better than Semantic Membership, Accounts for the Structure of Visual Object Representations in a Population of Monkey Inferotemporal Neurons

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex.Author Summary: To build meaningful representations of the external word, the stream of sensory information that reaches our senses is continuously processed and interpreted by the brain. Ultimately, such a processing allows the brain to arrange sensory (e.g., visual) inputs into a hierarchy of categories (such as animate and inanimate objects) and sub-categories (such as faces, animals, buildings, tools, etc). Crucially, while many objects can be assigned to the same category based on their visual similarity (e.g., oranges and apples), formation of most categories also requires arbitrarily associating objects sharing similar functions/meaning, but not similar shape (e.g., bananas and apples). A long-standing debate exists about whether the representation of visual objects in the higher visual centers of the brain (such as the inferotemporal cortex; IT) purely reflects shape similarity or also (and, perhaps, mainly) shape-unrelated categorical knowledge. In this study, we have addressed this issue by applying a variety of computational approaches. Our results show that the response patterns of a population of inferotemporal neurons are better accounted for by shape similarity than categorical membership. This reasserts the primary function of IT as a visual area and demonstrates how state-of-the-art computational approaches can advance our understanding of neuronal coding in the brain.