A mixture of sparse coding models explaining properties of face neurons related to holistic and parts-based processing

Experimental studies have revealed evidence of both parts-based and holistic representations of objects and faces in the primate visual system. However, it is still a mystery how such seemingly contradictory types of processing can coexist within a single system. Here, we propose a novel theory called mixture of sparse coding models, inspired by the formation of category-specific subregions in the inferotemporal (IT) cortex. We developed a hierarchical network that constructed a mixture of two sparse coding submodels on top of a simple Gabor analysis. The submodels were each trained with face or non-face object images, which resulted in separate representations of facial parts and object parts. Importantly, evoked neural activities were modeled by Bayesian inference, which had a top-down explaining-away effect that enabled recognition of an individual part to depend strongly on the category of the whole input. We show that this explaining-away effect was indeed crucial for the units in the face submodel to exhibit significant selectivity to face images over object images in a similar way to actual face-selective neurons in the macaque IT cortex. Furthermore, the model explained, qualitatively and quantitatively, several tuning properties to facial features found in the middle patch of face processing in IT as documented by Freiwald, Tsao, and Livingstone (2009). These included, in particular, tuning to only a small number of facial features that were often related to geometrically large parts like face outline and hair, preference and anti-preference of extreme facial features (e.g., very large/small inter-eye distance), and reduction of the gain of feature tuning for partial face stimuli compared to whole face stimuli. Thus, we hypothesize that the coding principle of facial features in the middle patch of face processing in the macaque IT cortex may be closely related to mixture of sparse coding models.Author summary: Does the brain represent an object as a combination of parts or as a whole? Past experiments have found both types of representation; but how can such opposing notions coexist in a single visual system? Here, we introduce a novel theory called mixture of sparse coding models for investigating the possible computational principles underlying the primate visual object processing. We constructed a hierarchical network combining two sparse coding modules that each represented one feature set, of either facial parts or non-facial object parts. Competitive computation between the modules, formalized as Bayesian inference, enabled parts to be recognized with a strong top-down influence from the category of the whole input. We show that the latter computation is crucial to explain in detail neural selectivity and tuning properties that were experimentally reported for a particular face processing region called the middle patch. Thus, we offer the first theoretical account of neural face processing in relation to parts-based and holistic representations.