Noisy models of the ventral stream reveal the impact of recurrence and learned representations on information processing timescales

In vision neuroscience, the temporal dynamics of the sensory stream and of its neural representations are thought to be deeply linked to the function of the hierarchy of cortical areas that deal with object recognition, known as the visual ventral stream. Neural representations that are invariant under identity-preserving object transformations, and therefore allow for efficient learning of object identity, are theorized to emerge from a self-supervised learning process that attempts to extract “temporally stable” features from the sensory input. Conversely, invariance increases along the hierarchy, putatively implying progressively slower neural codes in higher-level areas. Recent neurophysiological evidence shows that indeed, as one moves along this cortical hierarchy, neural representations of dynamic stimuli become slower, and additionally the temporal scales of the within-trial fluctuations of these representations (called “intrinsic timescales”) increase starkly. However, while these timescale hierarchies have been reproduced in biologically grounded recurrent models, their network determinants have remained largely unexplored in image-computable models of the ventral stream. Here we investigate the temporal structure of the neural codes in a noisy, recurrent and adaptive model of the ventral visual stream. We show that, surprisingly, the organization of the representation timescales is set by the broad architectural features of the network, regardless of training, while the ordering of the intrinsic timescales across layers is sensitive to the details of the functions implemented by each layer. Our work underscores the importance of the temporal structure of the neural code as a probe for the link between structure and function in models of the vertebrate visual system.Author summary: Making sense of a constantly changing visual world requires the brain to integrate information over time. As visual signals travel through the hierarchy of cortical areas that support object recognition, neural representations are thought to become progressively more stable in time. Recent experiments have confirmed this, showing that both the timescales of stimulus-driven responses and those of the fluctuations around average responses (the “intrinsic timescales”) grow along the hierarchy. Yet the artificial neural networks most commonly used to model vision are static and cannot capture these dynamics. Here we built a family of biologically inspired convolutional–recurrent networks that process movies while incorporating noise, recurrence, and adaptation. By introducing these ingredients one at a time, we asked which are actually needed to reproduce the experimentally observed hierarchy of timescales. We found that the ordering of response timescales depends chiefly on the broad architecture of the network, even in untrained (random) networks. In contrast, the hierarchy of intrinsic timescales is far more fragile: it requires both slow internal dynamics and representations shaped by learning. Our results suggest that intrinsic fluctuations are an informative and underused benchmark for computational models of the visual system.