IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1003035.html
   My bibliography  Save this article

Learning Multisensory Integration and Coordinate Transformation via Density Estimation

Author

Listed:
  • Joseph G Makin
  • Matthew R Fellows
  • Philip N Sabes

Abstract

Sensory processing in the brain includes three key operations: multisensory integration—the task of combining cues into a single estimate of a common underlying stimulus; coordinate transformations—the change of reference frame for a stimulus (e.g., retinotopic to body-centered) effected through knowledge about an intervening variable (e.g., gaze position); and the incorporation of prior information. Statistically optimal sensory processing requires that each of these operations maintains the correct posterior distribution over the stimulus. Elements of this optimality have been demonstrated in many behavioral contexts in humans and other animals, suggesting that the neural computations are indeed optimal. That the relationships between sensory modalities are complex and plastic further suggests that these computations are learned—but how? We provide a principled answer, by treating the acquisition of these mappings as a case of density estimation, a well-studied problem in machine learning and statistics, in which the distribution of observed data is modeled in terms of a set of fixed parameters and a set of latent variables. In our case, the observed data are unisensory-population activities, the fixed parameters are synaptic connections, and the latent variables are multisensory-population activities. In particular, we train a restricted Boltzmann machine with the biologically plausible contrastive-divergence rule to learn a range of neural computations not previously demonstrated under a single approach: optimal integration; encoding of priors; hierarchical integration of cues; learning when not to integrate; and coordinate transformation. The model makes testable predictions about the nature of multisensory representations.Author Summary: Over the first few years of their lives, humans (and other animals) appear to learn how to combine signals from multiple sense modalities: when to “integrate” them into a single percept, as with visual and proprioceptive information about one's body; when not to integrate them (e.g., when looking somewhere else); how they vary over longer time scales (e.g., where in physical space my hand tends to be); as well as more complicated manipulations, like subtracting gaze angle from the visually-perceived position of an object to compute the position of that object with respect to the head—i.e., “coordinate transformation.” Learning which sensory signals to integrate, or which to manipulate in other ways, does not appear to require an additional supervisory signal; we learn to do so, rather, based on structure in the sensory signals themselves. We present a biologically plausible artificial neural network that learns all of the above in just this way, but by training it for a much more general statistical task: “density estimation”—essentially, learning to be able to reproduce the data on which it was trained. This also links coordinate transformation and multisensory integration to other cortical operations, especially in early sensory areas, that have have been modeled as density estimators.

Suggested Citation

  • Joseph G Makin & Matthew R Fellows & Philip N Sabes, 2013. "Learning Multisensory Integration and Coordinate Transformation via Density Estimation," PLOS Computational Biology, Public Library of Science, vol. 9(4), pages 1-17, April.
  • Handle: RePEc:plo:pcbi00:1003035
    DOI: 10.1371/journal.pcbi.1003035
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003035
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1003035&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1003035?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Marc O. Ernst & Martin S. Banks, 2002. "Humans integrate visual and haptic information in a statistically optimal fashion," Nature, Nature, vol. 415(6870), pages 429-433, January.
    2. Jan Eichhorn & Fabian Sinz & Matthias Bethge, 2009. "Natural Image Coding in V1: How Much Use Is Orientation Selectivity?," PLOS Computational Biology, Public Library of Science, vol. 5(4), pages 1-16, April.
    3. Jean-René Duhamel & Frank Bremmer & Suliann Ben Hamed & Werner Graf, 1997. "Spatial invariance of visual receptive fields in parietal cortex neurons," Nature, Nature, vol. 389(6653), pages 845-848, October.
    4. Konrad P. Körding & Daniel M. Wolpert, 2004. "Bayesian integration in sensorimotor learning," Nature, Nature, vol. 427(6971), pages 244-247, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Joseph G Makin & Benjamin K Dichter & Philip N Sabes, 2015. "Learning to Estimate Dynamical State with Probabilistic Population Codes," PLOS Computational Biology, Public Library of Science, vol. 11(11), pages 1-28, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wen-Hao Zhang & Si Wu & Krešimir Josić & Brent Doiron, 2023. "Sampling-based Bayesian inference in recurrent circuits of stochastic spiking neurons," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    2. Adam N Sanborn & Ulrik R Beierholm, 2016. "Fast and Accurate Learning When Making Discrete Numerical Estimates," PLOS Computational Biology, Public Library of Science, vol. 12(4), pages 1-28, April.
    3. Seth W. Egger & Stephen G. Lisberger, 2022. "Neural structure of a sensory decoder for motor control," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    4. Tim Genewein & Eduard Hez & Zeynab Razzaghpanah & Daniel A Braun, 2015. "Structure Learning in Bayesian Sensorimotor Integration," PLOS Computational Biology, Public Library of Science, vol. 11(8), pages 1-27, August.
    5. Brocas, Isabelle & Carrillo, Juan D., 2012. "From perception to action: An economic model of brain processes," Games and Economic Behavior, Elsevier, vol. 75(1), pages 81-103.
    6. Carrillo, Juan & Brocas, Isabelle, 2007. "Reason, Emotion and Information Processing in the Brain," CEPR Discussion Papers 6535, C.E.P.R. Discussion Papers.
    7. Jannes Jegminat & Maya A Jastrzębowska & Matthew V Pachai & Michael H Herzog & Jean-Pascal Pfister, 2020. "Bayesian regression explains how human participants handle parameter uncertainty," PLOS Computational Biology, Public Library of Science, vol. 16(5), pages 1-23, May.
    8. Guido Marco Cicchini & Giovanni D’Errico & David Charles Burr, 2022. "Crowding results from optimal integration of visual targets with contextual information," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    9. Udo A Ernst & Sunita Mandon & Nadja Schinkel–Bielefeld & Simon D Neitzel & Andreas K Kreiter & Klaus R Pawelzik, 2012. "Optimality of Human Contour Integration," PLOS Computational Biology, Public Library of Science, vol. 8(5), pages 1-17, May.
    10. Philipp Schustek & Rubén Moreno-Bote, 2018. "Instance-based generalization for human judgments about uncertainty," PLOS Computational Biology, Public Library of Science, vol. 14(6), pages 1-27, June.
    11. Edward J A Turnham & Daniel A Braun & Daniel M Wolpert, 2011. "Inferring Visuomotor Priors for Sensorimotor Learning," PLOS Computational Biology, Public Library of Science, vol. 7(3), pages 1-13, March.
    12. Jingwei Sun & Jian Li & Hang Zhang, 2019. "Human representation of multimodal distributions as clusters of samples," PLOS Computational Biology, Public Library of Science, vol. 15(5), pages 1-29, May.
    13. Jordi Grau-Moya & Pedro A Ortega & Daniel A Braun, 2012. "Risk-Sensitivity in Bayesian Sensorimotor Integration," PLOS Computational Biology, Public Library of Science, vol. 8(9), pages 1-7, September.
    14. Laurence Aitchison & Máté Lengyel, 2016. "The Hamiltonian Brain: Efficient Probabilistic Inference with Excitatory-Inhibitory Neural Circuit Dynamics," PLOS Computational Biology, Public Library of Science, vol. 12(12), pages 1-24, December.
    15. Luigi Acerbi & Daniel M Wolpert & Sethu Vijayakumar, 2012. "Internal Representations of Temporal Statistics and Feedback Calibrate Motor-Sensory Interval Timing," PLOS Computational Biology, Public Library of Science, vol. 8(11), pages 1-19, November.
    16. Shih-Wei Wu & Maria F Dal Martello & Laurence T Maloney, 2009. "Sub-Optimal Allocation of Time in Sequential Movements," PLOS ONE, Public Library of Science, vol. 4(12), pages 1-13, December.
    17. Simon Weiler & Vahid Rahmati & Marcel Isstas & Johann Wutke & Andreas Walter Stark & Christian Franke & Jürgen Graf & Christian Geis & Otto W. Witte & Mark Hübener & Jürgen Bolz & Troy W. Margrie & Kn, 2024. "A primary sensory cortical interareal feedforward inhibitory circuit for tacto-visual integration," Nature Communications, Nature, vol. 15(1), pages 1-24, December.
    18. Leopold Zizlsperger & Thomas Sauvigny & Thomas Haarmeier, 2012. "Selective Attention Increases Choice Certainty in Human Decision Making," PLOS ONE, Public Library of Science, vol. 7(7), pages 1-9, July.
    19. Catarina Mendonça & Pietro Mandelli & Ville Pulkki, 2016. "Modeling the Perception of Audiovisual Distance: Bayesian Causal Inference and Other Models," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-18, December.
    20. Geonhui Lee & Woong Choi & Hanjin Jo & Wookhyun Park & Jaehyo Kim, 2020. "Analysis of motor control strategy for frontal and sagittal planes of circular tracking movements using visual feedback noise from velocity change and depth information," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-22, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1003035. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.