IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i19p4192-d1255073.html
   My bibliography  Save this article

Self-Organizing Memory Based on Adaptive Resonance Theory for Vision and Language Navigation

Author

Listed:
  • Wansen Wu

    (College of Systems Engineering, National University of Defense Technology, Changsha 410073, China)

  • Yue Hu

    (College of Systems Engineering, National University of Defense Technology, Changsha 410073, China)

  • Kai Xu

    (College of Systems Engineering, National University of Defense Technology, Changsha 410073, China)

  • Long Qin

    (College of Systems Engineering, National University of Defense Technology, Changsha 410073, China)

  • Quanjun Yin

    (College of Systems Engineering, National University of Defense Technology, Changsha 410073, China)

Abstract

Vision and Language Navigation (VLN) is a task in which an agent needs to understand natural language instructions to reach the target location in a real-scene environment. To improve the model ability of long-horizon planning, emerging research focuses on extending the models with different types of memory structures, mainly including topological maps or a hidden state vector. However, the fixed-length hidden state vector is often insufficient to capture long-term temporal context. In comparison, topological maps have been shown to be beneficial for many robotic navigation tasks. Therefore, we focus on building a feasible and effective topological map representation and using it to improve the navigation performance and the generalization across seen and unseen environments. This paper presents a S elf-organizing Memory based on Adaptive Resonance Theory (SMART) module for incremental topological mapping and a framework for utilizing the SMART module to guide navigation. Based on fusion adaptive resonance theory networks, the SMART module can extract salient scenes from historical observations and build a topological map of the environmental layout. It provides a compact spatial representation and supports the discovery of novel shortcuts through inferences while being explainable in terms of cognitive science. Furthermore, given a language instruction and on top of the topological map, we propose a vision–language alignment framework for navigational decision-making. Notably, the framework utilizes three off-the-shelf pre-trained models to perform landmark extraction, node–landmark matching, and low-level controlling, without any fine-tuning on human-annotated datasets. We validate our approach using the Habitat simulator on VLN-CE tasks, which provides a photo-realistic environment for the embodied agent in continuous action space. The experimental results demonstrate that our approach achieves comparable performance to the supervised baseline.

Suggested Citation

  • Wansen Wu & Yue Hu & Kai Xu & Long Qin & Quanjun Yin, 2023. "Self-Organizing Memory Based on Adaptive Resonance Theory for Vision and Language Navigation," Mathematics, MDPI, vol. 11(19), pages 1-19, October.
  • Handle: RePEc:gam:jmathe:v:11:y:2023:i:19:p:4192-:d:1255073
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/19/4192/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/11/19/4192/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:19:p:4192-:d:1255073. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.