IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i18p3969-d1242735.html
   My bibliography  Save this article

RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets

Author

Listed:
  • Wenjun Ruan

    (School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China)

  • Wenbo Zhu

    (School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China)

  • Zhijia Zhao

    (School of Mechanical and Electrical Engineering, Guangzhou University, Guangzhou 510006, China)

  • Kai Wang

    (School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China)

  • Qinghua Lu

    (School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China)

  • Lufeng Luo

    (School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China)

  • Wei-Chang Yeh

    (Department of Industrial Engineering and Engineering Management, National Tsing Hua University, Hsinchu 30013, Taiwan)

Abstract

Accurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state detection network of the target. To address this issue, we present a novel visual–tactile fusion model (i.e., RFCT) and provide an incremental dimensional tensor product method for detecting grasping states of weak-stiffness targets. We investigate whether convolutional block attention mechanisms (CBAM) can enhance feature representations by selectively attending to salient visual and tactile cues while suppressing less important information and eliminating redundant information for the initial fusion. We conducted 2250 grasping experiments using 15 weak-stiffness targets. We used 12 targets for training and three for testing. When evaluated on untrained targets, our RFCT model achieved a precision of 82.89%, a recall rate of 82.07%, and an F1 score of 81.65%. We compared RFCT model performance with various combinations of Resnet50 + LSTM and C3D models commonly used in grasping state detection. The experimental results show that our RFCT model significantly outperforms these models. Our proposed method provides accurate grasping state detection and has the potential to provide robust support for robot grasping operations in real-world applications.

Suggested Citation

  • Wenjun Ruan & Wenbo Zhu & Zhijia Zhao & Kai Wang & Qinghua Lu & Lufeng Luo & Wei-Chang Yeh, 2023. "RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets," Mathematics, MDPI, vol. 11(18), pages 1-16, September.
  • Handle: RePEc:gam:jmathe:v:11:y:2023:i:18:p:3969-:d:1242735
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/18/3969/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/11/18/3969/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:18:p:3969-:d:1242735. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.