IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-023-44673-2.html
   My bibliography  Save this article

Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface

Author

Listed:
  • Jin Pyo Lee

    (Ulsan National Institute of Science and Technology
    Nanyang Technological University)

  • Hanhyeok Jang

    (Ulsan National Institute of Science and Technology)

  • Yeonwoo Jang

    (Ulsan National Institute of Science and Technology)

  • Hyeonseo Song

    (Ulsan National Institute of Science and Technology)

  • Suwoo Lee

    (Ulsan National Institute of Science and Technology)

  • Pooi See Lee

    (Nanyang Technological University)

  • Jiyun Kim

    (Ulsan National Institute of Science and Technology
    Ulsan National Institute of Science and Technology)

Abstract

Human affects such as emotions, moods, feelings are increasingly being considered as key parameter to enhance the interaction of human with diverse machines and systems. However, their intrinsically abstract and ambiguous nature make it challenging to accurately extract and exploit the emotional information. Here, we develop a multi-modal human emotion recognition system which can efficiently utilize comprehensive emotional information by combining verbal and non-verbal expression data. This system is composed of personalized skin-integrated facial interface (PSiFI) system that is self-powered, facile, stretchable, transparent, featuring a first bidirectional triboelectric strain and vibration sensor enabling us to sense and combine the verbal and non-verbal expression data for the first time. It is fully integrated with a data processing circuit for wireless data transfer allowing real-time emotion recognition to be performed. With the help of machine learning, various human emotion recognition tasks are done accurately in real time even while wearing mask and demonstrated digital concierge application in VR environment.

Suggested Citation

  • Jin Pyo Lee & Hanhyeok Jang & Yeonwoo Jang & Hyeonseo Song & Suwoo Lee & Pooi See Lee & Jiyun Kim, 2024. "Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2
    DOI: 10.1038/s41467-023-44673-2
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-44673-2
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-44673-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    2. Qilin Hua & Junlu Sun & Haitao Liu & Rongrong Bao & Ruomeng Yu & Junyi Zhai & Caofeng Pan & Zhong Lin Wang, 2018. "Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing," Nature Communications, Nature, vol. 9(1), pages 1-11, December.
    3. Domenico Consoli, 2010. "A New Concept of Marketing: The Emotional Marketing," BRAND. Broad Research in Accounting, Negotiation, and Distribution, EduSoft Publishing, vol. 1(1), pages 52-59, September.
    4. Faren Huo & Yeying Zhao & Chunlei Chai & Fei Fang, 2023. "A user experience map design method based on emotional quantification of in-vehicle HMI," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-10, December.
    5. Ryan Yung & Catheryn Khoo-Lattimore & Leigh Ellen Potter, 2021. "Virtual reality and tourism marketing: conceptualizing a framework on presence, emotion, and intention," Current Issues in Tourism, Taylor & Francis Journals, vol. 24(11), pages 1505-1525, June.
    6. Siyoung Lee & Junsoo Kim & Inyeol Yun & Geun Yeol Bae & Daegun Kim & Sangsik Park & Il-Min Yi & Wonkyu Moon & Yoonyoung Chung & Kilwon Cho, 2019. "An ultrathin conformable vibration-responsive electronic skin for quantitative vocal recognition," Nature Communications, Nature, vol. 10(1), pages 1-11, December.
    7. Chuah, Stephanie Hui-Wen & Yu, Joanne, 2021. "The future of service: The power of emotion in human-robot interaction," Journal of Retailing and Consumer Services, Elsevier, vol. 61(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sijia Xu & Jie-Xiang Yu & Hongshuang Guo & Shu Tian & You Long & Jing Yang & Lei Zhang, 2023. "Force-induced ion generation in zwitterionic hydrogels for a sensitive silent-speech sensor," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    2. You Wang & Ming Zhang & Ruifen Hu & Guang Li & Nan Li, 2020. "Silent Speech Recognition for BCI - A Review," Biomedical Journal of Scientific & Technical Research, Biomedical Research Network+, LLC, vol. 27(2), pages 20625-20627, April.
    3. Subhodeep Mukherjee & Manish Mohan Baral & Chittipaka Venkataiah & Surya Kant Pal & Ramji Nagariya, 2021. "Service robots are an option for contactless services due to the COVID-19 pandemic in the hotels," DECISION: Official Journal of the Indian Institute of Management Calcutta, Springer;Indian Institute of Management Calcutta, vol. 48(4), pages 445-460, December.
    4. Prentice, Catherine & Nguyen, Mai, 2021. "Robotic service quality – Scale development and validation," Journal of Retailing and Consumer Services, Elsevier, vol. 62(C).
    5. Linlin Li & Shufang Zhao & Wenhao Ran & Zhexin Li & Yongxu Yan & Bowen Zhong & Zheng Lou & Lili Wang & Guozhen Shen, 2022. "Dual sensing signal decoupling based on tellurium anisotropy for VR interaction and neuro-reflex system application," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    6. Jiayue Zhang & Yikui Gao & Di Liu & Jing-Shan Zhao & Jie Wang, 2023. "Discharge domains regulation and dynamic processes of direct-current triboelectric nanogenerator," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    7. Zhang, Yaqiong & Wang, Shifu, 2023. "The influence of anthropomorphic appearance of artificial intelligence products on consumer behavior and brand evaluation under different product types," Journal of Retailing and Consumer Services, Elsevier, vol. 74(C).
    8. Yoo, Jenny Jeongeun & Song, Sangyoung & Jhang, Jihoon, 2022. "Overhead aversion and facial expressions in crowdfunding," Journal of Retailing and Consumer Services, Elsevier, vol. 69(C).
    9. Yangyang Cui & Jia Zhu & Zhili Duan & Zhenhua Liao & Song Wang & Weiqiang Liu, 2022. "Artificial Intelligence in Spinal Imaging: Current Status and Future Directions," IJERPH, MDPI, vol. 19(18), pages 1-21, September.
    10. Dogerlioglu-Demir, Kivilcim & Ng, Andy H. & Koçaş, Cenk, 2023. "Gracefully yours: Would snap judgments of one's subtle graceful movements lead to inferences about their emotional intelligence?," Journal of Retailing and Consumer Services, Elsevier, vol. 75(C).
    11. Huang, Yu-Shan (Sandy) & Dootson, Paula, 2022. "Chatbots and service failure: When does it lead to customer aggression," Journal of Retailing and Consumer Services, Elsevier, vol. 68(C).
    12. Prentice, Catherine & Wong, IpKin Anthony & Lin, Zhiwei (CJ), 2023. "Artificial intelligence as a boundary-crossing object for employee engagement and performance," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    13. Hang Zhang & Sankaran Sundaresan & Michael A. Webb, 2024. "Thermodynamic driving forces in contact electrification between polymeric materials," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    14. Haoran Jin & Zesheng Zheng & Zequn Cui & Ying Jiang & Geng Chen & Wenlong Li & Zhimin Wang & Jilei Wang & Chuanshi Yang & Weitao Song & Xiaodong Chen & Yuanjin Zheng, 2023. "A flexible optoacoustic blood ‘stethoscope’ for noninvasive multiparametric cardiovascular monitoring," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    15. Roberto de Fazio & Donato Cafagna & Giorgio Marcuccio & Alessandro Minerba & Paolo Visconti, 2020. "A Multi-Source Harvesting System Applied to Sensor-Based Smart Garments for Monitoring Workers’ Bio-Physical Parameters in Harsh Environments," Energies, MDPI, vol. 13(9), pages 1-33, May.
    16. Taemin Kim & Yejee Shin & Kyowon Kang & Kiho Kim & Gwanho Kim & Yunsu Byeon & Hwayeon Kim & Yuyan Gao & Jeong Ryong Lee & Geonhui Son & Taeseong Kim & Yohan Jun & Jihyun Kim & Jinyoung Lee & Seyun Um , 2022. "Ultrathin crystalline-silicon-based strain gauges with deep learning algorithms for silent speech interfaces," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    17. Stanislav Ivanov & Craig Webster & Faruk Seyitoğlu, 2023. "Humans and/or robots? Tourists’ preferences towards the humans–robots mix in the service delivery system," Service Business, Springer;Pan-Pacific Business Association, vol. 17(1), pages 195-231, March.
    18. Schirpke, Uta & Ghermandi, Andrea & Sinclair, Michael & Van Berkel, Derek & Fox, Nathan & Vargas, Leonardo & Willemen, Louise, 2023. "Emerging technologies for assessing ecosystem services: A synthesis of opportunities and challenges," Ecosystem Services, Elsevier, vol. 63(C).
    19. Hira Hameed & Muhammad Usman & Ahsen Tahir & Amir Hussain & Hasan Abbas & Tie Jun Cui & Muhammad Ali Imran & Qammer H. Abbasi, 2022. "Pushing the limits of remote RF sensing by reading lips under the face mask," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    20. Mahmoud Wagih & Junjie Shi & Menglong Li & Abiodun Komolafe & Thomas Whittaker & Johannes Schneider & Shanmugam Kumar & William Whittow & Steve Beeby, 2024. "Wide-range soft anisotropic thermistor with a direct wireless radio frequency interface," Nature Communications, Nature, vol. 15(1), pages 1-10, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.