IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-023-44673-2.html
   My bibliography  Save this article

Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface

Author

Listed:
  • Jin Pyo Lee

    (Ulsan National Institute of Science and Technology
    Nanyang Technological University)

  • Hanhyeok Jang

    (Ulsan National Institute of Science and Technology)

  • Yeonwoo Jang

    (Ulsan National Institute of Science and Technology)

  • Hyeonseo Song

    (Ulsan National Institute of Science and Technology)

  • Suwoo Lee

    (Ulsan National Institute of Science and Technology)

  • Pooi See Lee

    (Nanyang Technological University)

  • Jiyun Kim

    (Ulsan National Institute of Science and Technology
    Ulsan National Institute of Science and Technology)

Abstract

Human affects such as emotions, moods, feelings are increasingly being considered as key parameter to enhance the interaction of human with diverse machines and systems. However, their intrinsically abstract and ambiguous nature make it challenging to accurately extract and exploit the emotional information. Here, we develop a multi-modal human emotion recognition system which can efficiently utilize comprehensive emotional information by combining verbal and non-verbal expression data. This system is composed of personalized skin-integrated facial interface (PSiFI) system that is self-powered, facile, stretchable, transparent, featuring a first bidirectional triboelectric strain and vibration sensor enabling us to sense and combine the verbal and non-verbal expression data for the first time. It is fully integrated with a data processing circuit for wireless data transfer allowing real-time emotion recognition to be performed. With the help of machine learning, various human emotion recognition tasks are done accurately in real time even while wearing mask and demonstrated digital concierge application in VR environment.

Suggested Citation

  • Jin Pyo Lee & Hanhyeok Jang & Yeonwoo Jang & Hyeonseo Song & Suwoo Lee & Pooi See Lee & Jiyun Kim, 2024. "Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2
    DOI: 10.1038/s41467-023-44673-2
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-44673-2
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-44673-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    2. Qilin Hua & Junlu Sun & Haitao Liu & Rongrong Bao & Ruomeng Yu & Junyi Zhai & Caofeng Pan & Zhong Lin Wang, 2018. "Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing," Nature Communications, Nature, vol. 9(1), pages 1-11, December.
    3. Domenico Consoli, 2010. "A New Concept of Marketing: The Emotional Marketing," BRAND. Broad Research in Accounting, Negotiation, and Distribution, EduSoft Publishing, vol. 1(1), pages 52-59, September.
    4. Faren Huo & Yeying Zhao & Chunlei Chai & Fei Fang, 2023. "A user experience map design method based on emotional quantification of in-vehicle HMI," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-10, December.
    5. Ryan Yung & Catheryn Khoo-Lattimore & Leigh Ellen Potter, 2021. "Virtual reality and tourism marketing: conceptualizing a framework on presence, emotion, and intention," Current Issues in Tourism, Taylor & Francis Journals, vol. 24(11), pages 1505-1525, June.
    6. Siyoung Lee & Junsoo Kim & Inyeol Yun & Geun Yeol Bae & Daegun Kim & Sangsik Park & Il-Min Yi & Wonkyu Moon & Yoonyoung Chung & Kilwon Cho, 2019. "An ultrathin conformable vibration-responsive electronic skin for quantitative vocal recognition," Nature Communications, Nature, vol. 10(1), pages 1-11, December.
    7. Chuah, Stephanie Hui-Wen & Yu, Joanne, 2021. "The future of service: The power of emotion in human-robot interaction," Journal of Retailing and Consumer Services, Elsevier, vol. 61(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sijia Xu & Jie-Xiang Yu & Hongshuang Guo & Shu Tian & You Long & Jing Yang & Lei Zhang, 2023. "Force-induced ion generation in zwitterionic hydrogels for a sensitive silent-speech sensor," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    2. You Wang & Ming Zhang & Ruifen Hu & Guang Li & Nan Li, 2020. "Silent Speech Recognition for BCI - A Review," Biomedical Journal of Scientific & Technical Research, Biomedical Research Network+, LLC, vol. 27(2), pages 20625-20627, April.
    3. Song, Christina Soyoung & Kim, Youn-Kyung, 2022. "The role of the human-robot interaction in consumers’ acceptance of humanoid retail service robots," Journal of Business Research, Elsevier, vol. 146(C), pages 489-503.
    4. Subhodeep Mukherjee & Manish Mohan Baral & Chittipaka Venkataiah & Surya Kant Pal & Ramji Nagariya, 2021. "Service robots are an option for contactless services due to the COVID-19 pandemic in the hotels," DECISION: Official Journal of the Indian Institute of Management Calcutta, Springer;Indian Institute of Management Calcutta, vol. 48(4), pages 445-460, December.
    5. Shuo Li & Yong Zhang & Xiaoping Liang & Haomin Wang & Haojie Lu & Mengjia Zhu & Huimin Wang & Mingchao Zhang & Xinping Qiu & Yafeng Song & Yingying Zhang, 2022. "Humidity-sensitive chemoelectric flexible sensors based on metal-air redox reaction for health management," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    6. María Dolores Aledo‐Ruiz & Eva Martínez‐Caro & José Manuel Santos‐Jaén, 2022. "The influence of corporate social responsibility on students' emotional appeal in the HEIs: The mediating effect of reputation and corporate image," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 29(3), pages 578-592, May.
    7. Prentice, Catherine & Nguyen, Mai, 2021. "Robotic service quality – Scale development and validation," Journal of Retailing and Consumer Services, Elsevier, vol. 62(C).
    8. Xueguang Lu & Feilong Zhang & Liguo Zhu & Shan Peng & Jiazhen Yan & Qiwu Shi & Kefan Chen & Xue Chang & Hongfu Zhu & Cheng Zhang & Wanxia Huang & Qiang Cheng, 2024. "A terahertz meta-sensor array for 2D strain mapping," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    9. Minglu Wang & Shanshan Liu & Lingling Hu & Jong-Yoon Lee, 2023. "A Study of Metaverse Exhibition Sustainability on the Perspective of the Experience Economy," Sustainability, MDPI, vol. 15(12), pages 1-14, June.
    10. Linlin Li & Shufang Zhao & Wenhao Ran & Zhexin Li & Yongxu Yan & Bowen Zhong & Zheng Lou & Lili Wang & Guozhen Shen, 2022. "Dual sensing signal decoupling based on tellurium anisotropy for VR interaction and neuro-reflex system application," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    11. Subero-Navarro, à urea & Pelegrín-Borondo, Jorge & Reinares-Lara, Eva & Olarte-Pascual, Cristina, 2022. "Proposal for modeling social robot acceptance by retail customers: CAN model + technophobia," Journal of Retailing and Consumer Services, Elsevier, vol. 64(C).
    12. Jiayue Zhang & Yikui Gao & Di Liu & Jing-Shan Zhao & Jie Wang, 2023. "Discharge domains regulation and dynamic processes of direct-current triboelectric nanogenerator," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    13. Zhang, Yaqiong & Wang, Shifu, 2023. "The influence of anthropomorphic appearance of artificial intelligence products on consumer behavior and brand evaluation under different product types," Journal of Retailing and Consumer Services, Elsevier, vol. 74(C).
    14. Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    15. Li, Sixian & Peluso, Alessandro M. & Duan, Jinyun, 2023. "Why do we prefer humans to artificial intelligence in telemarketing? A mind perception explanation," Journal of Retailing and Consumer Services, Elsevier, vol. 70(C).
    16. Tianyang Huang, 2022. "What Affects the Acceptance and Use of Hotel Service Robots by Elderly Customers?," Sustainability, MDPI, vol. 14(23), pages 1-17, December.
    17. Yoo, Jenny Jeongeun & Song, Sangyoung & Jhang, Jihoon, 2022. "Overhead aversion and facial expressions in crowdfunding," Journal of Retailing and Consumer Services, Elsevier, vol. 69(C).
    18. Meng, Lu (Monroe) & Duan, Shen & Zhao, Yijun & Lü, Kevin & Chen, Siyun, 2021. "The impact of online celebrity in livestreaming E-commerce on purchase intention from the perspective of emotional contagion," Journal of Retailing and Consumer Services, Elsevier, vol. 63(C).
    19. Yangyang Cui & Jia Zhu & Zhili Duan & Zhenhua Liao & Song Wang & Weiqiang Liu, 2022. "Artificial Intelligence in Spinal Imaging: Current Status and Future Directions," IJERPH, MDPI, vol. 19(18), pages 1-21, September.
    20. Wang, Yawei & Kang, Qi & Zhou, Shoujiang & Dong, Yuanyuan & Liu, Junqi, 2022. "The impact of service robots in retail: Exploring the effect of novelty priming on consumer behavior," Journal of Retailing and Consumer Services, Elsevier, vol. 68(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.