IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0281309.html
   My bibliography  Save this article

Training machine learning algorithms for automatic facial coding: The role of emotional facial expressions’ prototypicality

Author

Listed:
  • Björn Büdenbender
  • Tim T A Höfling
  • Antje B M Gerdes
  • Georg W Alpers

Abstract

Automatic facial coding (AFC) is a promising new research tool to efficiently analyze emotional facial expressions. AFC is based on machine learning procedures to infer emotion categorization from facial movements (i.e., Action Units). State-of-the-art AFC accurately classifies intense and prototypical facial expressions, whereas it is less accurate for non-prototypical and less intense facial expressions. A potential reason might be that AFC is typically trained with standardized and prototypical facial expression inventories. Because AFC would be useful to analyze less prototypical research material as well, we set out to determine the role of prototypicality in the training material. We trained established machine learning algorithms either with standardized expressions from widely used research inventories or with unstandardized emotional facial expressions obtained in a typical laboratory setting and tested them on identical or cross-over material. All machine learning models’ accuracies were comparable when trained and tested with held-out dataset from the same dataset (acc. = [83.4% to 92.5%]). Strikingly, we found a substantial drop in accuracies for models trained with the highly prototypical standardized dataset when tested in the unstandardized dataset (acc. = [52.8%; 69.8%]). However, when they were trained with unstandardized expressions and tested with standardized datasets, accuracies held up (acc. = [82.7%; 92.5%]). These findings demonstrate a strong impact of the training material’s prototypicality on AFC’s ability to classify emotional faces. Because AFC would be useful for analyzing emotional facial expressions in research or even naturalistic scenarios, future developments should include more naturalistic facial expressions for training. This approach will improve the generalizability of AFC to encode more naturalistic facial expressions and increase robustness for future applications of this promising technology.

Suggested Citation

  • Björn Büdenbender & Tim T A Höfling & Antje B M Gerdes & Georg W Alpers, 2023. "Training machine learning algorithms for automatic facial coding: The role of emotional facial expressions’ prototypicality," PLOS ONE, Public Library of Science, vol. 18(2), pages 1-16, February.
    • Handle: RePEc:plo:pone00:0281309
    DOI: 10.1371/journal.pone.0281309
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0281309
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0281309&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0281309?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:plo:pone00:0223905 is not listed on IDEAS
    2. Kuhn, Max, 2008. "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 28(i05).
    3. Bergmeir, Christoph & Benítez, José M., 2012. "Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 46(i07).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guopeng Jiang & Miles Grafton & Diane Pearson & Mike Bretherton & Allister Holmes, 2019. "Integration of Precision Farming Data and Spatial Statistical Modelling to Interpret Field-Scale Maize Productivity," Agriculture, MDPI, vol. 9(11), pages 1-22, November.
    2. Prabal Das & D. A. Sachindra & Kironmala Chanda, 2022. "Machine Learning-Based Rainfall Forecasting with Multiple Non-Linear Feature Selection Algorithms," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(15), pages 6043-6071, December.
    3. Paulo Infante & Gonçalo Jacinto & Anabela Afonso & Leonor Rego & Pedro Nogueira & Marcelo Silva & Vitor Nogueira & José Saias & Paulo Quaresma & Daniel Santos & Patrícia Góis & Paulo Rebelo Manuel, 2023. "Factors That Influence the Type of Road Traffic Accidents: A Case Study in a District of Portugal," Sustainability, MDPI, vol. 15(3), pages 1-16, January.
    4. Ephrem Habyarimana & Faheem S Baloch, 2021. "Machine learning models based on remote and proximal sensing as potential methods for in-season biomass yields prediction in commercial sorghum fields," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-23, March.
    5. Crespo, Cristian, 2020. "Two become one: improving the targeting of conditional cash transfers with a predictive model of school dropout," LSE Research Online Documents on Economics 123139, London School of Economics and Political Science, LSE Library.
    6. Alexander Wettstein & Gabriel Jenni & Ida Schneider & Fabienne Kühne & Martin grosse Holtforth & Roberto La Marca, 2023. "Predictors of Psychological Strain and Allostatic Load in Teachers: Examining the Long-Term Effects of Biopsychosocial Risk and Protective Factors Using a LASSO Regression Approach," IJERPH, MDPI, vol. 20(10), pages 1-20, May.
    7. Tang, Kayu & Parsons, David J. & Jude, Simon, 2019. "Comparison of automatic and guided learning for Bayesian networks to analyse pipe failures in the water distribution system," Reliability Engineering and System Safety, Elsevier, vol. 186(C), pages 24-36.
    8. Daifeng Xiang & Gangsheng Wang & Jing Tian & Wanyu Li, 2023. "Global patterns and edaphic-climatic controls of soil carbon decomposition kinetics predicted from incubation experiments," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    9. Joel Podgorski & Oliver Kracht & Luis Araguas-Araguas & Stefan Terzer-Wassmuth & Jodie Miller & Ralf Straub & Rolf Kipfer & Michael Berg, 2024. "Groundwater vulnerability to pollution in Africa’s Sahel region," Nature Sustainability, Nature, vol. 7(5), pages 558-567, May.
    10. Tranos, Emmanouil & Incera, Andre Carrascal & Willis, George, 2022. "Using the web to predict regional trade flows: data extraction, modelling, and validation," OSF Preprints 9bu5z, Center for Open Science.
    11. Štefan Lyócsa & Petra Vašaničová & Branka Hadji Misheva & Marko Dávid Vateha, 2022. "Default or profit scoring credit systems? Evidence from European and US peer-to-peer lending markets," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-21, December.
    12. Severinsen, A. & Myrland, Ø., 2022. "ShinyRBase: Near real-time energy saving models using reactive programming," Applied Energy, Elsevier, vol. 325(C).
    13. Marcos Rodrigues & Fermín Alcasena & Pere Gelabert & Cristina Vega‐García, 2020. "Geospatial Modeling of Containment Probability for Escaped Wildfires in a Mediterranean Region," Risk Analysis, John Wiley & Sons, vol. 40(9), pages 1762-1779, September.
    14. Siyu Han & Shixiang Yu & Mengya Shi & Makoto Harada & Jianhong Ge & Jiesheng Lin & Cornelia Prehn & Agnese Petrera & Ying Li & Flora Sam & Giuseppe Matullo & Jerzy Adamski & Karsten Suhre & Christian , 2025. "LEOPARD: missing view completion for multi-timepoint omics data via representation disentanglement and temporal knowledge transfer," Nature Communications, Nature, vol. 16(1), pages 1-20, December.
    15. Natalia Pardo-Lorente & Anestis Gkanogiannis & Luca Cozzuto & Antoni Gañez Zapater & Lorena Espinar & Ritobrata Ghose & Jacqueline Severino & Laura García-López & Rabia Gül Aydin & Laura Martin & Mari, 2024. "Nuclear localization of MTHFD2 is required for correct mitosis progression," Nature Communications, Nature, vol. 15(1), pages 1-23, December.
    16. Andrea Lazzari & Simone Giovinazzo & Giovanni Cabassi & Massimo Brambilla & Carlo Bisaglia & Elio Romano, 2025. "Evaluating Urban Sewage Sludge Distribution on Agricultural Land Using Interpolation and Machine Learning Techniques," Agriculture, MDPI, vol. 15(2), pages 1-13, January.
    17. Giovanny Pillajo-Quijia & Blanca Arenas-Ramírez & Camino González-Fernández & Francisco Aparicio-Izquierdo, 2020. "Influential Factors on Injury Severity for Drivers of Light Trucks and Vans with Machine Learning Methods," Sustainability, MDPI, vol. 12(4), pages 1-28, February.
    18. Zander S. Venter & Adam Sadilek & Charlotte Stanton & David N. Barton & Kristin Aunan & Sourangsu Chowdhury & Aaron Schneider & Stefano Maria Iacus, 2021. "Mobility in Blue-Green Spaces Does Not Predict COVID-19 Transmission: A Global Analysis," IJERPH, MDPI, vol. 18(23), pages 1-12, November.
    19. G. Brooke Anderson & Keith W. Oleson & Bryan Jones & Roger D. Peng, 2018. "Classifying heatwaves: developing health-based models to predict high-mortality versus moderate United States heatwaves," Climatic Change, Springer, vol. 146(3), pages 439-453, February.
    20. Van Belle, Jente & Guns, Tias & Verbeke, Wouter, 2021. "Using shared sell-through data to forecast wholesaler demand in multi-echelon supply chains," European Journal of Operational Research, Elsevier, vol. 288(2), pages 466-479.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0281309. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.