IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v12y2024i8p1164-d1374797.html
   My bibliography  Save this article

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

Author

Listed:
  • Walter Peter Vispoel

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

  • Hyeryung Lee

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

  • Tingting Chen

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

Abstract

We illustrate how structural equation models (SEMs) can be used to assess the reliability and generalizability of composite and subscale scores, proportions of multiple sources of measurement error, and subscale added value within multivariate designs using data from a popular inventory measuring hierarchically structured personality traits. We compare these techniques between standard SEMs representing congeneric relations between indicators and underlying factors versus SEM-based generalizability theory (GT) designs with simplified essential tau-equivalent constraints. Results strongly emphasized the importance of accounting for multiple sources of measurement error in both contexts and revealed that, in most but not all instances, congeneric designs yielded higher score accuracy, lower proportions of measurement error, greater average subscale score viability, stronger model fits, and differing magnitudes of disattenuated subscale intercorrelations. Extending the congeneric analyses to the item level further highlighted consistent weaknesses in the psychometric properties of negatively versus positively keyed items. Collectively, these findings demonstrate the practical value and advantages of applying GT-based principles to congeneric SEMs that are much more commonly encountered in the research literature and more directly linked to the specific measures being analyzed. We also provide prophecy formulas to estimate reliability and generalizability coefficients, proportions of individual sources of measurement error, and subscale added-value indices for changes made to measurement procedures and offer guidelines and examples for running all illustrated analyses using the lavaan (Version 0.6-17) and semTools (Version 0.5-6) packages in R. The methods described for the analyzed designs are applicable to any objectively or subjectively scored assessments for which both composite and subcomponent scores are reported.

Suggested Citation

  • Walter Peter Vispoel & Hyeryung Lee & Tingting Chen, 2024. "Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice," Mathematics, MDPI, vol. 12(8), pages 1-25, April.
    • Handle: RePEc:gam:jmathe:v:12:y:2024:i:8:p:1164-:d:1374797
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/12/8/1164/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/12/8/1164/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Goldine Gleser & Lee Cronbach & Nageswari Rajaratnam, 1965. "Generalizability of scores influenced by multiple sources of variance," Psychometrika, Springer;The Psychometric Society, vol. 30(4), pages 395-418, December.
    2. Shelby Haberman & Sandip Sinharay, 2010. "Reporting of Subscores Using Multidimensional Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 209-227, June.
    3. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    4. Rosseel, Yves, 2012. "lavaan: An R Package for Structural Equation Modeling," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i02).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Md. Mominur Rahman & Bilkis Akhter, 2021. "The impact of investment in human capital on bank performance: evidence from Bangladesh," Future Business Journal, Springer, vol. 7(1), pages 1-13, December.
    2. Andreea-Ionela Puiu & Anca Monica Ardeleanu & Camelia Cojocaru & Anca Bratu, 2021. "Exploring the Effect of Status Quo, Innovativeness, and Involvement Tendencies on Luxury Fashion Innovations: The Mediation Role of Status Consumption," Mathematics, MDPI, vol. 9(9), pages 1-18, May.
    3. Allen, Jaime & Muñoz, Juan Carlos & Ortúzar, Juan de Dios, 2019. "On evasion behaviour in public transport: Dissatisfaction or contagion?," Transportation Research Part A: Policy and Practice, Elsevier, vol. 130(C), pages 626-651.
    4. Sai-fu Fung & Esther Oi-wah Chow & Chau-kiu Cheung, 2020. "Development and Evaluation of the Psychometric Properties of a Brief Wisdom Development Scale," IJERPH, MDPI, vol. 17(8), pages 1-14, April.
    5. Ziaul Haque Munim & Hans-Joachim Schramm, 2018. "The impacts of port infrastructure and logistics performance on economic growth: the mediating role of seaborne trade," Journal of Shipping and Trade, Springer, vol. 3(1), pages 1-19, December.
    6. Sam S. S. Lau & Cherry C. Y. Ho & Rebecca C. K. Pang & Susan Su & Heather Kwok & Sai-fu Fung & Roger C. Ho, 2022. "COVID-19 Burnout Subject to the Dynamic Zero-COVID Policy in Hong Kong: Development and Psychometric Evaluation of the COVID-19 Burnout Frequency Scale," Sustainability, MDPI, vol. 14(14), pages 1-13, July.
    7. Mohammed Ali Sharafuddin & Meena Madhavan & Thanapong Chaichana, 2022. "The Effects of Innovation Adoption and Social Factors between Sustainable Supply Chain Management Practices and Sustainable Firm Performance: A Moderated Mediation Model," Sustainability, MDPI, vol. 14(15), pages 1-30, July.
    8. Leonie Kuen & Fiona Schürmann & Daniel Westmattelmann & Sophie Hartwig & Shay Tzafrir & Gerhard Schewe, 2023. "Trust transfer effects and associated risks in telemedicine adoption," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-22, December.
    9. Vitor Ciampolini & Fernando Santos & Ricardo Teixeira Quinaud & Martin Camiré & Maurício de Oliveira Migliano & Juarez Vieira do Nascimento & Michel Milistetd, 2021. "Cross-Cultural Adaptation and Psychometric Properties of the Portuguese Coaching Life Skills in Sport Questionnaire," SAGE Open, , vol. 11(2), pages 21582440211, June.
    10. Allen, Jaime & Muñoz, Juan Carlos & Rosell, Jordi, 2019. "Effect of a major network reform on bus transit satisfaction," Transportation Research Part A: Policy and Practice, Elsevier, vol. 124(C), pages 310-333.
    11. Silvia Mariela Méndez-Prado & Vanessa Rodriguez & Kevin Peralta-Rizzo & Patricia Everaert & Martin Valcke, 2023. "An Assessment Tool to Identify the Financial Literacy Level of Financial Education Programs Participants’ Executed by Ecuadorian Financial Institutions," Sustainability, MDPI, vol. 15(2), pages 1-24, January.
    12. Westmattelmann, Daniel & Grotenhermen, Jan-Gerrit & Sprenger, Marius & Rand, William & Schewe, Gerhard, 2021. "Apart we ride together: The motivations behind users of mixed-reality sports," Journal of Business Research, Elsevier, vol. 134(C), pages 316-328.
    13. Sebastian Kurten & David Winant & Kathleen Beullens, 2021. "Mothers Matter: Using Regression Tree Algorithms to Predict Adolescents’ Sharing of Drunk References on Social Media," IJERPH, MDPI, vol. 18(21), pages 1-16, October.
    14. Lena Busch & Till Utesch & Bernd Strauss, 2019. "Validation of the vignette-based German Exercise Causality Orientation Scale (G-ECOS)," PLOS ONE, Public Library of Science, vol. 14(10), pages 1-20, October.
    15. Alexander Robitzsch, 2022. "Comparing the Robustness of the Structural after Measurement (SAM) Approach to Structural Equation Modeling (SEM) against Local Model Misspecifications with Alternative Estimation Approaches," Stats, MDPI, vol. 5(3), pages 1-42, July.
    16. Érika Martins Silva Ramos & Cecilia Jakobsson Bergstad, 2021. "The Psychology of Sharing: Multigroup Analysis among Users and Non-Users of Carsharing," Sustainability, MDPI, vol. 13(12), pages 1-17, June.
    17. Allen, Jaime & Muñoz, Juan Carlos & Ortúzar, Juan de Dios, 2018. "Modelling service-specific and global transit satisfaction under travel and user heterogeneity," Transportation Research Part A: Policy and Practice, Elsevier, vol. 113(C), pages 509-528.
    18. Daniel Schulze & Kathrin Heinitz & Timo Lorenz, 2018. "Comparative organizational research starts with sound measurement: Validity and invariance of Turker’s corporate social responsibility scale in five cross-cultural samples," PLOS ONE, Public Library of Science, vol. 13(11), pages 1-15, November.
    19. Shenghai Dai & Dubravka Svetina & Xiaolin Wang, 2017. "Reporting Subscores Using R: A Software Review," Journal of Educational and Behavioral Statistics, , vol. 42(5), pages 617-638, October.
    20. Essie Sutton & Kimberly A. Schonert-Reichl & Amery D. Wu & Molly Stewart Lawlor, 2018. "Evaluating the Reliability and Validity of the Self-Compassion Scale Short Form Adapted for Children Ages 8–12," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 11(4), pages 1217-1236, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:12:y:2024:i:8:p:1164-:d:1374797. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.