IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v48y2014i4p2355-2369.html

Estimating intercoder reliability: a structural equation modeling approach

Author

Listed:
  • Guangchao Feng

Abstract

Intercoder reliability is usually estimated with a summary index, and yet the limitations concerning the indexing approach have been well noted. This study critically reviewed all the existing major modeling approaches to estimating intercoder reliability, and empirically tested and further compared these approaches. It was found that latent variable modeling, also called the second-generation SEM, generally perform better than log-linear modeling, and is able to explain the paradox haunting some indices, and to spot the sources of disagreement among coders. Implications were discussed at last. Copyright Springer Science+Business Media Dordrecht 2014

Suggested Citation

  • Guangchao Feng, 2014. "Estimating intercoder reliability: a structural equation modeling approach," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(4), pages 2355-2369, July.
    • Handle: RePEc:spr:qualqt:v:48:y:2014:i:4:p:2355-2369
    DOI: 10.1007/s11135-014-0034-7
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11135-014-0034-7
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11135-014-0034-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Agresti, Alan, 1989. "An agreement model with kappa as parameter," Statistics & Probability Letters, Elsevier, vol. 7(4), pages 271-273, February.
    2. Stanley Sclove, 1987. "Application of model-selection criteria to some problems in multivariate analysis," Psychometrika, Springer;The Psychometric Society, vol. 52(3), pages 333-343, September.
    3. Linzer, Drew A. & Lewis, Jeffrey B., 2011. "poLCA: An R Package for Polytomous Variable Latent Class Analysis," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i10).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Aline Riboli Marasca & Maurício Scopel Hoffmann & Anelise Reis Gaya & Denise Ruschel Bandeira, 2021. "Subjective Well-Being and Psychopathology Symptoms: Mental Health Profiles and their Relations with Academic Achievement in Brazilian Children," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 14(3), pages 1121-1137, June.
    2. Laurent Ott & Mehdi Farsi & Sylvain Weber, 2021. "Beyond political divides: analyzing public opinion on carbon taxation in Switzerland," Chapters, in: Axel Franzen & Sebastian Mader (ed.), Research Handbook on Environmental Sociology, chapter 17, pages 313-339, Edward Elgar Publishing.
    3. Heilala, Ville & Kelly, Riitta & Järvinen, Miitta & Hämäläinen, Raija, 2026. "Toward closing the zipper: Technology readiness and diverse technology use of university students in terms of academic major and gender," Technological Forecasting and Social Change, Elsevier, vol. 224(C).
    4. Collart, Alba J. & Palma, Marco A., 2014. "What Motivates Individuals to Participate in Economic Experiments? A Latent Class Analysis with Unobserved Heterogeneity," 2014 Annual Meeting, July 27-29, 2014, Minneapolis, Minnesota 170401, Agricultural and Applied Economics Association.
    5. Lehrl, Simone & Smidt, Wilfried & Grosse, Christiane & Richter, David, 2014. "Patterns of Literacy and Numeracy Activities in Preschool and Their Relation to Structural Characteristics and Children's Home Activities," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 29(5), pages 577-597.
    6. Marianna Virtanen & Jussi Vahtera & Jenny Head & Rosemary Dray-Spira & Annaleena Okuloff & Adam G Tabak & Marcel Goldberg & Jenni Ervasti & Markus Jokela & Archana Singh-Manoux & Jaana Pentti & Marie , 2015. "Work Disability among Employees with Diabetes: Latent Class Analysis of Risk Factors in Three Prospective Cohort Studies," PLOS ONE, Public Library of Science, vol. 10(11), pages 1-14, November.
    7. Jiwon Lee & Midam An & Yongku Kim & Jung-In Seo, 2021. "Optimal Allocation for Electric Vehicle Charging Stations," Energies, MDPI, vol. 14(18), pages 1-10, September.
    8. Palma, Marco A. & Ness, Meghan L. & Anderson, David P., 2015. "Buying More than Taste? A Latent Class Analysis of Health and Prestige Determinants of Healthy Food," 2015 Conference (59th), February 10-13, 2015, Rotorua, New Zealand 202566, Australian Agricultural and Resource Economics Society.
    9. Alan Crane & Kevin Crotty, 2020. "How Skilled Are Security Analysts?," Journal of Finance, American Finance Association, vol. 75(3), pages 1629-1675, June.
    10. Adrian O’Hagan & Arthur White, 2019. "Improved model-based clustering performance using Bayesian initialization averaging," Computational Statistics, Springer, vol. 34(1), pages 201-231, March.
    11. Schneider Lecy, Kate & Sun, Bangyao & Cash, Sean B. & Feng, Wenhui & Thorne-Lyman, Andrew & Love, David C., 2025. "What happens to diet quality when food prices rise? Revealed preference from national household scanner data, 2015-2018," 2025 AAEA & WAEA Joint Annual Meeting, July 27-29, 2025, Denver, CO 360876, Agricultural and Applied Economics Association.
    12. repec:plo:pone00:0143517 is not listed on IDEAS
    13. repec:plo:pone00:0228377 is not listed on IDEAS
    14. Danks, Nicholas P. & Sharma, Pratyush N. & Sarstedt, Marko, 2020. "Model selection uncertainty and multimodel inference in partial least squares structural equation modeling (PLS-SEM)," Journal of Business Research, Elsevier, vol. 113(C), pages 13-24.
    15. Rosa S. Wong & Keith T. S. Tung & Nirmala Rao & Ko Ling Chan & King-Wa Fu & Jason C. Yam & Winnie W. Y. Tso & Wilfred H. S. Wong & Terry Y. S. Lum & Ian C. K. Wong & Patrick Ip, 2022. "Using Latent Class Analyses to Examine Health Disparities among Young Children in Socially Disadvantaged Families during the COVID-19 Pandemic," IJERPH, MDPI, vol. 19(13), pages 1-13, June.
    16. Martin Lukac & Nadja Doerflinger & Valeria Pulignano, 2019. "Developing a Cross-National Comparative Framework for Studying Labour Market Segmentation: Measurement Equivalence with Latent Class Analysis," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 145(1), pages 233-255, August.
    17. Po-Hsien Huang, 2017. "Asymptotics of AIC, BIC, and RMSEA for Model Selection in Structural Equation Modeling," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 407-426, June.
    18. Saengryeol Park & So-Youn Park & Gapjin Oh & In-Hwan Oh, 2019. "Latent Profiles Based on Light Physical Activity, Sedentary Behavior, Perceived Body Shape, and Body Mass Index in Patients with Dyslipidemia Influence Their Quality of Life," IJERPH, MDPI, vol. 16(20), pages 1-10, October.
    19. Lisa Blaydes, 2023. "Assessing the Labor Conditions of Migrant Domestic Workers in the Arab Gulf States," ILR Review, Cornell University, ILR School, vol. 76(4), pages 724-747, August.
    20. Hanna Lee & Jeong-Won Han, 2021. "Analysis of Parenting Attitude Types and Influencing Factors of Korean Parents by Using Latent Transition Analysis," IJERPH, MDPI, vol. 18(14), pages 1-13, July.
    21. Lisbeth Lund & Christian Ritz, 2025. "Flexible and modular latent transition analysis—A tutorial using R," PLOS ONE, Public Library of Science, vol. 20(1), pages 1-13, January.
    22. Seohee Park & Seongeun Kim & Ji Hoon Ryoo, 2020. "Latent Class Regression Utilizing Fuzzy Clusterwise Generalized Structured Component Analysis," Mathematics, MDPI, vol. 8(11), pages 1-16, November.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:48:y:2014:i:4:p:2355-2369. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.