IDEAS home Printed from https://ideas.repec.org/a/spr/advdac/v19y2025i1d10.1007_s11634-024-00581-x.html
   My bibliography  Save this article

Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

Author

Listed:
  • A. Martín Andrés

    (University of Granada)

  • M. Álvarez Hernández

    (CITMAga
    Spanish Naval Academy)

Abstract

To measure the degree of agreement between R observers who independently classify n subjects within K categories, various kappa-type coefficients are often used. When R = 2, it is common to use the Cohen' kappa, Scott's pi, Gwet’s AC1/2, and Krippendorf's alpha coefficients (weighted or not). When R > 2, some pairwise version based on the aforementioned coefficients is normally used; with the same order as above: Hubert's kappa, Fleiss's kappa, Gwet's AC1/2, and Krippendorf's alpha. However, all these statistics are based on biased estimators of the expected index of agreements, since they estimate the product of two population proportions through the product of their sample estimators. The aims of this article are three. First, to provide statistics based on unbiased estimators of the expected index of agreements and determine their variance based on the variance of the original statistic. Second, to make pairwise extensions of some measures. And third, to show that the old and new estimators of the Cohen’s kappa and Hubert’s kappa coefficients match the well-known estimators of concordance and intraclass correlation coefficients, if the former are defined by assuming quadratic weights. The article shows that the new estimators are always greater than or equal the classic ones, except for the case of Gwet where it is the other way around, although these differences are only relevant with small sample sizes (e.g. n ≤ 30).

Suggested Citation

  • A. Martín Andrés & M. Álvarez Hernández, 2025. "Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 19(1), pages 177-207, March.
    • Handle: RePEc:spr:advdac:v:19:y:2025:i:1:d:10.1007_s11634-024-00581-x
    DOI: 10.1007/s11634-024-00581-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11634-024-00581-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11634-024-00581-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christof Schuster & David Smith, 2005. "Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 135-146, March.
    2. J. Richard Landis & Gary G. Koch, 1975. "A review of statistical methods in the analysis of data arising from observer reliability studies (Part II)," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 29(4), pages 151-161, December.
    3. J. Richard Landis & Gary G. Koch, 1975. "A review of statistical methods in the analysis of data arising from observer reliability studies (Part I)," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 29(3), pages 101-123, September.
    4. Klaus krippendorff, 2004. "Measuring the Reliability of Qualitative Text Analysis Data," Quality & Quantity: International Journal of Methodology, Springer, vol. 38(6), pages 787-800, December.
    5. Matthijs Warrens, 2010. "Inequalities between multi-rater kappas," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 4(4), pages 271-286, December.
    6. Josep L. Carrasco & Lluís Jover, 2003. "Estimating the Generalized Concordance Correlation Coefficient through Variance Components," Biometrics, The International Biometric Society, vol. 59(4), pages 849-858, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Debby L Gerritsen & Nardi Steverink & Dinnus HM Frijters & Marcel E Ooms & Miel W Ribbe, 2010. "Social well‐being and its measurement in the nursing home, the SWON‐scale," Journal of Clinical Nursing, John Wiley & Sons, vol. 19(9‐10), pages 1243-1251, May.
    2. N. Lu & T. Chen & P. Wu & D. Gunzler & H. Zhang & H. He & X.M. Tu, 2014. "Functional response models for intraclass correlation coefficients," Journal of Applied Statistics, Taylor & Francis Journals, vol. 41(11), pages 2539-2556, November.
    3. Christof Schuster & David Smith, 2005. "Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 135-146, March.
    4. Högberg, Hans & Svensson, Elisabeth, 2008. "An Overview of Methods in the Analysis of Dependent ordered catagorical Data: Assumptions and Implications," Working Papers 2008:7, Örebro University, School of Business.
    5. Fabio Rapallo, 2005. "Algebraic exact inference for rater agreement models," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 14(1), pages 45-66, February.
    6. Cheng, Song-Show & Cheng, Yu-Chun, 1998. "An ordered relation between the ANOVA estimator of the intraclass correlation and a kappa-type statistic in binary data," Statistics & Probability Letters, Elsevier, vol. 38(3), pages 275-280, June.
    7. Jonas Moss, 2024. "Measures of Agreement with Multiple Raters: Fréchet Variances and Inference," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 517-541, June.
    8. Alexandra Raadt & Matthijs J. Warrens & Roel J. Bosker & Henk A. L. Kiers, 2021. "A Comparison of Reliability Coefficients for Ordinal Rating Scales," Journal of Classification, Springer;The Classification Society, vol. 38(3), pages 519-543, October.
    9. Högberg, Hans & Svensson, Elisabeth, 2008. "Comparison of methods in the analysis of dependent ordered catagorical data," Working Papers 2008:6, Örebro University, School of Business.
    10. Claudia Arena & Simona Catuogno & Nicola Moscariello, 2021. "The unusual debate on non-GAAP reporting in the current standard practice. The lens of corporate governance," Journal of Management & Governance, Springer;Accademia Italiana di Economia Aziendale (AIDEA), vol. 25(3), pages 655-684, September.
    11. Matthijs J. Warrens, 2021. "Kappa coefficients for dichotomous-nominal classifications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(1), pages 193-208, March.
    12. Sebastián Feu & Javier García-Rubio & María de Gracia Gamero & Sergio J Ibáñez, 2019. "Task planning for sports learning by physical education teachers in the pre-service phase," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-18, March.
    13. Juan Molero & Isabel Rodríguez-Tejedo, 2010. "An index of political support for decentralization: the Spanish case," Constitutional Political Economy, Springer, vol. 21(1), pages 50-79, March.
    14. Helenowski Irene B & Vonesh Edward F & Demirtas Hakan & Rademaker Alfred W & Ananthanarayanan Vijayalakshmi & Gann Peter H & Jovanovic Borko D, 2011. "Defining Reproducibility Statistics as a Function of the Spatial Covariance Structures in Biomarker Studies," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-21, January.
    15. Miruna Radu-Lefebvre & James Davis & William Gartner, 2024. "Legacy in Family Business: A Systematic Literature Review and Future Research Agenda," Post-Print hal-04515862, HAL.
    16. Somendra Narayan & Jatinder S. Sidhu & Henk W. Volberda, 2021. "From Attention to Action: The Influence of Cognitive and Ideological Diversity in Top Management Teams on Business Model Innovation," Journal of Management Studies, Wiley Blackwell, vol. 58(8), pages 2082-2110, December.
    17. Alexandre Rodrigues da Silva & Claudia Brito Silva Cirani & Fernando Antonio Ribeiro Serra & Angélica Pigola & Priscila Rezende da Costa & Isabel Cristina Scafuto & Roberto Lima Ruas & Marcos Rogério , 2023. "Determining Factors on Green Innovation Adoption: An Empirical Study in Brazilian Agribusiness Firms," Sustainability, MDPI, vol. 15(7), pages 1-23, April.
    18. Boons, Mark & Stam, Daan, 2019. "Crowdsourcing for innovation: How related and unrelated perspectives interact to increase creative performance," Research Policy, Elsevier, vol. 48(7), pages 1758-1770.
    19. Matthijs Warrens, 2014. "Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa," Journal of Classification, Springer;The Classification Society, vol. 31(2), pages 179-193, July.
    20. Zarr, Siegalit & Carone, Nicola & Gartrell, Nanette & Koh, Audrey & Bos, Henny, 2022. "Concerns of emerging adults who were born and raised in planned lesbian-parent families," Children and Youth Services Review, Elsevier, vol. 136(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:advdac:v:19:y:2025:i:1:d:10.1007_s11634-024-00581-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.