IDEAS home Printed from https://ideas.repec.org/a/spr/advdac/v19y2025i1d10.1007_s11634-024-00581-x.html
   My bibliography  Save this article

Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

Author

Listed:
  • A. Martín Andrés

    (University of Granada)

  • M. Álvarez Hernández

    (CITMAga
    Spanish Naval Academy)

Abstract

To measure the degree of agreement between R observers who independently classify n subjects within K categories, various kappa-type coefficients are often used. When R = 2, it is common to use the Cohen' kappa, Scott's pi, Gwet’s AC1/2, and Krippendorf's alpha coefficients (weighted or not). When R > 2, some pairwise version based on the aforementioned coefficients is normally used; with the same order as above: Hubert's kappa, Fleiss's kappa, Gwet's AC1/2, and Krippendorf's alpha. However, all these statistics are based on biased estimators of the expected index of agreements, since they estimate the product of two population proportions through the product of their sample estimators. The aims of this article are three. First, to provide statistics based on unbiased estimators of the expected index of agreements and determine their variance based on the variance of the original statistic. Second, to make pairwise extensions of some measures. And third, to show that the old and new estimators of the Cohen’s kappa and Hubert’s kappa coefficients match the well-known estimators of concordance and intraclass correlation coefficients, if the former are defined by assuming quadratic weights. The article shows that the new estimators are always greater than or equal the classic ones, except for the case of Gwet where it is the other way around, although these differences are only relevant with small sample sizes (e.g. n ≤ 30).

Suggested Citation

  • A. Martín Andrés & M. Álvarez Hernández, 2025. "Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 19(1), pages 177-207, March.
  • Handle: RePEc:spr:advdac:v:19:y:2025:i:1:d:10.1007_s11634-024-00581-x
    DOI: 10.1007/s11634-024-00581-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11634-024-00581-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11634-024-00581-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christof Schuster & David Smith, 2005. "Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 135-146, March.
    2. J. Richard Landis & Gary G. Koch, 1975. "A review of statistical methods in the analysis of data arising from observer reliability studies (Part II)," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 29(4), pages 151-161, December.
    3. J. Richard Landis & Gary G. Koch, 1975. "A review of statistical methods in the analysis of data arising from observer reliability studies (Part I)," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 29(3), pages 101-123, September.
    4. Klaus krippendorff, 2004. "Measuring the Reliability of Qualitative Text Analysis Data," Quality & Quantity: International Journal of Methodology, Springer, vol. 38(6), pages 787-800, December.
    5. Matthijs Warrens, 2010. "Inequalities between multi-rater kappas," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 4(4), pages 271-286, December.
    6. Josep L. Carrasco & Lluís Jover, 2003. "Estimating the Generalized Concordance Correlation Coefficient through Variance Components," Biometrics, The International Biometric Society, vol. 59(4), pages 849-858, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Högberg, Hans & Svensson, Elisabeth, 2008. "An Overview of Methods in the Analysis of Dependent ordered catagorical Data: Assumptions and Implications," Working Papers 2008:7, Örebro University, School of Business.
    2. Debby L Gerritsen & Nardi Steverink & Dinnus HM Frijters & Marcel E Ooms & Miel W Ribbe, 2010. "Social well‐being and its measurement in the nursing home, the SWON‐scale," Journal of Clinical Nursing, John Wiley & Sons, vol. 19(9‐10), pages 1243-1251, May.
    3. N. Lu & T. Chen & P. Wu & D. Gunzler & H. Zhang & H. He & X.M. Tu, 2014. "Functional response models for intraclass correlation coefficients," Journal of Applied Statistics, Taylor & Francis Journals, vol. 41(11), pages 2539-2556, November.
    4. Christof Schuster & David Smith, 2005. "Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 135-146, March.
    5. Fabio Rapallo, 2005. "Algebraic exact inference for rater agreement models," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 14(1), pages 45-66, February.
    6. Cheng, Song-Show & Cheng, Yu-Chun, 1998. "An ordered relation between the ANOVA estimator of the intraclass correlation and a kappa-type statistic in binary data," Statistics & Probability Letters, Elsevier, vol. 38(3), pages 275-280, June.
    7. Jonas Moss, 2024. "Measures of Agreement with Multiple Raters: Fréchet Variances and Inference," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 517-541, June.
    8. Alexandra Raadt & Matthijs J. Warrens & Roel J. Bosker & Henk A. L. Kiers, 2021. "A Comparison of Reliability Coefficients for Ordinal Rating Scales," Journal of Classification, Springer;The Classification Society, vol. 38(3), pages 519-543, October.
    9. Högberg, Hans & Svensson, Elisabeth, 2008. "Comparison of methods in the analysis of dependent ordered catagorical data," Working Papers 2008:6, Örebro University, School of Business.
    10. Matthijs J. Warrens, 2021. "Kappa coefficients for dichotomous-nominal classifications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(1), pages 193-208, March.
    11. Helenowski Irene B & Vonesh Edward F & Demirtas Hakan & Rademaker Alfred W & Ananthanarayanan Vijayalakshmi & Gann Peter H & Jovanovic Borko D, 2011. "Defining Reproducibility Statistics as a Function of the Spatial Covariance Structures in Biomarker Studies," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-21, January.
    12. Somendra Narayan & Jatinder S. Sidhu & Henk W. Volberda, 2021. "From Attention to Action: The Influence of Cognitive and Ideological Diversity in Top Management Teams on Business Model Innovation," Journal of Management Studies, Wiley Blackwell, vol. 58(8), pages 2082-2110, December.
    13. Alexandre Rodrigues da Silva & Claudia Brito Silva Cirani & Fernando Antonio Ribeiro Serra & Angélica Pigola & Priscila Rezende da Costa & Isabel Cristina Scafuto & Roberto Lima Ruas & Marcos Rogério , 2023. "Determining Factors on Green Innovation Adoption: An Empirical Study in Brazilian Agribusiness Firms," Sustainability, MDPI, vol. 15(7), pages 1-23, April.
    14. Leyla Yılmaz Fındık & Şefika Şule Erçetin, 2023. "How Do Universities in Türkiye Integrate Sustainable Development Goals into Their Strategies?," Sustainability, MDPI, vol. 15(24), pages 1-16, December.
    15. Brennan, Michael & Rondón-Sulbarán, Janeet, 2019. "Transdisciplinary research: Exploring impact, knowledge and quality in the early stages of a sustainable development project," World Development, Elsevier, vol. 122(C), pages 481-491.
    16. Balakrishnan, Narayanaswamy & Ristić, Miroslav M., 2016. "Multivariate families of gamma-generated distributions with finite or infinite support above or below the diagonal," Journal of Multivariate Analysis, Elsevier, vol. 143(C), pages 194-207.
    17. Jason Wittenberg, 2013. "How similar are they? rethinking electoral congruence," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(3), pages 1687-1701, April.
    18. Andreas Armborst, 2017. "Thematic Proximity in Content Analysis," SAGE Open, , vol. 7(2), pages 21582440177, June.
    19. Stanislav Birko & Edward S Dove & Vural Özdemir, 2015. "Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-14, August.
    20. Geòrgia Escaramís & Josep L. Carrasco & Carlos Ascaso, 2008. "Detection of Significant Disease Risks Using a Spatial Conditional Autoregressive Model," Biometrics, The International Biometric Society, vol. 64(4), pages 1043-1053, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:advdac:v:19:y:2025:i:1:d:10.1007_s11634-024-00581-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.