IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v56y2012i5p1052-1060.html
   My bibliography  Save this article

Assessment of observer agreement for matched repeated binary measurements

Author

Listed:
  • Gao, Jingjing
  • Pan, Yi
  • Haber, Michael

Abstract

Agreement is a broad term simultaneously covering evaluations of accuracy and precision of measurements. Assessment of observer agreement is based on the similarity between readings made on the same subject by different observers. The assessment of agreement on categorical observations is traditionally based on kappa or weighted kappa coefficients. However, kappa statistics have been criticized because they attain implausible values when the marginal distributions are skewed and/or unbalanced. New scaled indices called the coefficients of individual agreement (CIAs) have been developed for the assessment of individual observer agreement by comparing the observed disagreement between two observers to the disagreement between replicated observations made by the same observer on the same subject. This is based on the notion that under a good agreement, the disagreement between the two observers is usually not expected to exceed the disagreement between replicated observations of the same observer, and hence, a satisfactory agreement is established if these quantities are similar. This idea is extended and a new method utilizing the generalized linear mixed model is proposed to estimate the CIAs for binary data which consist of matched sets of repeated measurements made by the same observer under different conditions. The conditions may represent different time points, raters, laboratories, treatments, etc. The new approach allows the values of the measured variable and the magnitude of agreement to vary across the conditions. The reliability of the estimation method is examined via a simulation study. Data from a study aiming at determining the validity of diagnosis of breast cancer based on mammography are used to illustrate the new concepts and methods.

Suggested Citation

  • Gao, Jingjing & Pan, Yi & Haber, Michael, 2012. "Assessment of observer agreement for matched repeated binary measurements," Computational Statistics & Data Analysis, Elsevier, vol. 56(5), pages 1052-1060.
    • Handle: RePEc:eee:csdana:v:56:y:2012:i:5:p:1052-1060
    DOI: 10.1016/j.csda.2011.11.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947311004026
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2011.11.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Huiman X. Barnhart & Michael Haber & Jingli Song, 2002. "Overall Concordance Correlation Coefficient for Evaluating Agreement Among Multiple Observers," Biometrics, The International Biometric Society, vol. 58(4), pages 1020-1027, December.
    2. Helena Kraemer, 1979. "Ramifications of a population model forκ as a coefficient of reliability," Psychometrika, Springer;The Psychometric Society, vol. 44(4), pages 461-472, December.
    3. Josep L. Carrasco, 2010. "A Generalized Concordance Correlation Coefficient Based on the Variance Components Generalized Linear Mixed Models for Overdispersed Count Data," Biometrics, The International Biometric Society, vol. 66(3), pages 897-904, September.
    4. Lin L. & Hedayat A. S. & Sinha B. & Yang M., 2002. "Statistical Methods in Assessing Agreement: Models, Issues, and Tools," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 257-270, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tsai, Miao-Yu, 2015. "Comparison of concordance correlation coefficient via variance components, generalized estimating equations and weighted approaches with model selection," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 47-58.
    2. Chen, Chia-Cheng & Barnhart, Huiman X., 2008. "Comparison of ICC and CCC for assessing agreement for data without and with replications," Computational Statistics & Data Analysis, Elsevier, vol. 53(2), pages 554-564, December.
    3. Tsai, Miao-Yu & Lin, Chao-Chun, 2018. "Concordance correlation coefficients estimated by variance components for longitudinal normal and Poisson data," Computational Statistics & Data Analysis, Elsevier, vol. 121(C), pages 57-70.
    4. Matthijs Warrens, 2010. "A Formal Proof of a Paradox Associated with Cohen’s Kappa," Journal of Classification, Springer;The Classification Society, vol. 27(3), pages 322-332, November.
    5. Jose M. Jimenez-Olmedo & Alfonso Penichet-Tomas & Basilio Pueo & Lamberto Villalon-Gasch, 2023. "Reliability of ADR Jumping Photocell: Comparison of Beam Cut at Forefoot and Midfoot," IJERPH, MDPI, vol. 20(11), pages 1-13, May.
    6. Matthijs J. Warrens, 2021. "Kappa coefficients for dichotomous-nominal classifications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(1), pages 193-208, March.
    7. Helenowski Irene B & Vonesh Edward F & Demirtas Hakan & Rademaker Alfred W & Ananthanarayanan Vijayalakshmi & Gann Peter H & Jovanovic Borko D, 2011. "Defining Reproducibility Statistics as a Function of the Spatial Covariance Structures in Biomarker Studies," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-21, January.
    8. Ying Guo & Amita K. Manatunga, 2007. "Nonparametric Estimation of the Concordance Correlation Coefficient under Univariate Censoring," Biometrics, The International Biometric Society, vol. 63(1), pages 164-172, March.
    9. Tarald O. Kvålseth, 2018. "An Alternative Interpretation of the Linearly Weighted Kappa Coefficients for Ordinal Data," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 618-627, September.
    10. Yang, Zhao & Zhou, Ming, 2015. "Kappa statistic for clustered physician–patients polytomous data," Computational Statistics & Data Analysis, Elsevier, vol. 87(C), pages 1-17.
    11. Liao Jason J. Z. & Capen Robert, 2011. "An Improved Bland-Altman Method for Concordance Assessment," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-17, January.
    12. Hutson, Alan D., 2010. "A multi-rater nonparametric test of agreement and corresponding agreement plot," Computational Statistics & Data Analysis, Elsevier, vol. 54(1), pages 109-119, January.
    13. Miao-Yu Tsai, 2012. "Assessing inter- and intra-agreement for dependent binary data: a Bayesian hierarchical correlation approach," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(1), pages 173-187, March.
    14. Masha Kocherginsky & Megan Huisingh-Scheetz & William Dale & Diane S Lauderdale & Linda Waite, 2017. "Measuring Physical Activity with Hip Accelerometry among U.S. Older Adults: How Many Days Are Enough?," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-13, January.
    15. Correndo, Adrian A. & Hefley, Trevor J. & Holzworth, Dean P. & Ciampitti, Ignacio A., 2021. "Revisiting linear regression to test agreement in continuous predicted-observed datasets," Agricultural Systems, Elsevier, vol. 192(C).
    16. Matthijs J. Warrens, 2014. "New Interpretations of Cohen’s Kappa," Journal of Mathematics, Hindawi, vol. 2014, pages 1-9, September.
    17. S. Vanbelle & A. Albert, 2009. "Agreement between an isolated rater and a group of raters," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 63(1), pages 82-100, February.
    18. Dejana Bouillet & Sandra Antulić Majcen, 2022. "Risks of Social Exclusion Among Children in ECEC Settings: Assessments by Parents and ECEC Teachers," SAGE Open, , vol. 12(3), pages 21582440221, September.
    19. Admassu N. Lamu, 2020. "Does linear equating improve prediction in mapping? Crosswalking MacNew onto EQ-5D-5L value sets," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 21(6), pages 903-915, August.
    20. John J. Chen & Guangxiang Zhang & Chen Ji & George F. Steinhardt, 2011. "Simple moment-based inferences of generalized concordance correlation," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(9), pages 1867-1882, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:56:y:2012:i:5:p:1052-1060. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.