IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v87y2022i4d10.1007_s11336-022-09847-9.html
   My bibliography  Save this article

The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment

Author

Listed:
  • Stephen R. Martin

    (University of California, Davis)

  • Philippe Rast

    (University of California, Davis)

Abstract

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a “Reliability Factor” that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

Suggested Citation

  • Stephen R. Martin & Philippe Rast, 2022. "The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1318-1342, December.
    • Handle: RePEc:spr:psycho:v:87:y:2022:i:4:d:10.1007_s11336-022-09847-9
    DOI: 10.1007/s11336-022-09847-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-022-09847-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-022-09847-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ian Brunton-Smith & Patrick Sturgis & George Leckie, 2017. "Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location–scale model," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(2), pages 551-568, February.
    2. Lewandowski, Daniel & Kurowicka, Dorota & Joe, Harry, 2009. "Generating random correlation matrices based on vines and extended onion method," Journal of Multivariate Analysis, Elsevier, vol. 100(9), pages 1989-2001, October.
    3. Carpenter, Bob & Gelman, Andrew & Hoffman, Matthew D. & Lee, Daniel & Goodrich, Ben & Betancourt, Michael & Brubaker, Marcus & Guo, Jiqiang & Li, Peter & Riddell, Allen, 2017. "Stan: A Probabilistic Programming Language," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i01).
    4. Rothenberg, Thomas J, 1971. "Identification in Parametric Models," Econometrica, Econometric Society, vol. 39(3), pages 577-591, May.
    5. Youngjo Lee & John A. Nelder, 2006. "Double hierarchical generalized linear models (with discussion)," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 55(2), pages 139-185, April.
    6. William Meredith, 1993. "Measurement invariance, factor analysis and factorial invariance," Psychometrika, Springer;The Psychometric Society, vol. 58(4), pages 525-543, December.
    7. Peter Bentler, 2009. "Alpha, Dimension-Free, and Model-Based Internal Consistency Reliability," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 137-143, March.
    8. Donald Hedeker & Robin J. Mermelstein & Hakan Demirtas, 2008. "An Application of a Mixed-Effects Location Scale Model for Analysis of Ecological Momentary Assessment (EMA) Data," Biometrics, The International Biometric Society, vol. 64(2), pages 627-634, June.
    9. Jules Ellis & Arnold Wollenberg, 1993. "Local homogeneity in latent trait models. A characterization of the homogeneous monotone irt model," Psychometrika, Springer;The Psychometric Society, vol. 58(3), pages 417-429, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Olbrich, Lukas & Kosyakova, Yuliya & Sakshaug, Joseph W., 2022. "The reliability of adult self-reported height: The role of interviewers," Economics & Human Biology, Elsevier, vol. 45(C).
    2. Louis Charlot, 2021. "Bayesian hierarchical analysis of a multifaceted program against extreme poverty," Papers 2109.06759, arXiv.org.
    3. Dellaportas, Petros & Titsias, Michalis K. & Petrova, Katerina & Plataniotis, Anastasios, 2023. "Scalable inference for a full multivariate stochastic volatility model," Journal of Econometrics, Elsevier, vol. 232(2), pages 501-520.
    4. Guowen Huang & Patrick E. Brown & Sze Hang Fu & Hwashin Hyun Shin, 2022. "Daily mortality/morbidity and air quality: Using multivariate time series with seasonally varying covariances," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(1), pages 148-174, January.
    5. Shelley A. Blozis, 2022. "A Latent Variable Mixed-Effects Location Scale Model with an Application to Daily Diary Data," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1548-1570, December.
    6. Peter Tea & Tim B. Swartz, 2023. "The analysis of serve decisions in tennis using Bayesian hierarchical models," Annals of Operations Research, Springer, vol. 325(1), pages 633-648, June.
    7. George Leckie & Robert French & Chris Charlton & William Browne, 2014. "Modeling Heterogeneous Variance–Covariance Components in Two-Level Models," Journal of Educational and Behavioral Statistics, , vol. 39(5), pages 307-332, October.
    8. Trung Dung Tran & Emmanuel Lesaffre & Geert Verbeke & Joke Duyck, 2021. "Latent Ornstein‐Uhlenbeck models for Bayesian analysis of multivariate longitudinal categorical responses," Biometrics, The International Biometric Society, vol. 77(2), pages 689-701, June.
    9. Lu, Rong, 2020. "Application of machine learning to gas flaring," Thesis Commons g6yvq, Center for Open Science.
    10. Rico Krueger & Taha H. Rashidi & Akshay Vij, 2020. "X vs. Y: an analysis of intergenerational differences in transport mode use among young adults," Transportation, Springer, vol. 47(5), pages 2203-2231, October.
    11. Matthias Breuer & Harm H. Schütt, 2023. "Accounting for uncertainty: an application of Bayesian methods to accruals models," Review of Accounting Studies, Springer, vol. 28(2), pages 726-768, June.
    12. Esther Ulitzsch & Steffi Pohl & Lale Khorramdel & Ulf Kroehne & Matthias Davier, 2022. "A Response-Time-Based Latent Response Mixture Model for Identifying and Modeling Careless and Insufficient Effort Responding in Survey Data," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 593-619, June.
    13. Jeanne A. Teresi & Katja Ocepek-Welikson & John A. Toner & Marjorie Kleinman & Mildred Ramirez & Joseph P. Eimicke & Barry J. Gurland & Albert Siu, 2017. "Methodological Issues in Measuring Subjective Well-Being and Quality-of-Life: Applications to Assessment of Affect in Older, Chronically and Cognitively Impaired, Ethnically Diverse Groups Using the F," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 12(2), pages 251-288, June.
    14. Hendrik Thiel & Stephan L. Thomsen & Bettina Büttner, 2014. "Variation of learning intensity in late adolescence and the effect on personality traits," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 177(4), pages 861-892, October.
    15. Johan Oud & Manuel Voelkle, 2014. "Do missing values exist? Incomplete data handling in cross-national longitudinal studies by means of continuous time modeling," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(6), pages 3271-3288, November.
    16. Liat Ayalon, 2018. "Perceived Age Discrimination: A Precipitator or a Consequence of Depressive Symptoms?," The Journals of Gerontology: Series B, The Gerontological Society of America, vol. 73(5), pages 860-869.
    17. Kocięcki, Andrzej & Kolasa, Marcin, 2023. "A solution to the global identification problem in DSGE models," Journal of Econometrics, Elsevier, vol. 236(2).
    18. Carvalho Lopes, Celia Mendes & Bolfarine, Heleno, 2012. "Random effects in promotion time cure rate models," Computational Statistics & Data Analysis, Elsevier, vol. 56(1), pages 75-87, January.
    19. Elisabeth Beckmann & Lukas Olbrich & Joseph Sakshaug, 2024. "Multivariate assessment of interviewer-related errors in a cross-national economic survey (Lukas Olbrich, Elisabeth Beckmann, Joseph W. Sakshaug)," Working Papers 253, Oesterreichische Nationalbank (Austrian Central Bank).
    20. Francis,David C. & Kubinec ,Robert, 2022. "Beyond Political Connections : A Measurement Model Approach to Estimating Firm-levelPolitical Influence in 41 Economies," Policy Research Working Paper Series 10119, The World Bank.

    More about this item

    Keywords

    Omega; Bayesian; reliability;
    All these keywords.

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:87:y:2022:i:4:d:10.1007_s11336-022-09847-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.