IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v35y2010i6p744-761.html
   My bibliography  Save this article

Prevalence and Magnitude of Paradoxical Results in Multidimensional Item Response Theory

Author

Listed:
  • Matthew D. Finkelman
  • Giles Hooker
  • Zhen Wang

Abstract

Multidimensional Item Response Theory (MIRT) has been proposed as a means to model the relation between examinee abilities and test responses. Three recent articles proved that when MIRT is used in ability estimation, an examinee’s score could theoretically decrease due to a correct answer or increase due to an incorrect answer. The current article examines the extent to which such “paradoxical results†can arise in practice. In an operational test designed to measure two dimensions, a substantial percentage of paradoxical results occurred when using a MIRT model with a prior correlation of 0 between abilities. Assuming a positive correlation between abilities reduced the prevalence of paradoxical results but did not eliminate them entirely. Associated issues in test fairness are discussed.

Suggested Citation

  • Matthew D. Finkelman & Giles Hooker & Zhen Wang, 2010. "Prevalence and Magnitude of Paradoxical Results in Multidimensional Item Response Theory," Journal of Educational and Behavioral Statistics, , vol. 35(6), pages 744-761, December.
    • Handle: RePEc:sae:jedbes:v:35:y:2010:i:6:p:744-761
    DOI: 10.3102/1076998610381402
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998610381402
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998610381402?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    2. Bernard Veldkamp & Wim Linden, 2002. "Multidimensional adaptive testing with constraints on test content," Psychometrika, Springer;The Psychometric Society, vol. 67(4), pages 575-588, December.
    3. Daniel Segall, 1996. "Multidimensional adaptive testing," Psychometrika, Springer;The Psychometric Society, vol. 61(2), pages 331-354, June.
    4. Giles Hooker, 2010. "On Separable Tests, Correlated Priors, and Paradoxical Results in Multidimensional Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 694-707, December.
    5. David Thissen & Lynne Steinberg, 1984. "A response model for multiple choice items," Psychometrika, Springer;The Psychometric Society, vol. 49(4), pages 501-519, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pascal Jordan & Martin Spiess, 2018. "On Fair Person Classification Based on Efficient Factor Score Estimates in the Multidimensional Factor Analysis Model," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 563-585, September.
    2. Pascal Jordan & Martin Spiess, 2018. "A New Explanation and Proof of the Paradoxical Scoring Results in Multidimensional Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 831-846, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chun Wang & David J. Weiss & Zhuoran Shang, 2019. "Variable-Length Stopping Rules for Multidimensional Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 749-771, September.
    2. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    3. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    4. Chun Wang & Hua-Hua Chang & Keith Boughton, 2011. "Kullback–Leibler Information and Its Applications in Multi-Dimensional Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 76(1), pages 13-39, January.
    5. Lihua Yao, 2012. "Multidimensional CAT Item Selection Methods for Domain Scores and Composite Scores: Theory and Applications," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 495-523, July.
    6. Ulf Böckenholt, 2012. "The Cognitive-Miser Response Model: Testing for Intuitive and Deliberate Reasoning," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 388-399, April.
    7. Irini Moustaki & Martin Knott, 2000. "Generalized latent trait models," Psychometrika, Springer;The Psychometric Society, vol. 65(3), pages 391-411, September.
    8. Daniel Bolt & James Wollack & Youngsuk Suh, 2012. "Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 339-357, April.
    9. Dylan Molenaar, 2015. "Heteroscedastic Latent Trait Models for Dichotomous Data," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 625-644, September.
    10. Pascal Jordan, 2023. "On Reverse Shrinkage Effects and Shrinkage Overshoot," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 274-301, March.
    11. Chun Wang & Hua-Hua Chang, 2011. "Item Selection in Multidimensional Computerized Adaptive Testing—Gaining Information from Different Angles," Psychometrika, Springer;The Psychometric Society, vol. 76(3), pages 363-384, July.
    12. Hua-Hua Chang, 2015. "Psychometrics Behind Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 1-20, March.
    13. Michal Abrahamowicz & James Ramsay, 1992. "Multicategorical spline model for item response theory," Psychometrika, Springer;The Psychometric Society, vol. 57(1), pages 5-27, March.
    14. Henk Kelderman & Carl Rijkes, 1994. "Loglinear multidimensional IRT models for polytomously scored items," Psychometrika, Springer;The Psychometric Society, vol. 59(2), pages 149-176, June.
    15. Javier Revuelta, 2010. "Estimating Difficulty from Polytomous Categorical Data," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 331-350, June.
    16. Joris Mulder & Wim Linden, 2009. "Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection," Psychometrika, Springer;The Psychometric Society, vol. 74(2), pages 273-296, June.
    17. David Budescu & Yuanchao Bo, 2015. "Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 1105-1122, December.
    18. Mariagiulia Matteucci & Bernard Veldkamp, 2013. "On the use of MCMC computerized adaptive testing with empirical prior information to improve efficiency," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 22(2), pages 243-267, June.
    19. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    20. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1996. "Polytomous IRT models and monotone likelihood ratio of the total score," Psychometrika, Springer;The Psychometric Society, vol. 61(4), pages 679-693, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:35:y:2010:i:6:p:744-761. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.