IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v86y2021i2d10.1007_s11336-021-09746-5.html
   My bibliography  Save this article

Differential Item Functioning Analysis Without A Priori Information on Anchor Items: QQ Plots and Graphical Test

Author

Listed:
  • Ke-Hai Yuan

    (University of Notre Dame)

  • Hongyun Liu

    (Beijing Normal University)

  • Yuting Han

    (Beijing Normal University)

Abstract

Differential item functioning (DIF) analysis is an important step in establishing the validity of measurements. Most traditional methods for DIF analysis use an item-by-item strategy via anchor items that are assumed DIF-free. If anchor items are flawed, these methods will yield misleading results due to biased scales. In this article, based on the fact that the item’s relative change of difficulty difference (RCD) does not depend on the mean ability of individual groups, a new DIF detection method (RCD-DIF) is proposed by comparing the observed differences against those with simulated data that are known DIF-free. The RCD-DIF method consists of a D-QQ (quantile quantile) plot that permits the identification of internal references points (similar to anchor items), a RCD-QQ plot that facilitates visual examination of DIF, and a RCD graphical test that synchronizes DIF analysis at the test level with that at the item level via confidence intervals on individual items. The RCD procedure visually reveals the overall pattern of DIF in the test and the size of DIF for each item and is expected to work properly even when the majority of the items possess DIF and the DIF pattern is unbalanced. Results of two simulation studies indicate that the RCD graphical test has Type I error rate comparable to those of existing methods but with greater power.

Suggested Citation

  • Ke-Hai Yuan & Hongyun Liu & Yuting Han, 2021. "Differential Item Functioning Analysis Without A Priori Information on Anchor Items: QQ Plots and Graphical Test," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 345-377, June.
    • Handle: RePEc:spr:psycho:v:86:y:2021:i:2:d:10.1007_s11336-021-09746-5
    DOI: 10.1007/s11336-021-09746-5
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-021-09746-5
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-021-09746-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robin Shealy & William Stout, 1993. "A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF," Psychometrika, Springer;The Psychometric Society, vol. 58(2), pages 159-194, June.
    2. Gerhard Tutz & Gunther Schauberger, 2015. "A Penalty Approach to Differential Item Functioning in Rasch Models," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 21-43, March.
    3. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    4. Carolin Strobl & Julia Kopf & Achim Zeileis, 2015. "Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 289-316, June.
    5. Choi, Seung W. & Gibbons, Laura E. & Crane, Paul K., 2011. "lordif: An R Package for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i08).
    6. Timo Bechger & Gunter Maris, 2015. "A Statistical Test for Differential Item Pair Functioning," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 317-340, June.
    7. Gerhard Tutz & Moritz Berger, 2016. "Item-focussed Trees for the Identification of Items in Differential Item Functioning," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 727-750, September.
    8. Yuan, Ke-Hai & Hayashi, Kentaro & Bentler, Peter M., 2007. "Normal theory likelihood ratio statistic for mean and covariance structure analysis under alternative hypotheses," Journal of Multivariate Analysis, Elsevier, vol. 98(6), pages 1262-1282, July.
    9. repec:ucp:bkecon:9780226316529 is not listed on IDEAS
    10. Yuan, Ke-Hai & Chan, Wai, 2008. "Structural equation modeling with near singular covariance matrices," Computational Statistics & Data Analysis, Elsevier, vol. 52(10), pages 4842-4858, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paul Boeck & Sun-Joo Cho, 2021. "Not all DIF is shaped similarly," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 712-716, September.
    2. Chen, Yunxiao & Li, Chengcheng & Ouyang, Jing & Xu, Gongjun, 2023. "DIF statistical inference without knowing anchoring items," LSE Research Online Documents on Economics 119923, London School of Economics and Political Science, LSE Library.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Yunxiao & Li, Chengcheng & Ouyang, Jing & Xu, Gongjun, 2023. "DIF statistical inference without knowing anchoring items," LSE Research Online Documents on Economics 119923, London School of Economics and Political Science, LSE Library.
    2. Ting Wang & Carolin Strobl & Achim Zeileis & Edgar C. Merkle, 2018. "Score-Based Tests of Differential Item Functioning via Pairwise Maximum Likelihood Estimation," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 132-155, March.
    3. Jeanne A. Teresi & Chun Wang & Marjorie Kleinman & Richard N. Jones & David J. Weiss, 2021. "Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS®) Measures: Methods, Challenges, Advances, and Future Directions," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 674-711, September.
    4. Cervantes, Víctor H., 2017. "DFIT: An R Package for Raju's Differential Functioning of Items and Tests Framework," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i05).
    5. Alexander Robitzsch, 2020. "L p Loss Functions in Invariance Alignment and Haberman Linking with Few or Many Groups," Stats, MDPI, vol. 3(3), pages 1-38, August.
    6. Benjamin D. Schalet & Sangdon Lim & David Cella & Seung W. Choi, 2021. "Linking Scores with Patient-Reported Health Outcome Instruments:A VALIDATION STUDY AND COMPARISON OF THREE LINKING METHODS," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 717-746, September.
    7. Gerhard Tutz & Moritz Berger, 2016. "Item-focussed Trees for the Identification of Items in Differential Item Functioning," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 727-750, September.
    8. Rikkert M. van der Lans & Ridwan Maulana & Michelle Helms-Lorenz & Carmen-María Fernández-García & Seyeoung Chun & Thelma de Jager & Yulia Irnidayanti & Mercedes Inda-Caro & Okhwa Lee & Thys Coetze, 2021. "Student Perceptions of Teaching Quality in Five Countries: A Partial Credit Model Approach to Assess Measurement Invariance," SAGE Open, , vol. 11(3), pages 21582440211, August.
    9. Kai Liu & Longfei Zhang & Dongbo Tu & Yan Cai, 2022. "Developing an Item Bank of Computerized Adaptive Testing for Eating Disorders in Chinese University Students," SAGE Open, , vol. 12(4), pages 21582440221, December.
    10. Gerhard Tutz, 2022. "Item Response Thresholds Models: A General Class of Models for Varying Types of Items," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1238-1269, December.
    11. Paul Boeck & Sun-Joo Cho, 2021. "Not all DIF is shaped similarly," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 712-716, September.
    12. K. B. S. Huth & L. J. Waldorp & J. Luigjes & A. E. Goudriaan & R. J. Holst & M. Marsman, 2022. "A Note on the Structural Change Test in Highly Parameterized Psychometric Models," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1064-1080, September.
    13. Ting Wang & Benjamin Graves & Yves Rosseel & Edgar C. Merkle, 2022. "Computation and application of generalized linear mixed model derivatives using lme4," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1173-1193, September.
    14. Ting Wang & Carolin Strobl & Achim Zeileis & Edgar C. Merkle, 2016. "Score-Based Tests of Differential Item Functioning in the Two-Parameter Model," Working Papers 2016-05, Faculty of Economics and Statistics, Universität Innsbruck.
    15. Hao Wu & Michael Browne, 2015. "Random Model Discrepancy: Interpretations and Technicalities (A Rejoinder)," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 619-624, September.
    16. Yuan, Ke-Hai & Chan, Wai, 2008. "Structural equation modeling with near singular covariance matrices," Computational Statistics & Data Analysis, Elsevier, vol. 52(10), pages 4842-4858, June.
    17. Melissa Gladstone & Gillian Lancaster & Gareth McCray & Vanessa Cavallera & Claudia R. L. Alves & Limbika Maliwichi & Muneera A. Rasheed & Tarun Dua & Magdalena Janus & Patricia Kariger, 2021. "Validation of the Infant and Young Child Development (IYCD) Indicators in Three Countries: Brazil, Malawi and Pakistan," IJERPH, MDPI, vol. 18(11), pages 1-19, June.
    18. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    19. Victoria T. Tanaka & George Engelhard & Matthew P. Rabbitt, 2020. "Using a Bifactor Model to Measure Food Insecurity in Households with Children," Journal of Family and Economic Issues, Springer, vol. 41(3), pages 492-504, September.
    20. Çetin Toraman & Güneş Korkmaz, 2023. "What is the “Meaning of School†to High School Students? A Scale Development and Implementation Study Based on IRT and CTT," SAGE Open, , vol. 13(3), pages 21582440231, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:86:y:2021:i:2:d:10.1007_s11336-021-09746-5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.