IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v48y2023i5p636-660.html

Using Item Scores and Distractors to Detect Item Compromise and Preknowledge

Author

Listed:
  • Kylie Gorney
  • James A. Wollack

    (University of Wisconsin-Madison)

  • Sandip Sinharay
  • Carol Eckerly

    (Educational Testing Service)

Abstract

Any time examinees have had access to items and/or answers prior to taking a test, the fairness of the test and validity of test score interpretations are threatened. Therefore, there is a high demand for procedures to detect both compromised items (CI) and examinees with preknowledge (EWP). In this article, we develop a procedure that uses item scores and distractors to simultaneously detect CI and EWP. The false positive rate and true positive rate are evaluated for both items and examinees using detailed simulations. A real data example is also provided using data from an information technology certification exam.

Suggested Citation

  • Kylie Gorney & James A. Wollack & Sandip Sinharay & Carol Eckerly, 2023. "Using Item Scores and Distractors to Detect Item Compromise and Preknowledge," Journal of Educational and Behavioral Statistics, , vol. 48(5), pages 636-660, October.
    • Handle: RePEc:sae:jedbes:v:48:y:2023:i:5:p:636-660
    DOI: 10.3102/10769986231159923
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231159923
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986231159923?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Xi Wang & Yang Liu, 2020. "Detecting Compromised Items Using Information From Secure Items," Journal of Educational and Behavioral Statistics, , vol. 45(6), pages 667-689, December.
    2. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    3. Insu Paek & Paul Holland, 2015. "A Note on Statistical Hypothesis Testing Based on Log Transformation of the Mantel–Haenszel Common Odds Ratio for Differential Item Functioning Classification," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 406-411, June.
    4. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    5. Chen, Yunxiao & Lu, Yan & Moustaki, Irini, 2022. "Detection of two-way outliers in multivariate data and application to cheating detection in educational tests," LSE Research Online Documents on Economics 112499, London School of Economics and Political Science, LSE Library.
    6. Daniel Bolt & James Wollack & Youngsuk Suh, 2012. "Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 339-357, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kylie Gorney & James A. Wollack, 2025. "Using Response Times in Answer Similarity Analysis," Journal of Educational and Behavioral Statistics, , vol. 50(3), pages 449-470, June.
    2. Kylie Gorney & Sandip Sinharay & Carol Eckerly, 2024. "Efficient Corrections for Standardized Person-Fit Statistics," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 569-591, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Luo, Nanyu & Ji, Feng & Han, Yuting & He, Jinbo & Zhang, Xiaoya, 2024. "Fitting item response theory models using deep learning computational frameworks," OSF Preprints tjxab, Center for Open Science.
    2. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    3. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    4. Yang Liu & Jan Hannig, 2017. "Generalized Fiducial Inference for Logistic Graded Response Models," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1097-1125, December.
    5. Yingbin Zhang & Zhaoxi Yang & Yehui Wang, 2022. "The Impact of Extreme Response Style on the Mean Comparison of Two Independent Samples," SAGE Open, , vol. 12(2), pages 21582440221, June.
    6. Carmen Köhler & Alexander Robitzsch & Johannes Hartig, 2020. "A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 251-273, June.
    7. James O. Ramsay & Juan Li & Charles N. Bernstein & Ruth Ann Marrie & Joakim Wallmark & Marie Wiberg, 2026. "Using the Information Metric to Analyze Clinical Rating Scales," Journal of Educational and Behavioral Statistics, , vol. 51(2), pages 395-418, April.
    8. Daphna Harel & Russell J. Steele, 2018. "An Information Matrix Test for the Collapsing of Categories Under the Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 721-750, December.
    9. Kylie Gorney & James A. Wollack, 2025. "Using Response Times in Answer Similarity Analysis," Journal of Educational and Behavioral Statistics, , vol. 50(3), pages 449-470, June.
    10. repec:osf:osfxxx:tjxab_v1 is not listed on IDEAS
    11. Youngjin Han & Ji Seung Yang & Yang Liu, 2025. "Assessing Item Fit Using Expected Score Curve Under Restricted Recalibration," Journal of Educational and Behavioral Statistics, , vol. 50(5), pages 896-923, October.
    12. Myszkowski, Nils & Storme, Martin, 2018. "A snapshot of g? Binary and polytomous item-response theory investigations of the last series of the Standard Progressive Matrices (SPM-LS)," Intelligence, Elsevier, vol. 68(C), pages 109-116.
    13. Peter W. Rijn & Usama S. Ali & Hyo Jeong Shin & Sean-Hwane Joo, 2024. "Adjusted Residuals for Evaluating Conditional Independence in IRT Models for Multistage Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 89(1), pages 317-346, March.
    14. Anne Thissen-Roe & David Thissen, 2013. "A Two-Decision Model for Responses to Likert-Type Items," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 522-547, October.
    15. Brzezińska Justyna, 2018. "Item Response Theory Models in the Measurement Theory with the Use of ltm Package in R," Econometrics. Advances in Applied Data Analysis, Sciendo, vol. 22(1), pages 11-25, March.
    16. Sandip Sinharay, 2018. "Detecting Fraudulent Erasures at an Aggregate Level," Journal of Educational and Behavioral Statistics, , vol. 43(3), pages 286-315, June.
    17. Izolda Pristojkovic Suko & Magdalena Holter & Erwin Stolz & Elfriede Renate Greimel & Wolfgang Freidl, 2022. "Acculturation, Adaptation, and Health among Croatian Migrants in Austria and Ireland: A Cross-Sectional Study," IJERPH, MDPI, vol. 19(24), pages 1-15, December.
    18. Benjamin W. Domingue & Klint Kanopka & Ben Stenhaug & Michael J. Sulik & Tanesia Beverly & Matthieu Brinkhuis & Ruhan Circi & Jessica Faul & Dandan Liao & Bruce McCandliss & Jelena Obradović & Chris , 2022. "Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings," Journal of Educational and Behavioral Statistics, , vol. 47(5), pages 576-602, October.
    19. Nana Kim & Daniel M. Bolt & James Wollack, 2022. "Noncompensatory MIRT For Passage-Based Tests," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 992-1009, September.
    20. Mi Jung Lee & Daejin Kim & Sergio Romero & Ickpyo Hong & Nikolay Bliznyuk & Craig Velozo, 2022. "Examining Older Adults’ Home Functioning Using the American Housing Survey," IJERPH, MDPI, vol. 19(8), pages 1-13, April.
    21. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:48:y:2023:i:5:p:636-660. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.