IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v50y2025i3p449-470.html

Using Response Times in Answer Similarity Analysis

Author

Listed:
  • Kylie Gorney

    (Michigan State University)

  • James A. Wollack

    (University of Wisconsin-Madison)

Abstract

Recent decades have seen a tremendous growth in the development of collusion detection methods, many of which rest on the assumption that examinees who engage in collusion will display unusually similar scores/responses. In this article, we expand the definition of answer similarity to include not only the item scores/responses but also the item response times (RTs). Using detailed simulations and an experimental data set, we show that (a) both the new and existing similarity statistics are able to control the Type I error rate in most of the studied conditions and (b) the new statistics are much more powerful, on average, than the existing statistics at detecting several types of simulated collusion.

Suggested Citation

  • Kylie Gorney & James A. Wollack, 2025. "Using Response Times in Answer Similarity Analysis," Journal of Educational and Behavioral Statistics, , vol. 50(3), pages 449-470, June.
    • Handle: RePEc:sae:jedbes:v:50:y:2025:i:3:p:449-470
    DOI: 10.3102/10769986241248770
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986241248770
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986241248770?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    2. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    3. Joseph B. Lang, 2023. "A Randomization P-Value Test for Detecting Copying on Multiple-Choice Exams," Journal of Educational and Behavioral Statistics, , vol. 48(3), pages 296-319, June.
    4. Kylie Gorney & James A. Wollack & Sandip Sinharay & Carol Eckerly, 2023. "Using Item Scores and Distractors to Detect Item Compromise and Preknowledge," Journal of Educational and Behavioral Statistics, , vol. 48(5), pages 636-660, October.
    5. Daniel Bolt & James Wollack & Youngsuk Suh, 2012. "Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 339-357, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kylie Gorney & James A. Wollack & Sandip Sinharay & Carol Eckerly, 2023. "Using Item Scores and Distractors to Detect Item Compromise and Preknowledge," Journal of Educational and Behavioral Statistics, , vol. 48(5), pages 636-660, October.
    2. Gunter Maris & Han Maas, 2012. "Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy," Psychometrika, Springer;The Psychometric Society, vol. 77(4), pages 615-633, October.
    3. Anne Thissen-Roe & David Thissen, 2013. "A Two-Decision Model for Responses to Likert-Type Items," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 522-547, October.
    4. Benjamin W. Domingue & Klint Kanopka & Ben Stenhaug & Michael J. Sulik & Tanesia Beverly & Matthieu Brinkhuis & Ruhan Circi & Jessica Faul & Dandan Liao & Bruce McCandliss & Jelena Obradović & Chris , 2022. "Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings," Journal of Educational and Behavioral Statistics, , vol. 47(5), pages 576-602, October.
    5. Steven Andrew Culpepper & James Joseph Balamuta, 2017. "A Hierarchical Model for Accuracy and Choice on Standardized Tests," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 820-845, September.
    6. Luo, Nanyu & Ji, Feng & Han, Yuting & He, Jinbo & Zhang, Xiaoya, 2024. "Fitting item response theory models using deep learning computational frameworks," OSF Preprints tjxab, Center for Open Science.
    7. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    8. Inhan Kang & Paul Boeck & Roger Ratcliff, 2022. "Modeling Conditional Dependence of Response Accuracy and Response Time with the Diffusion Item Response Theory Model," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 725-748, June.
    9. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1996. "Polytomous IRT models and monotone likelihood ratio of the total score," Psychometrika, Springer;The Psychometric Society, vol. 61(4), pages 679-693, December.
    10. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    11. Sijia Huang & Li Cai, 2024. "Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 311-341, June.
    12. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    13. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    14. Jochen Ranger, 2013. "A Note on the Hierarchical Model for Responses and Response Times in Tests of van der Linden (2007)," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 538-544, July.
    15. Kylie Gorney & Sandip Sinharay & Carol Eckerly, 2024. "Efficient Corrections for Standardized Person-Fit Statistics," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 569-591, June.
    16. Ghattas Badih & Michel Pierre & Boyer Laurent, 2019. "Assessing variable importance in clustering: a new method based on unsupervised binary decision trees," Computational Statistics, Springer, vol. 34(1), pages 301-321, March.
    17. Yang Liu & Jan Hannig & Abhishek Pal Majumder, 2019. "Second-Order Probability Matching Priors for the Person Parameter in Unidimensional IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 701-718, September.
    18. Yang Liu & Weimeng Wang, 2024. "What Can We Learn from a Semiparametric Factor Analysis of Item Responses and Response Time? An Illustration with the PISA 2015 Data," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 386-410, June.
    19. Hsiao, Cheng & Sun, Bao-Hong, 1998. "Modeling survey response bias - with an analysis of the demand for an advanced electronic device," Journal of Econometrics, Elsevier, vol. 89(1-2), pages 15-39, November.
    20. Roderick McDonald, 1986. "Describing the elephant: Structure and function in multivariate data," Psychometrika, Springer;The Psychometric Society, vol. 51(4), pages 513-534, December.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:50:y:2025:i:3:p:449-470. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.