IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v83y2018i1d10.1007_s11336-016-9525-x.html
   My bibliography  Save this article

A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing

Author

Listed:
  • Chun Wang

    () (University of Minnesota)

  • Gongjun Xu

    (University of Minnesota)

  • Zhuoran Shang

    (University of Minnesota)

Abstract

Abstract Statistical methods for identifying aberrances on psychological and educational tests are pivotal to detect flaws in the design of a test or irregular behavior of test takers. Two approaches have been taken in the past to address the challenge of aberrant behavior detection, which are (1) modeling aberrant behavior via mixture modeling methods, and (2) flagging aberrant behavior via residual based outlier detection methods. In this paper, we propose a two-stage method that is conceived of as a combination of both approaches. In the first stage, a mixture hierarchical model is fitted to the response and response time data to distinguish normal and aberrant behaviors using Markov chain Monte Carlo (MCMC) algorithm. In the second stage, a further distinction between rapid guessing and cheating behavior is made at a person level using a Bayesian residual index. Simulation results show that the two-stage method yields accurate item and person parameter estimates, as well as high true detection rate and low false detection rate, under different manipulated conditions mimicking NAEP parameters. A real data example is given in the end to illustrate the potential application of the proposed method.

Suggested Citation

  • Chun Wang & Gongjun Xu & Zhuoran Shang, 2018. "A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 223-254, March.
  • Handle: RePEc:spr:psycho:v:83:y:2018:i:1:d:10.1007_s11336-016-9525-x
    DOI: 10.1007/s11336-016-9525-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-016-9525-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    2. Wim J. van der Linden, 2006. "A Lognormal Model for Response Times on Test Items," Journal of Educational and Behavioral Statistics, , vol. 31(2), pages 181-204, June.
    3. Chun Wang & Zhewen Fan & Hua-Hua Chang & Jeffrey A. Douglas, 2013. "A Semiparametric Model for Jointly Analyzing Response Times and Accuracy in Computerized Testing," Journal of Educational and Behavioral Statistics, , vol. 38(4), pages 381-417, August.
    4. Yu-Wei Chang & Rung-Ching Tsai & Nan-Jung Hsu, 2014. "A Speeded Item Response Model: Leave the Harder till Later," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 255-274, April.
    5. Zhewen Fan & Chun Wang & Hua-Hua Chang & Jeffrey Douglas, 2012. "Utilizing Response Time Distributions for Item Selection in CAT," Journal of Educational and Behavioral Statistics, , vol. 37(5), pages 655-670, October.
    6. Jeffrey Rouder & Dongchu Sun & Paul Speckman & Jun Lu & Duo Zhou, 2003. "A hierarchical bayesian statistical framework for response time distributions," Psychometrika, Springer;The Psychometric Society, vol. 68(4), pages 589-606, December.
    7. Yuri Goegebeur & Paul Boeck & James Wollack & Allan Cohen, 2008. "A Speeded Item Response Model with Gradual Process Change," Psychometrika, Springer;The Psychometric Society, vol. 73(1), pages 65-87, March.
    8. Wim Linden & Edith Krimpen-Stoop, 2003. "Using response times to detect aberrant responses in computerized adaptive testing," Psychometrika, Springer;The Psychometric Society, vol. 68(2), pages 251-265, June.
    9. Zhan Shu & Robert Henson & Richard Luecht, 2013. "Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 481-497, July.
    10. Robert Mislevy & Norman Verhelst, 1990. "Modeling item responses when different subjects employ different solution strategies," Psychometrika, Springer;The Psychometric Society, vol. 55(2), pages 195-215, June.
    11. Daniel O. Segall, 2002. "An Item Response Model for Characterizing Test Compromise," Journal of Educational and Behavioral Statistics, , vol. 27(2), pages 163-179, June.
    12. Wim Linden & Fanmin Guo, 2008. "Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 73(3), pages 365-384, September.
    13. Michael V. Levine & Donald B. Rubin, 1979. "Measuring the Appropriateness of Multiple-Choice Test Scores," Journal of Educational and Behavioral Statistics, , vol. 4(4), pages 269-290, December.
    Full references (including those not matched with items on IDEAS)

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:83:y:2018:i:1:d:10.1007_s11336-016-9525-x. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Mallaigh Nolan). General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.