IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v47y2022i5p576-602.html

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings

Author

Listed:
  • Benjamin W. Domingue
  • Klint Kanopka
  • Ben Stenhaug
  • Michael J. Sulik

    (Stanford Graduate School of Education)

  • Tanesia Beverly
  • Matthieu Brinkhuis
  • Ruhan Circi
  • Jessica Faul
  • Dandan Liao
  • Bruce McCandliss
  • Jelena Obradović
  • Chris Piech
  • Tenelle Porter
  • Project iLEAD Consortium
  • James Soland
  • Jon Weeks
  • Steven L. Wise
  • Jason Yeatman

Abstract

The speed–accuracy trade-off (SAT) suggests that time constraints reduce response accuracy. Its relevance in observational settings—where response time (RT) may not be constrained but respondent speed may still vary—is unclear. Using 29 data sets containing data from cognitive tasks, we use a flexible method for identification of the SAT (which we test in extensive simulation studies) to probe whether the SAT holds. We find inconsistent relationships between time and accuracy; marginal increases in time use for an individual do not necessarily predict increases in accuracy. Additionally, the speed–accuracy relationship may depend on the underlying difficulty of the interaction. We also consider the analysis of items and individuals; of particular interest is the observation that respondents who exhibit more within-person variation in response speed are typically of lower ability. We further find that RT is typically a weak predictor of response accuracy. Our findings document a range of empirical phenomena that should inform future modeling of RTs collected in observational settings.

Suggested Citation

  • Benjamin W. Domingue & Klint Kanopka & Ben Stenhaug & Michael J. Sulik & Tanesia Beverly & Matthieu Brinkhuis & Ruhan Circi & Jessica Faul & Dandan Liao & Bruce McCandliss & Jelena Obradović & Chris , 2022. "Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings," Journal of Educational and Behavioral Statistics, , vol. 47(5), pages 576-602, October.
    • Handle: RePEc:sae:jedbes:v:47:y:2022:i:5:p:576-602
    DOI: 10.3102/10769986221099906
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986221099906
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986221099906?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    2. Chen, Haiqin & De Boeck, Paul & Grady, Matthew & Yang, Chien-Lin & Waldschmidt, David, 2018. "Curvilinear dependency of response accuracy on response time in cognitive tests," Intelligence, Elsevier, vol. 69(C), pages 16-23.
    3. De Boeck, Paul & Bakker, Marjan & Zwitser, Robert & Nivard, Michel & Hofman, Abe & Tuerlinckx, Francis & Partchev, Ivailo, 2011. "The Estimation of Item Response Models with the lmer Function from the lme4 Package in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i12).
    4. Molenaar, Dylan & Tuerlinckx, Francis & van der Maas, Han L. J., 2015. "Fitting Diffusion Item Response Theory Models for Responses and Response Times Using the R Package diffIRT," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 66(i04).
    5. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    6. Carol Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    7. Carol M. Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    8. Gunter Maris & Han Maas, 2012. "Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy," Psychometrika, Springer;The Psychometric Society, vol. 77(4), pages 615-633, October.
    9. Jochen Ranger & Jörg-Tobias Kuhn & José-Luis Gaviria, 2015. "A Race Model for Responses and Response Times in Tests," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 791-810, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Inhan Kang & Paul Boeck & Roger Ratcliff, 2022. "Modeling Conditional Dependence of Response Accuracy and Response Time with the Diffusion Item Response Theory Model," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 725-748, June.
    2. Joakim Wallmark & James O. Ramsay & Juan Li & Marie Wiberg, 2024. "Analyzing Polytomous Test Data: A Comparison Between an Information-Based IRT Model and the Generalized Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 49(5), pages 753-779, October.
    3. Peter W. Rijn & Usama S. Ali, 2018. "A Generalized Speed–Accuracy Response Model for Dichotomous Items," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 109-131, March.
    4. Inhan Kang & Minjeong Jeon & Ivailo Partchev, 2023. "A Latent Space Diffusion Item Response Theory Model to Explore Conditional Dependence between Responses and Response Times," Psychometrika, Springer;The Psychometric Society, vol. 88(3), pages 830-864, September.
    5. Yang Liu & Ji Seung Yang, 2018. "Bootstrap-Calibrated Interval Estimates for Latent Variable Scores in Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 333-354, June.
    6. Steven P. Reise & Han Du & Emily F. Wong & Anne S. Hubbard & Mark G. Haviland, 2021. "Matching IRT Models to Patient-Reported Outcomes Constructs: The Graded Response and Log-Logistic Models for Scaling Depression," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 800-824, September.
    7. Christopher J. Urban & Daniel J. Bauer, 2021. "A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 1-29, March.
    8. Kang, Inhan & De Boeck, Paul & Partchev, Ivailo, 2022. "A randomness perspective on intelligence processes," Intelligence, Elsevier, vol. 91(C).
    9. Xi Wang & Yang Liu, 2020. "Detecting Compromised Items Using Information From Secure Items," Journal of Educational and Behavioral Statistics, , vol. 45(6), pages 667-689, December.
    10. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    11. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    12. Yang Liu, 2020. "A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 439-468, June.
    13. Joshua B. Gilbert & James S. Kim & Luke W. Miratrix, 2023. "Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 889-913, December.
    14. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.
    15. Yi-Hsuan Lee & Zhiliang Ying, 2015. "A Mixture Cure-Rate Model for Responses and Response Times in Time-Limit Tests," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 748-775, September.
    16. J. R. Lockwood & Katherine E. Castellano & Benjamin R. Shear, 2018. "Flexible Bayesian Models for Inferences From Coarsened, Group-Level Achievement Data," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 663-692, December.
    17. Frederik Coomans & Abe Hofman & Matthieu Brinkhuis & Han L J van der Maas & Gunter Maris, 2016. "Distinguishing Fast and Slow Processes in Accuracy - Response Time Data," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-19, May.
    18. Shiyu Wang & Yinghan Chen, 2020. "Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models," Psychometrika, Springer;The Psychometric Society, vol. 85(3), pages 600-629, September.
    19. James O. Ramsay & Marie Wiberg, 2017. "A Strategy for Replacing Sum Scoring," Journal of Educational and Behavioral Statistics, , vol. 42(3), pages 282-307, June.
    20. Sun-Joo Cho & Sarah Brown-Schmidt & Paul De Boeck & Matthew Naveiras & Si On Yoon & Aaron Benjamin, 2023. "Incorporating Functional Response Time Effects into a Signal Detection Theory Model," Psychometrika, Springer;The Psychometric Society, vol. 88(3), pages 1056-1086, September.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:47:y:2022:i:5:p:576-602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.