IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v12y2022i2p21582440221108168.html
   My bibliography  Save this article

The Impact of Extreme Response Style on the Mean Comparison of Two Independent Samples

Author

Listed:
  • Yingbin Zhang
  • Zhaoxi Yang
  • Yehui Wang

Abstract

Extreme response style (ERS) is prevalent in survey research using rating scales. It may cause biased results in group comparisons. This research conducted two sets of simulation studies to explore the magnitude of the ERS impact on mean comparisons between two independent samples. Data were generated from a multidimensional nominal response model. Study 1 examined the influence of ERS on the estimate of group differences in the variable of interest. The results indicated that ERS led to biased estimates, especially when these groups differed significantly in ERS. The correlation between ERS and the variable of interest also moderated the ERS impact. The results were illustrated with an empirical example. Study 2 investigated the impact of ERS on the type I error and type II error in the independent t -test based on scale scores. When the variable of interest had no true difference between groups, ERS inflated the type I error rate . When the difference existed, ERS inflated the type II error rate . Two groups’ true difference in ERS and the variable of interest, unequal ERS variances, the correlation between ERS and the variable of interest, and the number of items moderated the impact of ERS on type I and II error rates. The implications for practices and further research are discussed.

Suggested Citation

  • Yingbin Zhang & Zhaoxi Yang & Yehui Wang, 2022. "The Impact of Extreme Response Style on the Mean Comparison of Two Independent Samples," SAGE Open, , vol. 12(2), pages 21582440221, June.
    • Handle: RePEc:sae:sagope:v:12:y:2022:i:2:p:21582440221108168
    DOI: 10.1177/21582440221108168
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/21582440221108168
    Download Restriction: no
    File URL: https://libkey.io/10.1177/21582440221108168?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. David Thissen & Lynne Steinberg, 1986. "A taxonomy of item response models," Psychometrika, Springer;The Psychometric Society, vol. 51(4), pages 567-577, December.
    2. Hoffmann, Stefan & Mai, Robert & Cristescu, Anamaria, 2013. "Do culture-dependent response styles distort substantial relationships?," International Business Review, Elsevier, vol. 22(5), pages 814-827.
    3. De Boeck, Paul & Partchev, Ivailo, 2012. "IRTrees: Tree-Based Item Response Models of the GLMM Family," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(c01).
    4. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    5. Mingnan Liu & Frederick G. Conrad & Sunghee Lee, 2017. "Comparing acquiescent and extreme response styles in face-to-face and web surveys," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(2), pages 941-958, March.
    6. Ke-Hai Yuan & Peter Bentler, 2006. "Mean Comparison: Manifest Variable Versus Latent Variable," Psychometrika, Springer;The Psychometric Society, vol. 71(1), pages 139-159, March.
    7. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    8. Natalia Kieruj & Guy Moors, 2013. "Response style behavior: question format dependent or personal style?," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(1), pages 193-211, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anne Thissen-Roe & David Thissen, 2013. "A Two-Decision Model for Responses to Likert-Type Items," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 522-547, October.
    2. Nana Kim & Daniel M. Bolt & James Wollack, 2022. "Noncompensatory MIRT For Passage-Based Tests," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 992-1009, September.
    3. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    4. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1996. "Polytomous IRT models and monotone likelihood ratio of the total score," Psychometrika, Springer;The Psychometric Society, vol. 61(4), pages 679-693, December.
    5. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    6. Gerhard Tutz & Moritz Berger, 2016. "Response Styles in Rating Scales," Journal of Educational and Behavioral Statistics, , vol. 41(3), pages 239-268, June.
    7. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    8. Albert Maydeu-Olivares & Harry Joe, 2006. "Limited Information Goodness-of-fit Testing in Multidimensional Contingency Tables," Psychometrika, Springer;The Psychometric Society, vol. 71(4), pages 713-732, December.
    9. Brooke E. Magnus & David Thissen, 2017. "Item Response Modeling of Multivariate Count Data With Zero Inflation, Maximum Inflation, and Heaping," Journal of Educational and Behavioral Statistics, , vol. 42(5), pages 531-558, October.
    10. Yang Liu & Jan Hannig, 2017. "Generalized Fiducial Inference for Logistic Graded Response Models," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1097-1125, December.
    11. repec:jss:jstsof:35:i12 is not listed on IDEAS
    12. Maydeu Olivares, Alberto & D'Zurilla, Thomas J. & Morera, Osvaldo, 1996. "Assessing measurement invariance in questionnaires within latent trait models using item response theory," DES - Working Papers. Statistics and Econometrics. WS 10456, Universidad Carlos III de Madrid. Departamento de Estadística.
    13. Daniel Bolt & James Wollack & Youngsuk Suh, 2012. "Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 339-357, April.
    14. Dylan Molenaar, 2015. "Heteroscedastic Latent Trait Models for Dichotomous Data," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 625-644, September.
    15. Yang Liu & Weimeng Wang, 2022. "Semiparametric Factor Analysis for Item-Level Response Time Data," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 666-692, June.
    16. Carmen Köhler & Alexander Robitzsch & Johannes Hartig, 2020. "A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 251-273, June.
    17. Gerhard Tutz, 2022. "Item Response Thresholds Models: A General Class of Models for Varying Types of Items," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1238-1269, December.
    18. Daphna Harel & Russell J. Steele, 2018. "An Information Matrix Test for the Collapsing of Categories Under the Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 721-750, December.
    19. Singh, Jagdip, 2004. "Tackling measurement problems with Item Response Theory: Principles, characteristics, and assessment, with an illustrative example," Journal of Business Research, Elsevier, vol. 57(2), pages 184-208, February.
    20. David Magis, 2015. "A Note on the Equivalence Between Observed and Expected Information Functions With Polytomous IRT Models," Journal of Educational and Behavioral Statistics, , vol. 40(1), pages 96-105, February.
    21. Pengsheng Ni & Molly Marino & Emily Dore & Lily Sonis & Colleen M Ryan & Jeffrey C Schneider & Alan M Jette & Lewis E Kazis, 2019. "Extreme response style bias in burn survivors," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-13, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:12:y:2022:i:2:p:21582440221108168. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.