IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v38y2014i5p420-448.html
   My bibliography  Save this article

Evaluating Response Shift in Training Evaluation

Author

Listed:
  • Valentina Piwowar
  • Felicitas Thiel

Abstract

Background: Response shift (RS) can threaten the internal validity of pre–post designs. As RS may indicate a redefinition of the target construct, its occurrence in training evaluation is rather likely. The most common approach to deal with RS is to implement a retrospective pretest (then-test) instead of the traditional pre-test. In health psychology, an adapted measurement invariance approach (MI ad ) was developed as an alternative technique to study RS. Results produced by identifying RS with the two approaches were rarely studied simultaneously or within an experimental framework. Objectives: To study RS in two different treatment conditions and compare results produced by both techniques in identifying various types of RS. We further studied validity aspects of the then-test. Research Design: We evaluated RS by applying the then-test procedure (TP) and the measurement invariance apporach MI ad within an experimental design: Participants either attended a short-term or a long-term classroom management training program. Subjects: Participants were 146 student teachers in their first year of master’s study. Measures: Pre (before training), post, and then self-ratings (after training) on classroom management knowledge were administered. Results: Results indicated that the two approaches do not yield the same results. The MI ad identified more and also group-specific RS as opposed to the findings of the TP, which found less and only little evidence for group-specific RS. Conclusions: Further research is needed to study the usability and validity of the respective approaches. In particular, the usability of the then-test seems to be challenged.

Suggested Citation

  • Valentina Piwowar & Felicitas Thiel, 2014. "Evaluating Response Shift in Training Evaluation," Evaluation Review, , vol. 38(5), pages 420-448, October.
    • Handle: RePEc:sae:evarev:v:38:y:2014:i:5:p:420-448
    DOI: 10.1177/0193841X14546932
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X14546932
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X14546932?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gibbons, F. X., 1999. "Social comparison as a mediator of response shift," Social Science & Medicine, Elsevier, vol. 48(11), pages 1517-1530, June.
    2. Schwartz, Carolyn E. & Sprangers, Mirjam A. G., 1999. "Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research," Social Science & Medicine, Elsevier, vol. 48(11), pages 1531-1548, June.
    3. William Meredith, 1993. "Measurement invariance, factor analysis and factorial invariance," Psychometrika, Springer;The Psychometric Society, vol. 58(4), pages 525-543, December.
    4. Sprangers, Mirjam A. G. & Schwartz, Carolyn E., 1999. "Integrating response shift into health-related quality of life research: a theoretical model," Social Science & Medicine, Elsevier, vol. 48(11), pages 1507-1515, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Damschroder, Laura J. & Zikmund-Fisher, Brian J. & Ubel, Peter A., 2005. "The impact of considering adaptation in health state valuation," Social Science & Medicine, Elsevier, vol. 61(2), pages 267-277, July.
    2. Wim Groot, 2003. "Scale of reference bias and the evolution of health," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 4(3), pages 176-183, September.
    3. Mohamed Boucekine & Laurent Boyer & Karine Baumstarck & Aurelie Millier & Badih Ghattas & Pascal Auquier & Mondher Toumi, 2015. "Exploring the Response Shift Effect on the Quality of Life of Patients with Schizophrenia," Medical Decision Making, , vol. 35(3), pages 388-397, April.
    4. Astrid Wahl & Tone Rustøen & Berit Hanestad & Eva Gjengedal & Torbjørn Moum, 2005. "Self-Efficacy, Pulmonary Function, Perceived Health and Global Quality of Life of Cystic Fibrosis Patients," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 72(2), pages 239-261, June.
    5. Carolyn Schwartz & Penelope Keyl & John Marcum & Rita Bode, 2009. "Helping Others Shows Differential Benefits on Health and Well-being for Male and Female Teens," Journal of Happiness Studies, Springer, vol. 10(4), pages 431-448, August.
    6. Kaiser, Caspar, 2022. "Using memories to assess the intrapersonal comparability of wellbeing reports," Journal of Economic Behavior & Organization, Elsevier, vol. 193(C), pages 410-442.
    7. Peter A. Ubel & George Loewenstein & John Hershey & Jonathan Baron & Tara Mohr & David A. Asch & Christopher Jepson, 2001. "Do Nonpatients Underestimate the Quality of Life Associated with Chronic Health Conditions because of a Focusing Illusion?," Medical Decision Making, , vol. 21(3), pages 190-199, May.
    8. Katrin Boström & Gerd Ahlström & Helena Sunvisson, 2006. "Being the Next of Kin of an Adult Person With Muscular Dystrophy," Clinical Nursing Research, , vol. 15(2), pages 86-104, May.
    9. Thérèse Shaw & Donna Cross & Stephen R. Zubrick, 2015. "Testing for Response Shift Bias in Evaluations of School Antibullying Programs," Evaluation Review, , vol. 39(6), pages 527-554, December.
    10. Dibb, Bridget & Yardley, Lucy, 2006. "How does social comparison within a self-help group influence adjustment to chronic illness? A longitudinal study," Social Science & Medicine, Elsevier, vol. 63(6), pages 1602-1613, September.
    11. Maes, Kenneth C. & Hadley, Craig & Tesfaye, Fikru & Shifferaw, Selamawit, 2010. "Food insecurity and mental health: Surprising trends among community health volunteers in Addis Ababa, Ethiopia during the 2008 food crisis," Social Science & Medicine, Elsevier, vol. 70(9), pages 1450-1457, May.
    12. Rissanen, Ritva & Warnqvist, Anna & Hasselberg, Marie, 2023. "“I knew who I was this morning, but I've changed a few times since then”: A study combining register and self-reported QoL data in assessing how response shift may occur in an injury population," Social Science & Medicine, Elsevier, vol. 326(C).
    13. B. King-Kallimanis & F. Oort & G. Garst, 2010. "Using structural equation modelling to detect measurement bias and response shift in longitudinal data," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 94(2), pages 139-156, June.
    14. Richard Milne & Kathy Heaton-Brown & Paul Hansen & David Thomas & Vernon Harvey & Alison Cubitt, 2006. "Quality-of-Life Valuations of Advanced Breast Cancer by New Zealand Women," PharmacoEconomics, Springer, vol. 24(3), pages 281-292, March.
    15. Winter, Laraine & Parker, Barbara, 2007. "Current health and preferences for life-prolonging treatments: An application of prospect theory to end-of-life decision making," Social Science & Medicine, Elsevier, vol. 65(8), pages 1695-1707, October.
    16. Johan Oud & Manuel Voelkle, 2014. "Do missing values exist? Incomplete data handling in cross-national longitudinal studies by means of continuous time modeling," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(6), pages 3271-3288, November.
    17. Nick Bansback & Huiying Sun & Daphne P. Guh & Xin Li & Bohdan Nosyk & Susan Griffin & Paul G. Barnett & Aslam H. Anis, 2008. "Impact of the recall period on measuring health utilities for acute events," Health Economics, John Wiley & Sons, Ltd., vol. 17(12), pages 1413-1419.
    18. Liat Ayalon, 2018. "Perceived Age Discrimination: A Precipitator or a Consequence of Depressive Symptoms?," The Journals of Gerontology: Series B, The Gerontological Society of America, vol. 73(5), pages 860-869.
    19. Steven Andrew Culpepper & Herman Aguinis & Justin L. Kern & Roger Millsap, 2019. "High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 285-309, March.
    20. Ihsana Sabriani Borualogo & Ferran Casas, 2023. "Bullying Victimisation and Children’s Subjective Well-being: A Comparative Study in Seven Asian Countries," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 16(1), pages 1-27, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:38:y:2014:i:5:p:420-448. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.