IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v103y2024ics0149718924000120.html
   My bibliography  Save this article

Designing a questionnaire with retrospective pre-post items: Format matters

Author

Listed:
  • Hwalek, Melanie
  • Pierce, Kate
  • Straub, Victoria

Abstract

Evaluators are frequently asked to evaluate educational interventions that are one-time events. The Retrospective Pretest (RPT) methodology is well suited for these circumstances and the evaluation literature is replete with discussions about the pros and cons of this approach. In RPT, program participants rate their attitudes or knowledge now, and also rate how they were before participation in the intervention. The difference between now and before ratings constitutes the measure of change. Little published literature exists about whether the layout of RPT items within the evaluation questionnaire yields different results. This study compared six different layouts using a sample of 1941 caregivers who participated in one of 96 training workshops. The layouts were compared on inattentiveness, unexpected decline in perceived knowledge, and the degree of before-now change. Findings show that design matters. The best performing layout is where items are placed in the center with before response options on the left of the page and the now response options on the right. Results point to the need for evaluators to pay attention to RPT layout, and for the field to establish criteria for assessing survey layout quality. The unexpectedly high rate of inattentiveness calls for evaluators to pay more attention identifying and addressing this in their survey data.

Suggested Citation

  • Hwalek, Melanie & Pierce, Kate & Straub, Victoria, 2024. "Designing a questionnaire with retrospective pre-post items: Format matters," Evaluation and Program Planning, Elsevier, vol. 103(C).
  • Handle: RePEc:eee:epplan:v:103:y:2024:i:c:s0149718924000120
    DOI: 10.1016/j.evalprogplan.2024.102411
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718924000120
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.evalprogplan.2024.102411?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Alvarez, R. Michael & Atkeson, Lonna Rae & Levin, Ines & Li, Yimeng, 2019. "Paying Attention to Inattentive Survey Respondents," Political Analysis, Cambridge University Press, vol. 27(2), pages 145-162, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vecchio, Riccardo & Caso, Gerarda & Cembalo, Luigi & Borrello, Massimiliano, 2020. "Is respondents’ inattention in online surveys a major issue for research?," Economia agro-alimentare / Food Economy, Italian Society of Agri-food Economics/Società Italiana di Economia Agro-Alimentare (SIEA), vol. 22(01), March.
    2. Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    3. Elgaaied-Gambier, Leila & Bertrandias, Laurent & Bernard, Yohan, 2020. "Cutting the Internet's Environmental Footprint: An Analysis of Consumers' Self-Attribution of Responsibility," Journal of Interactive Marketing, Elsevier, vol. 50(C), pages 120-135.
    4. Jianhua Xu & Shiwei Fan & Jiakun Zheng, 2025. "Valuing mortality risk reductions in the time of COVID-19: A stated-preference analysis," Post-Print hal-04909840, HAL.
    5. Eiichiro Watamura & Tomohiro Ioku, 2023. "Wrongful Conviction Rates and Death Penalty Support: Acceptability Hypothesis in Japan," SAGE Open, , vol. 13(3), pages 21582440231, August.
    6. repec:osf:socarx:b4fc7_v1 is not listed on IDEAS
    7. Alfredo Manuli & Maria Grazia Maggio & Gianluca La Rosa & Vera Gregoli & Daniele Tripoli & Fausto Famà & Valentina Oddo & Giovanni Pioggia & Rocco Salvatore Calabrò, 2022. "Emotional Impact of COVID-19 Pandemic on Nursing Students Receiving Distance Learning: An Explorative Study," IJERPH, MDPI, vol. 19(17), pages 1-10, August.
    8. Riccardo Vecchio & Gerarda Caso & Luigi Cembalo & Massimiliano Borrello, 2020. "Is respondents? inattention in online surveys a major issue for research?," Economia agro-alimentare, FrancoAngeli Editore, vol. 22(1), pages 1-18.
    9. Zhiyi Chen & Yancheng Tang & Xuerong Liu & Wei Li & Yuanyuan Hu & Bowen Hu & Ting Xu & Rong Zhang & Lei Xia & Jing-Xuan Zhang & Zhibing Xiao & Ji Chen & Zhengzhi Feng & Yuan Zhou & Qinghua He & Jiang , 2024. "Edge-centric connectome-genetic markers of bridging factor to comorbidity between depression and anxiety," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    10. Siddique, Abu & Islam, Asad & Mozumder, Tanvir Ahmed & Rahman, Tabassum & Shatil, Tanvir, 2022. "Forced Displacement, Mental Health, and Child Development: Evidence from the Rohingya Refugees," SocArXiv b4fc7, Center for Open Science.
    11. Del Ponte, Alessandro & Li, Lianjun & Ang, Lina & Lim, Noah & Seow, Wei Jie, 2024. "Evaluating SoJump.com as a tool for online behavioral research in China," Journal of Behavioral and Experimental Finance, Elsevier, vol. 41(C).
    12. Victoria Gevorkova & Ivan Sangiorgi & Julia Vogt, 2024. "Cleansing Investor’s Conscience: The Effects of Incidental Guilt on Socially Responsible Investment Decisions," Journal of Business Ethics, Springer, vol. 193(1), pages 89-114, August.
    13. Bansak, Kirk & Paulson, Elisabeth, 2023. "Public Opinion on Fairness and Efficiency for Algorithmic and Human Decision-Makers," OSF Preprints pghmx, Center for Open Science.
    14. Daniel Y. Park & Hyungsook Kim, 2023. "Determinants of Intentions to Use Digital Mental Healthcare Content among University Students, Faculty, and Staff: Motivation, Perceived Usefulness, Perceived Ease of Use, and Parasocial Interaction w," Sustainability, MDPI, vol. 15(1), pages 1-17, January.
    15. Arielle S. Keller & Adam R. Pines & Sheila Shanmugan & Valerie J. Sydnor & Zaixu Cui & Maxwell A. Bertolero & Ran Barzilay & Aaron F. Alexander-Bloch & Nora Byington & Andrew Chen & Gregory M. Conan &, 2023. "Personalized functional brain network topography is associated with individual differences in youth cognition," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    16. Sarah Lord Ferguson & Pierre Berthon, 2022. "A renewable resource model of health decision-making: insights to improve health marketing," AMS Review, Springer;Academy of Marketing Science, vol. 12(1), pages 71-84, June.
    17. repec:osf:osfxxx:pghmx_v1 is not listed on IDEAS

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:103:y:2024:i:c:s0149718924000120. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.