IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v98y2023ics0149718923000514.html

Consequences of survey modification in a program evaluation: An exploratory research on evaluation study

Author

Listed:
  • Braman, Colin
  • Azzam, Tarek

Abstract

The purpose of this exploratory research on evaluation study was to examine how modified and unmodified scales of critical thinking and interest in science careers would affect the evaluation conclusions. Surveys measuring various outcomes that are used in program evaluation are frequently modified from their original versions in response to the unique context of programs. Modifying existing published surveys by removing or adding items can affect the psychometric properties of the original scale and may produce differing results. The results of the comparisons found that unmodified and modified surveys had similar reliabilities; however, one of the scales produced contradictory evaluation findings. Lessons learned from this study suggest that scales can be modified in evaluation, but great care is needed to address the potential strengths and limitations of the modified scale and balance the technical needs with responsiveness to program context.

Suggested Citation

  • Braman, Colin & Azzam, Tarek, 2023. "Consequences of survey modification in a program evaluation: An exploratory research on evaluation study," Evaluation and Program Planning, Elsevier, vol. 98(C).
    • Handle: RePEc:eee:epplan:v:98:y:2023:i:c:s0149718923000514
    DOI: 10.1016/j.evalprogplan.2023.102274
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718923000514
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2023.102274?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Flynn, Leisa Reinecke & Goldsmith, Ronald E., 1999. "A Short, Reliable Measure of Subjective Knowledge," Journal of Business Research, Elsevier, vol. 46(1), pages 57-66, September.
    2. Greene, Jennifer C., 1990. "Technical quality versus user responsiveness in evaluation practice," Evaluation and Program Planning, Elsevier, vol. 13(3), pages 267-274, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Groza, Mark D. & Groza, Mya Pronschinske, 2018. "Salesperson regulatory knowledge and sales performance," Journal of Business Research, Elsevier, vol. 89(C), pages 37-46.
    2. Bernadette M. Clarke & Emma McKinley & Rhoda C. Ballinger, 2025. "UK Public Attitudes and Perceptions of Seafood Sustainability: A Case Study of the Marine Conservation Society’s Good Fish Guide," Sustainability, MDPI, vol. 17(2), pages 1-32, January.
    3. Benjamin Polak & Andreas Herrmann & Mark Heitmann & Marc Cäsar & Jan Landwehr, 2009. "„Radio oder Schiebedach?“–Wirkung von mentalen Budgets auf das individuelle Produktwahlverhalten," Schmalenbach Journal of Business Research, Springer, vol. 61(7), pages 783-803, November.
    4. Javornik, Ana & Marder, Ben & Pizzetti, Marta & Warlop, Luk, 2021. "Augmented self - The effects of virtual face augmentation on consumers' self-concept," Journal of Business Research, Elsevier, vol. 130(C), pages 170-187.
    5. Lim, Thien Sang & Mail, Rasid & Abd Karim, Mohd Rahimie & Ahmad Baharul Ulum, Zatul Karamah & Jaidi, Junainah & Noordin, Raman, 2018. "A serial mediation model of financial knowledge on the intention to invest: The central role of risk perception and attitude," Journal of Behavioral and Experimental Finance, Elsevier, vol. 20(C), pages 74-79.
    6. Alain d’Astous & Amélie Legendre, 2009. "Understanding Consumers’ Ethical Justifications: A Scale for Appraising Consumers’ Reasons for Not Behaving Ethically," Journal of Business Ethics, Springer, vol. 87(2), pages 255-268, June.
    7. Owusu-Sekyere, Enoch & Jordaan, Henry, 2019. "The Development and Promotion of Environmentally Sustainable Food Products in South Africa," International Journal of Agricultural Management, Institute of Agricultural Management, vol. 8(3), August.
    8. Schweisfurth, Tim G., 2017. "Comparing internal and external lead users as sources of innovation," Research Policy, Elsevier, vol. 46(1), pages 238-248.
    9. de Lauwere, Carolien & Slegers, Monique & Meeusen, Marieke, 2022. "The influence of behavioural factors and external conditions on Dutch farmers’ decision making in the transition towards circular agriculture," Land Use Policy, Elsevier, vol. 120(C).
    10. Catherine Viot & Juliette Passebois-Ducros, 2010. "Wine brands or branded wines? The specificity of the French market in terms of the brand," Post-Print hal-01803728, HAL.
    11. Carla S. Marques & Carlos P. Marques & João J. M. Ferreira & Fernando A. F. Ferreira, 2019. "Effects of traits, self-motivation and managerial skills on nursing intrapreneurship," International Entrepreneurship and Management Journal, Springer, vol. 15(3), pages 733-748, September.
    12. House, Lisa & Lusk, Jayson L. & Jaeger, Sara & Traill, W. Bruce & Moore, Melissa & Valli, Carlotta & Morrow, Bert & Yee, Wallace M.S., 2004. "Objective And Subjective Knowledge: Impacts On Consumer Demand For Genetically Modified Foods In The United States And The European Union," 2004 Annual meeting, August 1-4, Denver, CO 20125, American Agricultural Economics Association (New Name 2008: Agricultural and Applied Economics Association).
    13. Tan, Ser Zian & Yang, Lin & Loo, Wee Hong, 2026. "Green Specificity: Igniting curiosity and arousing emotional ambivalence," Journal of Retailing and Consumer Services, Elsevier, vol. 88(C).
    14. Eva M. Murgado-Armenteros & María Gutierrez-Salcedo & Francisco José Torres-Ruiz, 2020. "The Concern about Biodiversity as a Criterion for the Classification of the Sustainable Consumer: A Cross-Cultural Approach," Sustainability, MDPI, vol. 12(8), pages 1-14, April.
    15. Benedikt Berger & Martin Adam & Alexander Rühr & Alexander Benlian, 2021. "Watch Me Improve—Algorithm Aversion and Demonstrating the Ability to Learn," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(1), pages 55-68, February.
    16. Amira Osman & Sarah D. Kowitt & Paschal Sheeran & Kristen L. Jarman & Leah M. Ranney & Adam O. Goldstein, 2018. "Information to Improve Public Perceptions of the Food and Drug Administration (FDA’s) Tobacco Regulatory Role," IJERPH, MDPI, vol. 15(4), pages 1-13, April.
    17. Feucht, Yvonne & Zander, Katrin, 2017. "Consumers’ attitudes on carbon footprint labelling. Results of the SUSDIET project," Thünen Working Paper 266396, Johann Heinrich von Thünen-Institut (vTI), Federal Research Institute for Rural Areas, Forestry and Fisheries.
    18. Arun Kumar Kaushik & Zillur Rahman, 2016. "Self-service innovativeness scale: introduction, development, and validation of scale," Service Business, Springer;Pan-Pacific Business Association, vol. 10(4), pages 799-822, December.
    19. Kwon, Kyoung-Nan & Lee, Jinkook, 2009. "The effects of reference point, knowledge, and risk propensity on the evaluation of financial products," Journal of Business Research, Elsevier, vol. 62(7), pages 719-725, July.
    20. Hatice Aydýn & Cemal Zehir, 2017. "What Type Relationship Do We Have with Our Brands? Is the Name of this Relationship Brand Romance?," International Review of Management and Marketing, Econjournals, vol. 7(2), pages 272-283.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:98:y:2023:i:c:s0149718923000514. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.