IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i2p237-269.html

Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions

Author

Listed:
  • Pamela R. Buckley
  • Katie Massey Combs
  • Karen M. Drewelow
  • Brittany L. Hubler
  • Marion Amanda Lain

Abstract

As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers’ ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane’s (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.

Suggested Citation

  • Pamela R. Buckley & Katie Massey Combs & Karen M. Drewelow & Brittany L. Hubler & Marion Amanda Lain, 2025. "Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions," Evaluation Review, , vol. 49(2), pages 237-269, April.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:237-269
    DOI: 10.1177/0193841X241248864
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241248864
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241248864?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lee, Meng-Jung & Wu, Wen-Chi & Chang, Hung-Chieh & Chen, Hsing-Jung & Lin, Wen-Shan & Feng, Joyce Yen & Lee, Tony Szu-Hsien, 2020. "Effectiveness of a school-based life skills program on emotional regulation and depression among elementary school students: A randomized study," Children and Youth Services Review, Elsevier, vol. 118(C).
    2. Huang, Chien-Chung & Chen, Yafan & Greene, Loren & Cheung, Shannon & Wei, Yuhan, 2019. "Resilience and emotional and behavioral problems of adolescents in China: Effects of a short-term and intensive mindfulness and life skills training," Children and Youth Services Review, Elsevier, vol. 100(C), pages 291-297.
    3. Burkhardt, Jason T. & Schröter, Daniela C. & Magura, Stephen & Means, Stephanie N. & Coryn, Chris L.S., 2015. "An overview of evidence-based program registers (EBPRs) for behavioral health," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 92-99.
    4. Jiajuan Liang & Peter Bentler, 2004. "An EM algorithm for fitting two-level structural equation models," Psychometrika, Springer;The Psychometric Society, vol. 69(1), pages 101-122, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Majid Ghasemy & Isabel Maria Rosa-Díaz & James Eric Gaskin, 2021. "The Roles of Supervisory Support and Involvement in Influencing Scientists’ Job Satisfaction to Ensure the Achievement of SDGs in Academic Organizations," SAGE Open, , vol. 11(3), pages 21582440211, July.
    2. Benjamin Agbo & Hussain Al-Aqrabi & Richard Hill & Tariq Alsboui, 2022. "Missing Data Imputation in the Internet of Things Sensor Networks," Future Internet, MDPI, vol. 14(5), pages 1-16, May.
    3. Chien-Chung Huang & Shuang Lu & Juan Rios & Yafan Chen & Marci Stringham & Shannon Cheung, 2020. "Associations between Mindfulness, Executive Function, Social-Emotional Skills, and Quality of Life among Hispanic Children," IJERPH, MDPI, vol. 17(21), pages 1-12, October.
    4. Zack, Melissa K. & Karre, Jennifer K. & Olson, Jonathan & Perkins, Daniel F., 2019. "Similarities and differences in program registers: A case study," Evaluation and Program Planning, Elsevier, vol. 76(C), pages 1-1.
    5. Thomas E. Strayer & Laura E. Balis & NithyaPriya S. Ramalingam & Samantha M. Harden, 2022. "Dissemination in Extension: Health Specialists’ Information Sources and Channels for Health Promotion Programming," IJERPH, MDPI, vol. 19(24), pages 1-13, December.
    6. Jonathan Schweig, 2014. "Multilevel Factor Analysis by Model Segregation," Journal of Educational and Behavioral Statistics, , vol. 39(5), pages 394-422, October.
    7. Chen, Yafan & Xie, Xiaoxia & Huang, Chien-Chung, 2021. "Resilience of vocational students with disadvantaged characteristics in China: The role of mindfulness," Children and Youth Services Review, Elsevier, vol. 122(C).
    8. Wai-Yin Poon & Hai-Bin Wang, 2010. "Analysis of a Two-Level Structural Equation Model With Missing Data," Sociological Methods & Research, , vol. 39(1), pages 25-55, August.
    9. Bujold, M. & El Sherif, R. & Bush, P.L. & Johnson-Lafleur, J. & Doray, G. & Pluye, P., 2018. "Ecological content validation of the Information Assessment Method for parents (IAM-parent): A mixed methods study," Evaluation and Program Planning, Elsevier, vol. 66(C), pages 79-88.
    10. Oliver, Jason A. & Lang, Jason M., 2018. "Barriers and consultation needs regarding implementation of evidence-based treatment in community agencies," Children and Youth Services Review, Elsevier, vol. 94(C), pages 368-377.
    11. Magura, Stephen & Lee-Easton, Miranda J. & Abu-Obaid, Ruqayyah N. & Landsverk, John & DeCamp, Whitney & Rolls-Reutz, Jennifer & Moore, Kristin & Firpo-Triplett, Regina & Buckley, Pamela R. & Stout, El, 2023. "The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare," Evaluation and Program Planning, Elsevier, vol. 97(C).
    12. Nicholas J. Rockwood, 2020. "Maximum Likelihood Estimation of Multilevel Structural Equation Models with Random Slopes for Latent Covariates," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 275-300, June.
    13. Ke-Hai Yuan & Kentaro Hayashi, 2005. "On muthén’s maximum likelihood for two-level covariance structure models," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 147-167, March.
    14. Eryılmaz, Ali & Yıldırım-Kurtuluş, Hacer & Yıldırım, Murat, 2024. "A cross-sectional and longitudinal study on the mediation effect of positive and negative affects in the relationship between self-compassion and resilience in early adolescents," Children and Youth Services Review, Elsevier, vol. 161(C).
    15. Wu, Wen-Chi & Lee, Meng-Jung & Chang, Yuwen, 2023. "Effects of the positive Interpersonal and life orientation training (PILOT) program among elementary school students in Taiwan," Children and Youth Services Review, Elsevier, vol. 155(C).
    16. Piotr Tarka, 2018. "An overview of structural equation modeling: its beginnings, historical development, usefulness and controversies in the social sciences," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(1), pages 313-354, January.
    17. Iudici, Antonio & Gagliardo Corsi, Agnese, 2017. "Evaluation in the field of social services for minors: Measuring the efficacy of interventions in the Italian service for health protection and promotion," Evaluation and Program Planning, Elsevier, vol. 61(C), pages 160-168.
    18. He, Dan & Shen, Xi & Liu, Qing-Qi, 2020. "The relationship between upward social comparison on SNSs and excessive smartphone use: A moderated mediation analysis," Children and Youth Services Review, Elsevier, vol. 116(C).
    19. Axford, Nick & Morpeth, Louise & Bjornstad, Gretchen & Hobbs, Tim & Berry, Vashti, 2022. "“What works” registries of interventions to improve child and youth psychosocial outcomes: A critical appraisal," Children and Youth Services Review, Elsevier, vol. 137(C).
    20. Joop J. Hox & Cora J. M. Maas & Matthieu J. S. Brinkhuis, 2010. "The effect of estimation method and sample size in multilevel structural equation modeling," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 64(2), pages 157-170, May.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:237-269. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.