IDEAS home Printed from https://ideas.repec.org/a/eee/cysrev/v33y2011i11p2146-2151.html
   My bibliography  Save this article

Developing and validating measures for child welfare agencies to self-monitor fidelity to a child safety intervention

Author

Listed:
  • Kaye, Sarah
  • Osteen, Philip J.

Abstract

Building evidence of effective practice in child welfare requires practitioners and researchers to know the extent to which programs are implemented in order to understand evaluation results. Fidelity monitoring is a critical strategy for ensuring that evidence-based and promising practices are implemented as intended and can be studied in real-world contexts. This paper addresses challenges to measuring fidelity in child welfare systems and presents an approach taken with one state to define fidelity criteria and measure fidelity to a child safety intervention. Measurement challenges were addressed by using existing documents and case review mechanisms to assess fidelity, and measuring the quality of practitioner judgment using expert reviewers. Validity of fidelity criteria and fidelity review instruments was established through consensus with model developers and local purveyors. Twelve cases were reviewed by a panel of raters to assess inter-rater reliability and discrepancy between local purveyors and model developers. This participatory and capacity-building method can be replicated and used to develop and embed valid and reliable fidelity monitoring systems in public child welfare to continue to build evidence about what works in child welfare services.

Suggested Citation

  • Kaye, Sarah & Osteen, Philip J., 2011. "Developing and validating measures for child welfare agencies to self-monitor fidelity to a child safety intervention," Children and Youth Services Review, Elsevier, vol. 33(11), pages 2146-2151.
    • Handle: RePEc:eee:cysrev:v:33:y:2011:i:11:p:2146-2151
    DOI: 10.1016/j.childyouth.2011.06.020
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0190740911002349
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.childyouth.2011.06.020?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Klaus Krippendorff, 1987. "Association, agreement, and equity," Quality & Quantity: International Journal of Methodology, Springer, vol. 21(2), pages 109-123, June.
    2. Holden, E. Wayne & Rousseau O'Connell, Susan & Connor, Tim & Brannan, Ana Maria & Foster, E. Michael & Blau, Gary & Panciera, Heather, 2002. "Evaluation of the Connecticut Title IV-E Waiver Program: Assessing the Effectiveness, Implementation Fidelity, and Cost/Benefits of a Continuum of Care," Children and Youth Services Review, Elsevier, vol. 24(6-7), pages 409-430.
    3. Crea, Thomas M. & Usher, Charles L. & Wildfire, Judith B., 2009. "Implementation fidelity of Team Decisionmaking," Children and Youth Services Review, Elsevier, vol. 31(1), pages 119-124, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bearman, Sarah Kate & Garland, Ann F. & Schoenwald, Sonja K., 2014. "From practice to evidence in child welfare: Model specification and fidelity measurement of Team Decisionmaking," Children and Youth Services Review, Elsevier, vol. 39(C), pages 153-159.
    2. Moon, Deborah J. & Johnson-Motoyama, Michelle & Bailey Nichols, C. & Rolock, Nancy & Haran, Hana & Bai, Rong, 2022. "Collaborative fidelity assessment planning for the evaluation of a community-based maltreatment prevention model: The family success network (FSN)," Children and Youth Services Review, Elsevier, vol. 137(C).
    3. Barak, Adi & Spielberger, Julie & Gitlow, Elissa, 2014. "The challenge of relationships and fidelity: Home visitors' perspectives," Children and Youth Services Review, Elsevier, vol. 42(C), pages 50-58.
    4. Rauktis, Mary Elizabeth & Bishop-Fitzpatrick, Lauren & Jung, Nahri & Pennell, Joan, 2013. "Family group decision making: Measuring fidelity to practice principles in public child welfare," Children and Youth Services Review, Elsevier, vol. 35(2), pages 287-295.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matthijs Warrens, 2010. "Inequalities Between Kappa and Kappa-Like Statistics for k×k Tables," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 176-185, March.
    2. Matthijs J. Warrens, 2014. "Power Weighted Versions of Bennett, Alpert, and Goldstein’s," Journal of Mathematics, Hindawi, vol. 2014, pages 1-9, December.
    3. Bearman, Sarah Kate & Garland, Ann F. & Schoenwald, Sonja K., 2014. "From practice to evidence in child welfare: Model specification and fidelity measurement of Team Decisionmaking," Children and Youth Services Review, Elsevier, vol. 39(C), pages 153-159.
    4. Matthijs Warrens, 2010. "Inequalities between multi-rater kappas," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 4(4), pages 271-286, December.
    5. Crea, Thomas M. & Barth, Richard P. & Chintapalli, Laura K. & Buchanan, Rachel L., 2009. "The implementation and expansion of SAFE: Frontline responses and the transfer of technology to practice," Children and Youth Services Review, Elsevier, vol. 31(8), pages 903-910, August.
    6. Wang, Dan & Xiang, Zheng & Fesenmaier, Daniel R., 2014. "Adapting to the mobile world: A model of smartphone use," Annals of Tourism Research, Elsevier, vol. 48(C), pages 11-26.
    7. Chambers, Ruth M. & Crutchfield, Rashida M. & Goddu Harper, Stephanie G. & Fatemi, Maryam & Rodriguez, Angel Y., 2018. "Family reunification in child welfare practice: A pilot study of parent and staff experiences," Children and Youth Services Review, Elsevier, vol. 91(C), pages 221-231.
    8. Roel Popping, 2010. "Some views on agreement to be used in content analysis studies," Quality & Quantity: International Journal of Methodology, Springer, vol. 44(6), pages 1067-1078, October.
    9. Matthijs J. Warrens, 2014. "New Interpretations of Cohen’s Kappa," Journal of Mathematics, Hindawi, vol. 2014, pages 1-9, September.
    10. Isokuortti, Nanne & Aaltio, Elina, 2020. "Fidelity and influencing factors in the Systemic Practice Model of children's social care in Finland," Children and Youth Services Review, Elsevier, vol. 119(C).
    11. Matthijs Warrens, 2008. "On Association Coefficients for 2×2 Tables and Properties That Do Not Depend on the Marginal Distributions," Psychometrika, Springer;The Psychometric Society, vol. 73(4), pages 777-789, December.
    12. Stephanie Stray, 2009. "Lies, damned lies and statistics: the accuracy of survey responses," Quality & Quantity: International Journal of Methodology, Springer, vol. 43(1), pages 161-171, January.
    13. Klaus krippendorff, 2004. "Measuring the Reliability of Qualitative Text Analysis Data," Quality & Quantity: International Journal of Methodology, Springer, vol. 38(6), pages 787-800, December.
    14. Matthijs Warrens, 2008. "On Similarity Coefficients for 2×2 Tables and Correction for Chance," Psychometrika, Springer;The Psychometric Society, vol. 73(3), pages 487-502, September.
    15. Lietz, Cynthia A., 2011. "Theoretical adherence to family centered practice: Are strengths-based principles illustrated in families' descriptions of child welfare services?," Children and Youth Services Review, Elsevier, vol. 33(6), pages 888-893, June.
    16. Snyder, Elizabeth H. & Lawrence, C. Nicole & Dodge, Kenneth A., 2012. "The impact of system of care support in adherence to wraparound principles in Child and Family Teams in child welfare in North Carolina," Children and Youth Services Review, Elsevier, vol. 34(4), pages 639-647.
    17. Stephanie Stray, 2008. "Environmental Reporting: The U.K. Water and Energy Industries: A Research Note," Journal of Business Ethics, Springer, vol. 80(4), pages 697-710, July.
    18. Crea, Thomas M. & Crampton, David S., 2011. "The context of program implementation and evaluation: A pilot study of interorganizational differences to improve child welfare reform efforts," Children and Youth Services Review, Elsevier, vol. 33(11), pages 2273-2281.
    19. Crea, Thomas M. & Crampton, David S. & Abramson-Madden, Anne & Usher, Charles L., 2008. "Variability in the implementation of Team Decisionmaking (TDM): Scope and compliance with the Family to Family practice model," Children and Youth Services Review, Elsevier, vol. 30(11), pages 1221-1232, November.
    20. Lieuwe Dijkstra & Frans Eijnatten, 2009. "Agreement and consensus in a Q-mode research design: an empirical comparison of measures, and an application," Quality & Quantity: International Journal of Methodology, Springer, vol. 43(5), pages 757-771, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:cysrev:v:33:y:2011:i:11:p:2146-2151. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/childyouth .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.