IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v48y2015icp100-116.html
   My bibliography  Save this article

Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs

Author

Listed:
  • Means, Stephanie N.
  • Magura, Stephen
  • Burkhardt, Jason T.
  • Schröter, Daniela C.
  • Coryn, Chris L.S.

Abstract

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being “evidence-based” has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of “no effect,” which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.

Suggested Citation

  • Means, Stephanie N. & Magura, Stephen & Burkhardt, Jason T. & Schröter, Daniela C. & Coryn, Chris L.S., 2015. "Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 100-116.
    • Handle: RePEc:eee:epplan:v:48:y:2015:i:c:p:100-116
    DOI: 10.1016/j.evalprogplan.2014.09.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718914001116
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2014.09.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rieckmann, Traci R. & Kovas, Anne E. & Cassidy, Elaine F. & McCarty, Dennis, 2011. "Employing policy and purchasing levers to increase the use of evidence-based practices in community-based substance abuse treatment settings: Reports from single state authorities," Evaluation and Program Planning, Elsevier, vol. 34(4), pages 366-374, November.
    2. Hilda Bastian & Paul Glasziou & Iain Chalmers, 2010. "Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?," PLOS Medicine, Public Library of Science, vol. 7(9), pages 1-6, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Iniesta, Julián & Verdugo, Miguel A. & Schalock, Robert L., 2023. "Organizational change and evidence-based practices in support services for people with intellectual and developmental disabilities," Evaluation and Program Planning, Elsevier, vol. 100(C).
    2. T’Pring R. Westbrook & Sarah A. Avellar & Neil Seftor, 2017. "Reviewing the Reviews: Examining Similarities and Differences Between Federally Funded Evidence Reviews," Evaluation Review, , vol. 41(3), pages 183-211, June.
    3. Maranda, Michael J. & Magura, Stephen & Gugerty, Ryan & Lee, Miranda J. & Landsverk, John A. & Rolls-Reutz, Jennifer & Green, Brandn, 2021. "State behavioral health agency website references to evidence-based program registers," Evaluation and Program Planning, Elsevier, vol. 85(C).
    4. Magura, Stephen & Lee-Easton, Miranda J. & Abu-Obaid, Ruqayyah N. & Landsverk, John & DeCamp, Whitney & Rolls-Reutz, Jennifer & Moore, Kristin & Firpo-Triplett, Regina & Buckley, Pamela R. & Stout, El, 2023. "The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare," Evaluation and Program Planning, Elsevier, vol. 97(C).
    5. Iudici, Antonio & Gagliardo Corsi, Agnese, 2017. "Evaluation in the field of social services for minors: Measuring the efficacy of interventions in the Italian service for health protection and promotion," Evaluation and Program Planning, Elsevier, vol. 61(C), pages 160-168.
    6. Pamela R. Buckley & Abigail A. Fagan & Fred C. Pampel & Karl G. Hill, 2020. "Making Evidence-Based Interventions Relevant for Users: A Comparison of Requirements for Dissemination Readiness Across Program Registries," Evaluation Review, , vol. 44(1), pages 51-83, February.
    7. Carr, E. Summerson & Obertino-Norwood, Hannah, 2022. "Legitimizing evidence: The trans-institutional life of evidence-based practice," Social Science & Medicine, Elsevier, vol. 310(C).
    8. Axford, Nick & Morpeth, Louise & Bjornstad, Gretchen & Hobbs, Tim & Berry, Vashti, 2022. "“What works” registries of interventions to improve child and youth psychosocial outcomes: A critical appraisal," Children and Youth Services Review, Elsevier, vol. 137(C).
    9. Zack, Melissa K. & Karre, Jennifer K. & Olson, Jonathan & Perkins, Daniel F., 2019. "Similarities and differences in program registers: A case study," Evaluation and Program Planning, Elsevier, vol. 76(C), pages 1-1.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maranda, Michael J. & Magura, Stephen & Gugerty, Ryan & Lee, Miranda J. & Landsverk, John A. & Rolls-Reutz, Jennifer & Green, Brandn, 2021. "State behavioral health agency website references to evidence-based program registers," Evaluation and Program Planning, Elsevier, vol. 85(C).
    2. Dreesens, Dunja & Kremer, Leontien & Burgers, Jako & van der Weijden, Trudy, 2020. "Lost in definitions: Reducing duplication and clarifying definitions of knowledge and decision support tools. A RAND-modified Delphi consensus study," Health Policy, Elsevier, vol. 124(5), pages 531-539.
    3. Blake M. Louscher & Veerasathpurush Allareddy & Satheesh Elangovan, 2019. "Predictors of Citations of Systematic Reviews in Oral Implantology: A Cross-Sectional Bibliometric Analysis," SAGE Open, , vol. 9(1), pages 21582440198, March.
    4. Hansen, Henrik & Trifkovic, Neda, 2013. "Systematic Reviews: Questions, Methods and Usage," MPRA Paper 47993, University Library of Munich, Germany.
    5. Mark J Bolland & Andrew Grey, 2014. "A Case Study of Discordant Overlapping Meta-Analyses: Vitamin D Supplements and Fracture," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-20, December.
    6. Shi, Xuanyu & Du, Jian, 2022. "Distinguishing transformative from incremental clinical evidence: A classifier of clinical research using textual features from abstracts and citing sentences," Journal of Informetrics, Elsevier, vol. 16(2).
    7. Jason Portenoy & Jevin D. West, 2020. "Constructing and evaluating automated literature review systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 3233-3251, December.
    8. Dreesens, Dunja & Kremer, Leontien & van der Weijden, Trudy, 2019. "The Dutch chaos case: A scoping review of knowledge and decision support tools available to clinicians in the Netherlands," Health Policy, Elsevier, vol. 123(12), pages 1288-1297.
    9. Joerg J Meerpohl & Florian Herrle & Gerd Antes & Erik von Elm, 2012. "Scientific Value of Systematic Reviews: Survey of Editors of Core Clinical Journals," PLOS ONE, Public Library of Science, vol. 7(5), pages 1-5, May.
    10. Lucy Turner & James Galipeau & Chantelle Garritty & Eric Manheimer & L Susan Wieland & Fatemeh Yazdi & David Moher, 2013. "An Evaluation of Epidemiological and Reporting Characteristics of Complementary and Alternative Medicine (CAM) Systematic Reviews (SRs)," PLOS ONE, Public Library of Science, vol. 8(1), pages 1-11, January.
    11. K. M. Saif‐Ur‐Rahman & Md. Hasan & Shahed Hossain & Iqbal Anwar & Yoshihisa Hirakawa & Hiroshi Yatsuya, 2022. "Prioritization and sequential exclusion of articles in systematic reviews," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(2), June.
    12. Monique Heijmans & Rune Poortvliet & Marieke Van der Gaag & Ana I. González-González & Jessica Beltran Puerta & Carlos Canelo-Aybar & Claudia Valli & Marta Ballester & Claudio Rocha & Montserrat León , 2022. "Using a Taxonomy to Systematically Identify and Describe Self-Management Interventions Components in Randomized Trials for COPD," IJERPH, MDPI, vol. 19(19), pages 1-24, October.
    13. Tanja Burgard & Holger Steinmetz, 2023. "Evidence in management science related to psychology: benefits, tools, and an example of a community-augmented meta-analysis," Management Review Quarterly, Springer, vol. 73(3), pages 1135-1150, September.
    14. Laura Sheble, 2017. "Macro‐level diffusion of a methodological knowledge innovation: Research synthesis methods, 1972–2011," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(12), pages 2693-2708, December.
    15. Md Mahbub Hossain & Abida Sultana & Samia Tasnim & Qiping Fan & Ping Ma & E Lisako J McKyer & Neetu Purohit, 2020. "Prevalence of mental disorders among people who are homeless: An umbrella review," International Journal of Social Psychiatry, , vol. 66(6), pages 528-541, September.
    16. Costa, King & Ntsobi, Mfanelo Patrick, 2023. "A Systematic Review of the Potential for Promotion of Southern Epistemologies in Educational Research: Ubuntu Philosophy as a Research Paradigm, a Conceptual Model," AfricArxiv qshp8, Center for Open Science.
    17. Porjai Pattanittum & Malinee Laopaiboon & David Moher & Pisake Lumbiganon & Chetta Ngamjarus, 2012. "A Comparison of Statistical Methods for Identifying Out-of-Date Systematic Reviews," PLOS ONE, Public Library of Science, vol. 7(11), pages 1-6, November.
    18. Gillian L Currie & Helena N Angel-Scott & Lesley Colvin & Fala Cramond & Kaitlyn Hair & Laila Khandoker & Jing Liao & Malcolm Macleod & Sarah K McCann & Rosie Morland & Nicki Sherratt & Robert Stewart, 2019. "Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-34, May.
    19. Melanie Lum & Luke Wolfenden & Jannah Jones & Alice Grady & Hayley Christian & Kathryn Reilly & Sze Lin Yoong, 2022. "Interventions to Improve Child Physical Activity in the Early Childhood Education and Care Setting: An Umbrella Review," IJERPH, MDPI, vol. 19(4), pages 1-22, February.
    20. Tiago S. Jesus & Helen Hoenig & Michel D. Landry, 2020. "Development of the Rehabilitation Health Policy, Systems, and Services Research Field: Quantitative Analyses of Publications over Time (1990–2017) and across Country Type," IJERPH, MDPI, vol. 17(3), pages 1-17, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:48:y:2015:i:c:p:100-116. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.