IDEAS home Printed from https://ideas.repec.org/a/eee/enscpo/v64y2016icp75-82.html
   My bibliography  Save this article

The reliability of evidence review methodology in environmental science and conservation

Author

Listed:
  • O’Leary, Bethan C.
  • Kvist, Kristian
  • Bayliss, Helen R.
  • Derroire, Géraldine
  • Healey, John R.
  • Hughes, Kathryn
  • Kleinschroth, Fritz
  • Sciberras, Marija
  • Woodcock, Paul
  • Pullin, Andrew S.

Abstract

Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0–30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decision-makers to select reviews that are most likely to accurately reflect the evidence base.

Suggested Citation

  • O’Leary, Bethan C. & Kvist, Kristian & Bayliss, Helen R. & Derroire, Géraldine & Healey, John R. & Hughes, Kathryn & Kleinschroth, Fritz & Sciberras, Marija & Woodcock, Paul & Pullin, Andrew S., 2016. "The reliability of evidence review methodology in environmental science and conservation," Environmental Science & Policy, Elsevier, vol. 64(C), pages 75-82.
    • Handle: RePEc:eee:enscpo:v:64:y:2016:i:c:p:75-82
    DOI: 10.1016/j.envsci.2016.06.012
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1462901116303318
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.envsci.2016.06.012?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiménez-Olivencia, Yolanda & Ibáñez-Jiménez, Álvaro & Porcel-Rodríguez, Laura & Zimmerer, Karl, 2021. "Land use change dynamics in Euro-mediterranean mountain regions: Driving forces and consequences for the landscape," Land Use Policy, Elsevier, vol. 109(C).
    2. Nicole Stremlau & Anna Tsalapatanis, 2022. "Social Media, Mobile Phones and Migration in Africa: A Review of the Evidence," Progress in Development Studies, , vol. 22(1), pages 56-71, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:enscpo:v:64:y:2016:i:c:p:75-82. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/environmental-science-and-policy/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.