IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v172y2009i1p21-47.html
   My bibliography  Save this article

Bias modelling in evidence synthesis

Author

Listed:
  • Rebecca M. Turner
  • David J. Spiegelhalter
  • Gordon C. S. Smith
  • Simon G. Thompson

Abstract

Summary. Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question. We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis. The methods are developed in the context of reanalysing a UK National Institute for Clinical Excellence technology appraisal in antenatal care, which includes eight comparative studies. Many were historically controlled, only one was a randomized trial and doses, populations and outcomes varied between studies and differed from the target UK setting. Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias‐adjusted meta‐analysis. Adjustment had the effect of shifting the combined estimate away from the null by approximately 10%, and the variance of the combined estimate was almost tripled. Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant studies downweighted by using computationally simple methods.

Suggested Citation

  • Rebecca M. Turner & David J. Spiegelhalter & Gordon C. S. Smith & Simon G. Thompson, 2009. "Bias modelling in evidence synthesis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 21-47, January.
    • Handle: RePEc:bla:jorssa:v:172:y:2009:i:1:p:21-47
    DOI: 10.1111/j.1467-985X.2008.00547.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-985X.2008.00547.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1467-985X.2008.00547.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sander Greenland, 2005. "Multiple‐bias modelling for analysis of observational data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 267-306, March.
    2. Jeremy E. Oakley & Anthony O'Hagan, 2007. "Uncertainty in prior elicitations: a nonparametric approach," Biometrika, Biometrika Trust, vol. 94(2), pages 427-441.
    3. A. E. Ades & A. J. Sutton, 2006. "Multiparameter evidence synthesis in epidemiology and medical decision‐making: current approaches," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(1), pages 5-35, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. van der Bles, Anne Marthe & van der Liden, Sander & Freeman, Alessandra L. J. & Mitchell, James & Galvao, Ana Beatriz & Spiegelhalter, David J., 2019. "Communicating uncertainty about facts, numbers, and science," EMF Research Papers 22, Economic Modelling and Forecasting Group.
    2. Karla Hemming & Peter J Chilton & Richard J Lilford & Anthony Avery & Aziz Sheikh, 2012. "Bayesian Cohort and Cross-Sectional Analyses of the PINCER Trial: A Pharmacist-Led Intervention to Reduce Medication Errors in Primary Care," PLOS ONE, Public Library of Science, vol. 7(6), pages 1-8, June.
    3. Giancarlo Manzi & David J. Spiegelhalter & Rebecca M. Turner & Julian Flowers & Simon G. Thompson, 2011. "Modelling bias in combining small area prevalence estimates from multiple surveys," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(1), pages 31-50, January.
    4. John J. Graff & Nalini Sathiakumar & Maurizio Macaluso & George Maldonado & Robert Matthews & Elizabeth Delzell, 2009. "The Effect of Uncertainty in Exposure Estimation on the Exposure-Response Relation between 1,3-Butadiene and Leukemia," IJERPH, MDPI, vol. 6(9), pages 1-20, September.
    5. Geneletti, Sara & Mason, Alexina & Best, Nicky, 2011. "Adjusting for selection effects in epidemiologic studies: why sensitivity analysis is the only “solution”," LSE Research Online Documents on Economics 31520, London School of Economics and Political Science, LSE Library.
    6. Desiree C Wilks & Stephen J Sharp & Ulf Ekelund & Simon G Thompson & Adrian P Mander & Rebecca M Turner & Susan A Jebb & Anna Karin Lindroos, 2011. "Objectively Measured Physical Activity and Fat Mass in Children: A Bias-Adjusted Meta-Analysis of Prospective Studies," PLOS ONE, Public Library of Science, vol. 6(2), pages 1-8, February.
    7. McCandless Lawrence C., 2012. "Meta-Analysis of Observational Studies with Unmeasured Confounders," The International Journal of Biostatistics, De Gruyter, vol. 8(2), pages 1-31, January.
    8. Christopher H. Jackson & Linda D. Sharples & Simon G. Thompson, 2010. "Structural and parameter uncertainty in Bayesian cost‐effectiveness models," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 59(2), pages 233-253, March.
    9. S. Dias & N. J. Welton & V. C. C. Marinho & G. Salanti & J. P. T. Higgins & A. E. Ades, 2010. "Estimation and adjustment of bias in randomized evidence by using mixed treatment comparison meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(3), pages 613-629, July.
    10. Ian Wadsworth & Lisa V. Hampson & Thomas Jaki & Graeme J. Sills & Anthony G. Marson & Richard Appleton, 2020. "A quantitative framework to inform extrapolation decisions in children," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 515-534, February.
    11. Rebecca M Turner & Myfanwy Lloyd-Jones & Dilly O C Anumba & Gordon C S Smith & David J Spiegelhalter & Hazel Squires & John W Stevens & Michael J Sweeting & Stanislaw J Urbaniak & Robert Webster & Sim, 2012. "Routine Antenatal Anti-D Prophylaxis in Women Who Are Rh(D) Negative: Meta-Analyses Adjusted for Differences in Study Design and Quality," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-10, February.
    12. Petra Schnell‐Inderst & Cynthia P. Iglesias & MARJAN Arvandi & ORIANA Ciani & Raffaella Matteucci Gothe & Jaime Peters & Ashley W. Blom & Rod S. Taylor & Uwe Siebert, 2017. "A bias‐adjusted evidence synthesis of RCT and observational data: the case of total hip replacement," Health Economics, John Wiley & Sons, Ltd., vol. 26(S1), pages 46-69, February.
    13. Danila Azzolina & Paola Berchialla & Dario Gregori & Ileana Baldi, 2021. "Prior Elicitation for Use in Clinical Trial Design and Analysis: A Literature Review," IJERPH, MDPI, vol. 18(4), pages 1-21, February.
    14. K. M. Rhodes & J. Savović & R. Elbers & H. E. Jones & J. P. T. Higgins & J. A. C. Sterne & N. J. Welton & R. M. Turner, 2020. "Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(1), pages 193-209, January.
    15. Isabelle Albert & Emmanuelle Espié & Henriette de Valk & Jean‐Baptiste Denis, 2011. "A Bayesian Evidence Synthesis for Estimating Campylobacteriosis Prevalence," Risk Analysis, John Wiley & Sons, vol. 31(7), pages 1141-1155, July.
    16. Giancarlo MANZI & Pier Alda FERRARI, "undated". "Statistical methods for evaluating satisfaction with public services Abstract: Contrary to private enterprises, public enterprises can be unaware of the impact of their performance when providing serv," CIRIEC Working Papers 1404, CIRIEC - Université de Liège.
    17. David M. Phillippo & Sofia Dias & A. E. Ades & Vanessa Didelez & Nicky J. Welton, 2018. "Sensitivity of treatment recommendations to bias in network meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(3), pages 843-867, June.
    18. Davis, Alexander L. & Krishnamurti, Tamar & Fischhoff, Baruch & Bruine de Bruin, Wandi, 2013. "Setting a standard for electricity pilot studies," Energy Policy, Elsevier, vol. 62(C), pages 401-409.
    19. C Elizabeth McCarron & Eleanor M Pullenayegum & Lehana Thabane & Ron Goeree & Jean-Eric Tarride, 2011. "Bayesian Hierarchical Models Combining Different Study Types and Adjusting for Covariate Imbalances: A Simulation Study to Assess Model Performance," PLOS ONE, Public Library of Science, vol. 6(10), pages 1-7, October.
    20. Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. A. M. Presanis & D. De Angelis & D. J. Spiegelhalter & S. Seaman & A. Goubar & A. E. Ades, 2008. "Conflicting evidence in a Bayesian synthesis of surveillance data to estimate human immunodeficiency virus prevalence," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(4), pages 915-937, October.
    2. Lawrence C. McCandless & Sylvia Richardson & Nicky Best, 2012. "Adjustment for Missing Confounders Using External Validation Data and Propensity Scores," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 40-51, March.
    3. Julian P. T. Higgins & Simon G. Thompson & David J. Spiegelhalter, 2009. "A re‐evaluation of random‐effects meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 137-159, January.
    4. Douglas E. Schaubel & Guanghui Wei, 2011. "Double Inverse-Weighted Estimation of Cumulative Treatment Effects Under Nonproportional Hazards and Dependent Censoring," Biometrics, The International Biometric Society, vol. 67(1), pages 29-38, March.
    5. Carlos Díaz-Venegas, 2014. "Identifying the Confounders of Marginalization and Mortality in Mexico, 2003–2007," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 118(2), pages 851-875, September.
    6. A. Goubar & A. E. Ades & D. De Angelis & C. A. McGarrigle & C. H. Mercer & P. A. Tookey & K. Fenton & O. N. Gill, 2008. "Estimates of human immunodeficiency virus prevalence and proportion diagnosed based on Bayesian multiparameter synthesis of surveillance data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(3), pages 541-580, June.
    7. McCandless Lawrence C., 2012. "Meta-Analysis of Observational Studies with Unmeasured Confounders," The International Journal of Biostatistics, De Gruyter, vol. 8(2), pages 1-31, January.
    8. Maria Gheorghe & Susan Picavet & Monique Verschuren & Werner B. F. Brouwer & Pieter H. M. Baal, 2017. "Health losses at the end of life: a Bayesian mixed beta regression approach," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(3), pages 723-749, June.
    9. K. M. Rhodes & J. Savović & R. Elbers & H. E. Jones & J. P. T. Higgins & J. A. C. Sterne & N. J. Welton & R. M. Turner, 2020. "Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(1), pages 193-209, January.
    10. Andrew Gelman & Christian Hennig, 2017. "Beyond subjective and objective in statistics," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(4), pages 967-1033, October.
    11. S. Dias & N. J. Welton & V. C. C. Marinho & G. Salanti & J. P. T. Higgins & A. E. Ades, 2010. "Estimation and adjustment of bias in randomized evidence by using mixed treatment comparison meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(3), pages 613-629, July.
    12. Qi Zhou & Yoo-Mi Chin & James D. Stamey & Joon Jin Song, 2020. "Bayesian sensitivity analysis to unmeasured confounding for misclassified data," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 104(4), pages 577-596, December.
    13. de Luna, Xavier & Lundin, Mathias, 2009. "Sensitivity analysis of the unconfoundedness assumption in observational studies," Working Paper Series 2009:12, IFAU - Institute for Evaluation of Labour Market and Education Policy.
    14. Leah Comment & Brent A. Coull & Corwin Zigler & Linda Valeri, 2022. "Bayesian data fusion: Probabilistic sensitivity analysis for unmeasured confounding using informative priors based on secondary data," Biometrics, The International Biometric Society, vol. 78(2), pages 730-741, June.
    15. N. J. Welton & A. E. Ades & J. B. Carlin & D. G. Altman & J. A. C. Sterne, 2009. "Models for potentially biased evidence in meta‐analysis using empirically based priors," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 119-136, January.
    16. Enrique A. Navarro-Camba & Jaume Segura-García & Claudio Gomez-Perretta, 2018. "Exposure to 50 Hz Magnetic Fields in Homes and Areas Surrounding Urban Transformer Stations in Silla (Spain): Environmental Impact Assessment," Sustainability, MDPI, vol. 10(8), pages 1-11, July.
    17. Michael D. Teter & Johannes O. Royset & Alexandra M. Newman, 2019. "Modeling uncertainty of expert elicitation for use in risk-based optimization," Annals of Operations Research, Springer, vol. 280(1), pages 189-210, September.
    18. P. Gustafson & L. C. McCandless & A. R. Levy & S. Richardson, 2010. "Simplified Bayesian Sensitivity Analysis for Mismeasured and Unobserved Confounders," Biometrics, The International Biometric Society, vol. 66(4), pages 1129-1137, December.
    19. Leeka Kheifets & Jack D. Sahl & Riti Shimkhada & Mike H. Repacholi, 2005. "Developing Policy in the Face of Scientific Uncertainty: Interpreting 0.3 μT or 0.4 μT Cutpoints from EMF Epidemiologic Studies," Risk Analysis, John Wiley & Sons, vol. 25(4), pages 927-935, August.
    20. Graham Scotland & Stirling Bryan, 2017. "Why Do Health Economists Promote Technology Adoption Rather Than the Search for Efficiency? A Proposal for a Change in Our Approach to Economic Evaluation in Health Care," Medical Decision Making, , vol. 37(2), pages 139-147, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:172:y:2009:i:1:p:21-47. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.