IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v172y2009i1p21-47.html
   My bibliography  Save this article

Bias modelling in evidence synthesis

Author

Listed:
  • Rebecca M. Turner
  • David J. Spiegelhalter
  • Gordon C. S. Smith
  • Simon G. Thompson

Abstract

Summary. Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question. We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis. The methods are developed in the context of reanalysing a UK National Institute for Clinical Excellence technology appraisal in antenatal care, which includes eight comparative studies. Many were historically controlled, only one was a randomized trial and doses, populations and outcomes varied between studies and differed from the target UK setting. Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias‐adjusted meta‐analysis. Adjustment had the effect of shifting the combined estimate away from the null by approximately 10%, and the variance of the combined estimate was almost tripled. Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant studies downweighted by using computationally simple methods.

Suggested Citation

  • Rebecca M. Turner & David J. Spiegelhalter & Gordon C. S. Smith & Simon G. Thompson, 2009. "Bias modelling in evidence synthesis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 21-47, January.
    • Handle: RePEc:bla:jorssa:v:172:y:2009:i:1:p:21-47
    DOI: 10.1111/j.1467-985X.2008.00547.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-985X.2008.00547.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1467-985X.2008.00547.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sander Greenland, 2005. "Multiple‐bias modelling for analysis of observational data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 267-306, March.
    2. Jeremy E. Oakley & Anthony O'Hagan, 2007. "Uncertainty in prior elicitations: a nonparametric approach," Biometrika, Biometrika Trust, vol. 94(2), pages 427-441.
    3. A. E. Ades & A. J. Sutton, 2006. "Multiparameter evidence synthesis in epidemiology and medical decision‐making: current approaches," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(1), pages 5-35, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Petra Schnell‐Inderst & Cynthia P. Iglesias & MARJAN Arvandi & ORIANA Ciani & Raffaella Matteucci Gothe & Jaime Peters & Ashley W. Blom & Rod S. Taylor & Uwe Siebert, 2017. "A bias‐adjusted evidence synthesis of RCT and observational data: the case of total hip replacement," Health Economics, John Wiley & Sons, Ltd., vol. 26(S1), pages 46-69, February.
    2. Danila Azzolina & Paola Berchialla & Dario Gregori & Ileana Baldi, 2021. "Prior Elicitation for Use in Clinical Trial Design and Analysis: A Literature Review," IJERPH, MDPI, vol. 18(4), pages 1-21, February.
    3. Davis, Alexander L. & Krishnamurti, Tamar & Fischhoff, Baruch & Bruine de Bruin, Wandi, 2013. "Setting a standard for electricity pilot studies," Energy Policy, Elsevier, vol. 62(C), pages 401-409.
    4. K. M. Rhodes & J. Savović & R. Elbers & H. E. Jones & J. P. T. Higgins & J. A. C. Sterne & N. J. Welton & R. M. Turner, 2020. "Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(1), pages 193-209, January.
    5. Christopher H. Jackson & Linda D. Sharples & Simon G. Thompson, 2010. "Structural and parameter uncertainty in Bayesian cost‐effectiveness models," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 59(2), pages 233-253, March.
    6. repec:osf:thesis:kd6ja_v1 is not listed on IDEAS
    7. Isabelle Albert & Emmanuelle Espié & Henriette de Valk & Jean‐Baptiste Denis, 2011. "A Bayesian Evidence Synthesis for Estimating Campylobacteriosis Prevalence," Risk Analysis, John Wiley & Sons, vol. 31(7), pages 1141-1155, July.
    8. C Elizabeth McCarron & Eleanor M Pullenayegum & Lehana Thabane & Ron Goeree & Jean-Eric Tarride, 2011. "Bayesian Hierarchical Models Combining Different Study Types and Adjusting for Covariate Imbalances: A Simulation Study to Assess Model Performance," PLOS ONE, Public Library of Science, vol. 6(10), pages 1-7, October.
    9. S. Dias & N. J. Welton & V. C. C. Marinho & G. Salanti & J. P. T. Higgins & A. E. Ades, 2010. "Estimation and adjustment of bias in randomized evidence by using mixed treatment comparison meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(3), pages 613-629, July.
    10. Karla Hemming & Peter J Chilton & Richard J Lilford & Anthony Avery & Aziz Sheikh, 2012. "Bayesian Cohort and Cross-Sectional Analyses of the PINCER Trial: A Pharmacist-Led Intervention to Reduce Medication Errors in Primary Care," PLOS ONE, Public Library of Science, vol. 7(6), pages 1-8, June.
    11. Giancarlo Manzi & David J. Spiegelhalter & Rebecca M. Turner & Julian Flowers & Simon G. Thompson, 2011. "Modelling bias in combining small area prevalence estimates from multiple surveys," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(1), pages 31-50, January.
    12. Ian Wadsworth & Lisa V. Hampson & Thomas Jaki & Graeme J. Sills & Anthony G. Marson & Richard Appleton, 2020. "A quantitative framework to inform extrapolation decisions in children," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 515-534, February.
    13. Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.
    14. John J. Graff & Nalini Sathiakumar & Maurizio Macaluso & George Maldonado & Robert Matthews & Elizabeth Delzell, 2009. "The Effect of Uncertainty in Exposure Estimation on the Exposure-Response Relation between 1,3-Butadiene and Leukemia," IJERPH, MDPI, vol. 6(9), pages 1-20, September.
    15. Giancarlo MANZI & Pier Alda FERRARI, "undated". "Statistical methods for evaluating satisfaction with public services Abstract: Contrary to private enterprises, public enterprises can be unaware of the impact of their performance when providing serv," CIRIEC Working Papers 1404, CIRIEC - Université de Liège.
    16. Geneletti, Sara & Mason, Alexina & Best, Nicky, 2011. "Adjusting for selection effects in epidemiologic studies: why sensitivity analysis is the only “solution”," LSE Research Online Documents on Economics 31520, London School of Economics and Political Science, LSE Library.
    17. Desiree C Wilks & Stephen J Sharp & Ulf Ekelund & Simon G Thompson & Adrian P Mander & Rebecca M Turner & Susan A Jebb & Anna Karin Lindroos, 2011. "Objectively Measured Physical Activity and Fat Mass in Children: A Bias-Adjusted Meta-Analysis of Prospective Studies," PLOS ONE, Public Library of Science, vol. 6(2), pages 1-8, February.
    18. David M. Phillippo & Sofia Dias & A. E. Ades & Vanessa Didelez & Nicky J. Welton, 2018. "Sensitivity of treatment recommendations to bias in network meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(3), pages 843-867, June.
    19. McCandless Lawrence C., 2012. "Meta-Analysis of Observational Studies with Unmeasured Confounders," The International Journal of Biostatistics, De Gruyter, vol. 8(2), pages 1-31, January.
    20. Rebecca M Turner & Myfanwy Lloyd-Jones & Dilly O C Anumba & Gordon C S Smith & David J Spiegelhalter & Hazel Squires & John W Stevens & Michael J Sweeting & Stanislaw J Urbaniak & Robert Webster & Sim, 2012. "Routine Antenatal Anti-D Prophylaxis in Women Who Are Rh(D) Negative: Meta-Analyses Adjusted for Differences in Study Design and Quality," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-10, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. A. M. Presanis & D. De Angelis & D. J. Spiegelhalter & S. Seaman & A. Goubar & A. E. Ades, 2008. "Conflicting evidence in a Bayesian synthesis of surveillance data to estimate human immunodeficiency virus prevalence," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(4), pages 915-937, October.
    2. Lawrence C. McCandless & Sylvia Richardson & Nicky Best, 2012. "Adjustment for Missing Confounders Using External Validation Data and Propensity Scores," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 40-51, March.
    3. Julian P. T. Higgins & Simon G. Thompson & David J. Spiegelhalter, 2009. "A re‐evaluation of random‐effects meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 137-159, January.
    4. Woertman, Willem & Vermeulen, Bram & Groenewoud, Hans & van der Wilt, Gert Jan, 2013. "Evidence based policy decisions through a Bayesian approach: The case of a statin appraisal in the Netherlands," Health Policy, Elsevier, vol. 112(3), pages 234-240.
    5. Douglas E. Schaubel & Guanghui Wei, 2011. "Double Inverse-Weighted Estimation of Cumulative Treatment Effects Under Nonproportional Hazards and Dependent Censoring," Biometrics, The International Biometric Society, vol. 67(1), pages 29-38, March.
    6. Carlos Díaz-Venegas, 2014. "Identifying the Confounders of Marginalization and Mortality in Mexico, 2003–2007," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 118(2), pages 851-875, September.
    7. Isabelle Albert & Emmanuelle Espié & Henriette de Valk & Jean‐Baptiste Denis, 2011. "A Bayesian Evidence Synthesis for Estimating Campylobacteriosis Prevalence," Risk Analysis, John Wiley & Sons, vol. 31(7), pages 1141-1155, July.
    8. A. Goubar & A. E. Ades & D. De Angelis & C. A. McGarrigle & C. H. Mercer & P. A. Tookey & K. Fenton & O. N. Gill, 2008. "Estimates of human immunodeficiency virus prevalence and proportion diagnosed based on Bayesian multiparameter synthesis of surveillance data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(3), pages 541-580, June.
    9. Xavier de Luna & Mathias Lundin, 2014. "Sensitivity analysis of the unconfoundedness assumption with an application to an evaluation of college choice effects on earnings," Journal of Applied Statistics, Taylor & Francis Journals, vol. 41(8), pages 1767-1784, August.
    10. A. E. Ades & A. J. Sutton, 2006. "Multiparameter evidence synthesis in epidemiology and medical decision‐making: current approaches," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(1), pages 5-35, January.
    11. C Elizabeth McCarron & Eleanor M Pullenayegum & Lehana Thabane & Ron Goeree & Jean-Eric Tarride, 2011. "Bayesian Hierarchical Models Combining Different Study Types and Adjusting for Covariate Imbalances: A Simulation Study to Assess Model Performance," PLOS ONE, Public Library of Science, vol. 6(10), pages 1-7, October.
    12. McCandless Lawrence C., 2012. "Meta-Analysis of Observational Studies with Unmeasured Confounders," The International Journal of Biostatistics, De Gruyter, vol. 8(2), pages 1-31, January.
    13. Jiang, Zhichao & Ding, Peng, 2017. "The directions of selection bias," Statistics & Probability Letters, Elsevier, vol. 125(C), pages 104-109.
    14. Maria Gheorghe & Susan Picavet & Monique Verschuren & Werner B. F. Brouwer & Pieter H. M. Baal, 2017. "Health losses at the end of life: a Bayesian mixed beta regression approach," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(3), pages 723-749, June.
    15. Tom Engsted & Jesper W. Schneider, 2024. "Non-Experimental Data, Hypothesis Testing, and the Likelihood Principle: A Social Science Perspective," Foundations and Trends(R) in Econometrics, now publishers, vol. 13(1), pages 1-66, February.
    16. Rebecca M Turner & Myfanwy Lloyd-Jones & Dilly O C Anumba & Gordon C S Smith & David J Spiegelhalter & Hazel Squires & John W Stevens & Michael J Sweeting & Stanislaw J Urbaniak & Robert Webster & Sim, 2012. "Routine Antenatal Anti-D Prophylaxis in Women Who Are Rh(D) Negative: Meta-Analyses Adjusted for Differences in Study Design and Quality," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-10, February.
    17. K. M. Rhodes & J. Savović & R. Elbers & H. E. Jones & J. P. T. Higgins & J. A. C. Sterne & N. J. Welton & R. M. Turner, 2020. "Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(1), pages 193-209, January.
    18. Nuoo‐Ting Molitor & Nicky Best & Chris Jackson & Sylvia Richardson, 2009. "Using Bayesian graphical models to model biases in observational studies and to combine multiple sources of data: application to low birth weight and water disinfection by‐products," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(3), pages 615-637, June.
    19. H. Lu & P. Yin & R.X. Yue & J.Q. Shi, 2015. "Robust confidence intervals for trend estimation in meta-analysis with publication bias," Journal of Applied Statistics, Taylor & Francis Journals, vol. 42(12), pages 2715-2733, December.
    20. Paul Gustafson, 2006. "Sample size implications when biases are modelled rather than ignored," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(4), pages 865-881, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:172:y:2009:i:1:p:21-47. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.