IDEAS home Printed from
   My bibliography  Save this article

Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies


  • Amr Makady

    (The National Healthcare Institute (ZIN)
    Utrecht Institute for Pharmaceutical Sciences)

  • Ard Veelen

    (Utrecht Institute for Pharmaceutical Sciences)

  • Páll Jonsson

    (The National Institute for Health and Care Excellence (NICE))

  • Owen Moseley

    (The Scottish Medicines Consortium (SMC), Healthcare Improvement Scotland (HIS))

  • Anne D’Andon

    (La Haute Autorité de Santé (HAS))

  • Anthonius Boer

    (Utrecht Institute for Pharmaceutical Sciences)

  • Hans Hillege

    (University Medical Centre Groningen)

  • Olaf Klungel

    (Utrecht Institute for Pharmaceutical Sciences)

  • Wim Goettsch

    (The National Healthcare Institute (ZIN)
    Utrecht Institute for Pharmaceutical Sciences)


Background Reimbursement decisions are conventionally based on evidence from randomised controlled trials (RCTs), which often have high internal validity but low external validity. Real-world data (RWD) may provide complimentary evidence for relative effectiveness assessments (REAs) and cost-effectiveness assessments (CEAs). This study examines whether RWD is incorporated in health technology assessment (HTA) of melanoma drugs by European HTA agencies, as well as differences in RWD use between agencies and across time. Methods HTA reports published between 1 January 2011 and 31 December 2016 were retrieved from websites of agencies representing five jurisdictions: England [National Institute for Health and Care Excellence (NICE)], Scotland [Scottish Medicines Consortium (SMC)], France [Haute Autorité de santé (HAS)], Germany [Institute for Quality and Efficacy in Healthcare (IQWiG)] and The Netherlands [Zorginstituut Nederland (ZIN)]. A standardized data extraction form was used to extract information on RWD inclusion for both REAs and CEAs. Results Overall, 52 reports were retrieved, all of which contained REAs; CEAs were present in 25 of the reports. RWD was included in 28 of the 52 REAs (54%), mainly to estimate melanoma prevalence, and in 22 of the 25 (88%) CEAs, mainly to extrapolate long-term effectiveness and/or identify drug-related costs. Differences emerged between agencies regarding RWD use in REAs; the ZIN and IQWiG cited RWD for evidence on prevalence, whereas the NICE, SMC and HAS additionally cited RWD use for drug effectiveness. No visible trend for RWD use in REAs and CEAs over time was observed. Conclusion In general, RWD inclusion was higher in CEAs than REAs, and was mostly used to estimate melanoma prevalence in REAs or to predict long-term effectiveness in CEAs. Differences emerged between agencies’ use of RWD; however, no visible trends for RWD use over time were observed.

Suggested Citation

  • Amr Makady & Ard Veelen & Páll Jonsson & Owen Moseley & Anne D’Andon & Anthonius Boer & Hans Hillege & Olaf Klungel & Wim Goettsch, 2018. "Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies," PharmacoEconomics, Springer, vol. 36(3), pages 359-368, March.
  • Handle: RePEc:spr:pharme:v:36:y:2018:i:3:d:10.1007_s40273-017-0596-z
    DOI: 10.1007/s40273-017-0596-z

    Download full text from publisher

    File URL:
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL:
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Darío Rubio-Rodríguez & Silvia Diego Blanco & Maite Pérez & Carlos Rubio-Terrés, 2017. "Cost-Effectiveness of Drug Treatments for Advanced Melanoma: A Systematic Literature Review," PharmacoEconomics, Springer, vol. 35(9), pages 879-893, September.
    2. Briggs, Andrew & Sculpher, Mark & Claxton, Karl, 2006. "Decision Modelling for Health Economic Evaluation," OUP Catalogue, Oxford University Press, number 9780198526629, Decembrie.
    3. Drummond, Michael F. & Sculpher, Mark J. & Claxton, Karl & Stoddart, Greg L. & Torrance, George W., 2015. "Methods for the Economic Evaluation of Health Care Programmes," OUP Catalogue, Oxford University Press, edition 4, number 9780199665884, Decembrie.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Philip Klein & Hedwig Blommestein & Maiwenn Al & Benedetta Pongiglione & Aleksandra Torbica & Saskia de Groot, 2022. "Real‐world evidence in health technology assessment of high‐risk medical devices: Fit for purpose?," Health Economics, John Wiley & Sons, Ltd., vol. 31(S1), pages 10-24, September.
    2. Wranik, Wiesława Dominika & Zielińska, Dorota Anna & Gambold, Liesl & Sevgur, Serperi, 2019. "Threats to the value of Health Technology Assessment: Qualitative evidence from Canada and Poland," Health Policy, Elsevier, vol. 123(2), pages 191-202.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chiranjeev Sanyal & Don Husereau, 2020. "Systematic Review of Economic Evaluations of Services Provided by Community Pharmacists," Applied Health Economics and Health Policy, Springer, vol. 18(3), pages 375-392, June.
    2. Qi Cao & Erik Buskens & Hans L. Hillege & Tiny Jaarsma & Maarten Postma & Douwe Postmus, 2019. "Stratified treatment recommendation or one-size-fits-all? A health economic insight based on graphical exploration," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(3), pages 475-482, April.
    3. Yasuhiro Hagiwara & Takeru Shiroiwa, 2022. "Estimating Value-Based Price and Quantifying Uncertainty around It in Health Technology Assessment: Frequentist and Bayesian Approaches," Medical Decision Making, , vol. 42(5), pages 672-683, July.
    4. Matthew Franklin & James Lomas & Gerry Richardson, 2020. "Conducting Value for Money Analyses for Non-randomised Interventional Studies Including Service Evaluations: An Educational Review with Recommendations," PharmacoEconomics, Springer, vol. 38(7), pages 665-681, July.
    5. David Brain & Ruth Tulleners & Xing Lee & Qinglu Cheng & Nicholas Graves & Rosana Pacella, 2019. "Cost-effectiveness analysis of an innovative model of care for chronic wounds patients," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-13, March.
    6. Osvaldo Ulises Garay & Marie Libérée Nishimwe & Marwân-al-Qays Bousmah & Asmaa Janah & Pierre-Marie Girard & Geneviève Chêne & Laetitia Moinot & Luis Sagaon-Teyssier & Jean-Luc Meynard & Bruno Spire &, 2019. "Cost-Effectiveness Analysis of Lopinavir/Ritonavir Monotherapy Versus Standard Combination Antiretroviral Therapy in HIV-1 Infected Patients with Viral Suppression in France (ANRS 140 DREAM)," PharmacoEconomics - Open, Springer, vol. 3(4), pages 505-515, December.
    7. Andrija S Grustam & Nasuh Buyukkaramikli & Ron Koymans & Hubertus J M Vrijhoef & Johan L Severens, 2019. "Value of information analysis in telehealth for chronic heart failure management," PLOS ONE, Public Library of Science, vol. 14(6), pages 1-23, June.
    8. Caroline S. Clarke & Mariya Melnychuk & Angus I. G. Ramsay & Cecilia Vindrola-Padros & Claire Levermore & Ravi Barod & Axel Bex & John Hines & Muntzer M. Mughal & Kathy Pritchard-Jones & Maxine Tran &, 2022. "Cost-Utility Analysis of Major System Change in Specialist Cancer Surgery in London, England, Using Linked Patient-Level Electronic Health Records and Difference-in-Differences Analysis," Applied Health Economics and Health Policy, Springer, vol. 20(6), pages 905-917, November.
    9. Kasper M. Johannesen & Karl Claxton & Mark J. Sculpher & Allan J. Wailoo, 2018. "How to design the cost‐effectiveness appraisal process of new healthcare technologies to maximise population health: A conceptual framework," Health Economics, John Wiley & Sons, Ltd., vol. 27(2), pages 41-54, February.
    10. Matthew Franklin & James Lomas & Simon Walker & Tracey Young, 2019. "An Educational Review About Using Cost Data for the Purpose of Cost-Effectiveness Analysis," PharmacoEconomics, Springer, vol. 37(5), pages 631-643, May.
    11. Edward Cox & Simon Walker & Charlotte L. Edwardson & Stuart J. H. Biddle & Alexandra M. Clarke-Cornwell & Stacy A. Clemes & Melanie J. Davies & David W. Dunstan & Helen Eborall & Malcolm H. Granat & L, 2022. "The Cost-Effectiveness of the SMART Work & Life Intervention for Reducing Sitting Time," IJERPH, MDPI, vol. 19(22), pages 1-14, November.
    12. Tom L. Drake & Yoel Lubell, 2017. "Malaria and Economic Evaluation Methods: Challenges and Opportunities," Applied Health Economics and Health Policy, Springer, vol. 15(3), pages 291-297, June.
    13. Eric Kaun Santos Silva & June Alisson Westarb Cruz & Maria Alexandra Viegas Cortez Cunha & Thyago Proença Moraes & Sandro Marques & Eduardo Damião Silva, 2021. "Cost-effectiveness in health: consolidated research and contemporary challenges," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-10, December.
    14. Hill, Sarah R. & Vale, Luke & Hunter, David & Henderson, Emily & Oluboyede, Yemi, 2017. "Economic evaluations of alcohol prevention interventions: Is the evidence sufficient? A review of methodological challenges," Health Policy, Elsevier, vol. 121(12), pages 1249-1262.
    15. Devin Incerti & Jeffrey R. Curtis & Jason Shafrin & Darius N. Lakdawalla & Jeroen P. Jansen, 2019. "A Flexible Open-Source Decision Model for Value Assessment of Biologic Treatment for Rheumatoid Arthritis," PharmacoEconomics, Springer, vol. 37(6), pages 829-843, June.
    16. Marwân-al-Qays Bousmah & Marie Libérée Nishimwe & Tamara Tovar-Sanchez & Martial Lantche Wandji & Mireille Mpoudi-Etame & Gwenaëlle Maradan & Pierrette Omgba Bassega & Marie Varloteaux & Alice Montoyo, 2021. "Cost-Utility Analysis of a Dolutegravir-Based Versus Low-Dose Efavirenz-Based Regimen for the Initial Treatment of HIV-Infected Patients in Cameroon (NAMSAL ANRS 12313 Trial)," PharmacoEconomics, Springer, vol. 39(3), pages 331-343, March.
    17. Salah Ghabri & Erwan Autin & Anne-Isabelle Poullié & Jean Michel Josselin, 2018. "The French National Authority for Health (HAS) Guidelines for Conducting Budget Impact Analyses (BIA)," PharmacoEconomics, Springer, vol. 36(4), pages 407-417, April.
    18. Jonas Steel & Lode Godderis & Jeroen Luyten, 2018. "Methodological Challenges in the Economic Evaluation of Occupational Health and Safety Programmes," IJERPH, MDPI, vol. 15(11), pages 1-12, November.
    19. Ashley Layer & Emma McManus & N. J. Levell, 2020. "A Systematic Review of Model-Based Economic Evaluations of Treatments for Venous Leg Ulcers," PharmacoEconomics - Open, Springer, vol. 4(2), pages 211-222, June.
    20. Helen Weatherly & Rita Faria & Bernard Van den Berg & Mark Sculpher & Peter O’Neill & Kay Nolan & Julie Glanville & Jaana Isojarvi & Erin Baragula & Mary Edwards, 2017. "Scoping review on social care economic evaluation methods," Working Papers 150cherp, Centre for Health Economics, University of York.

    More about this item


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:pharme:v:36:y:2018:i:3:d:10.1007_s40273-017-0596-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.