IDEAS home Printed from https://ideas.repec.org/a/ijm/journl/v9y2016i2p106-122.html
   My bibliography  Save this article

Validation of Spatial Microsimulation Models: a Proposal to Adopt the Bland-Altman Method

Author

Listed:
  • Kate A Timmins

    (School of Sport and Exercise Science, University of Lincoln, Lincoln LN6 7TS, United Kingdom)

  • Kimberley L Edwards

    (Arthritis Research UK Centre for Sport Exercise and Osteoarthritis, University of Nottingham, C Floor, South Block, Queens Medical Centre, Nottingham, NG7 2UH, United Kingdom)

Abstract

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The Bland-Altman plot consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R², SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods.

Suggested Citation

  • Kate A Timmins & Kimberley L Edwards, 2016. "Validation of Spatial Microsimulation Models: a Proposal to Adopt the Bland-Altman Method," International Journal of Microsimulation, International Microsimulation Association, vol. 9(2), pages 106-122.
    • Handle: RePEc:ijm:journl:v:9:y:2016:i:2:p:106-122
    as

    Download full text from publisher

    File URL: http://www.microsimulation.org/IJM/V9_2/IJM_9_2_2016_Timmins.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Robert Tanton & Paul Williamson & Ann Harding, 2014. "Comparing Two Methods of Reweighting a Survey File to Small Area Data," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 76-99.
    2. Rafdzah Zaki & Awang Bulgiba & Roshidi Ismail & Noor Azina Ismail, 2012. "Statistical Methods Used to Test for Agreement of Medical Instruments Measuring Continuous Variables in Method Comparison Studies: A Systematic Review," PLOS ONE, Public Library of Science, vol. 7(5), pages 1-7, May.
    3. Adrian Mander, 2005. "BATPLOT: Stata module to produce Bland-Altman plots accounting for trend," Statistical Software Components S448703, Boston College Department of Economics, revised 23 Mar 2021.
    4. Dimitris Ballas & Graham P Clarke, 2001. "Modelling the Local Impacts of National Social Policies: A Spatial Microsimulation Approach," Environment and Planning C, , vol. 19(4), pages 587-606, August.
    5. Robert Tanton & Yogi Vidyattama, 2010. "Pushing It To The Edge: Extending Generalised Regression As A Spatial Microsimulation Method," International Journal of Microsimulation, International Microsimulation Association, vol. 3(2), pages 23-33.
    6. Robin Lovelace & Mark Birkin & Dimitris Ballas & Eveline van Leeuwen, 2015. "Evaluating the Performance of Iterative Proportional Fitting for Spatial Microsimulation: New Tests for an Established Technique," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-21.
    7. Cathal O'Donoghue & Karyn Morrissey & John Lennon, 2014. "Spatial Microsimulation Modelling: a Review of Applications and Methodological Choices," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 26-75.
    8. Richard Van Noorden & Brendan Maher & Regina Nuzzo, 2014. "The top 100 papers," Nature, Nature, vol. 514(7524), pages 550-553, October.
    9. Rahman, Azizur & Harding, Ann & Tanton, Robert & Liu, Shuangzhe, 2013. "Simulating the characteristics of populations at the small area level: New validation techniques for a spatial microsimulation model in Australia," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 149-165.
    10. P Williamson & M Birkin & P H Rees, 1998. "The Estimation of Population Microdata by Using Data from Small Area Statistics and Samples of Anonymised Records," Environment and Planning A, , vol. 30(5), pages 785-816, May.
    11. Edwards, Kimberley L. & Clarke, Graham P., 2009. "The design and validation of a spatial microsimulation model of obesogenic environments for children in Leeds, UK: SimObesity," Social Science & Medicine, Elsevier, vol. 69(7), pages 1127-1134, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert Tanton, 2018. "Spatial Microsimulation: Developments and Potential Future Directions," International Journal of Microsimulation, International Microsimulation Association, vol. 11(1), pages 143-161.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Loughrey, Jason & O’Donoghue, Cathal & Meredith, David & Murphy, Ger & Shanahan, Ultan & Miller, Corina, 2018. "The Local Impact of Cattle Farming," 166th Seminar, August 30-31, 2018, Galway, West of Ireland 276231, European Association of Agricultural Economists.
    2. Lovelace, Robin & Ballas, Dimitris & Watson, Matt, 2014. "A spatial microsimulation approach for the analysis of commuter patterns: from individual to regional levels," Journal of Transport Geography, Elsevier, vol. 34(C), pages 282-296.
    3. Rahman, Azizur & Harding, Ann & Tanton, Robert & Liu, Shuangzhe, 2013. "Simulating the characteristics of populations at the small area level: New validation techniques for a spatial microsimulation model in Australia," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 149-165.
    4. Robert Tanton & Paul Williamson & Ann Harding, 2014. "Comparing Two Methods of Reweighting a Survey File to Small Area Data," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 76-99.
    5. Robert Tanton, 2014. "A Review of Spatial Microsimulation Methods," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 4-25.
    6. Ann Harding & Robert Tanton, 2014. "Policy and people at the small-area level: using micro-simulation to create synthetic spatial data," Chapters, in: Robert Stimson (ed.), Handbook of Research Methods and Applications in Spatially Integrated Social Science, chapter 25, pages 560-586, Edward Elgar Publishing.
    7. Malcolm Campbell & Dimitris Ballas, 2013. "A spatial microsimulation approach to economic policy analysis in Scotland," Regional Science Policy & Practice, Wiley Blackwell, vol. 5(3), pages 263-288, August.
    8. Cathal O'Donoghue & Karyn Morrissey & John Lennon, 2014. "Spatial Microsimulation Modelling: a Review of Applications and Methodological Choices," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 26-75.
    9. Lucy Semerjian & Kunle Okaiyeto & Mike O. Ojemaye & Temitope Cyrus Ekundayo & Aboi Igwaran & Anthony I. Okoh, 2021. "Global Systematic Mapping of Road Dust Research from 1906 to 2020: Research Gaps and Future Direction," Sustainability, MDPI, vol. 13(20), pages 1-21, October.
    10. Tsukasa Kato, 2021. "Measurement Invariance in the Center for Epidemiologic Studies-Depression (CES-D) Scale among English-Speaking Whites and Asians," IJERPH, MDPI, vol. 18(10), pages 1-10, May.
    11. Martin Johnsen & Oliver Brandt & Sergio Garrido & Francisco C. Pereira, 2020. "Population synthesis for urban resident modeling using deep generative models," Papers 2011.06851, arXiv.org.
    12. Yogi Vidyattama & Robert Tanton & Nicholas Biddle, 2015. "Estimating small-area Indigenous cultural participation from synthetic survey data," Environment and Planning A, , vol. 47(5), pages 1211-1228, May.
    13. Dean Palejev & Mladen Savov, 2021. "On the Convergence of the Benjamini–Hochberg Procedure," Mathematics, MDPI, vol. 9(17), pages 1-19, September.
    14. Kelli Francis-Staite, 2022. "Internal multi-portfolio rebalancing processes: Linking resource allocation models and biproportional matrix techniques to portfolio management," Papers 2201.06183, arXiv.org.
    15. Alberto Vitalini & Simona Ballabio & Flavio Verrecchia, 2024. "Rebuilding a pseudo population register for estimating physical vulnerability at the local level: a case study of spatial microsimulation in Sondrio," RIEDS - Rivista Italiana di Economia, Demografia e Statistica - The Italian Journal of Economic, Demographic and Statistical Studies, SIEDS Societa' Italiana di Economia Demografia e Statistica, vol. 78(1), pages 55-64, January-M.
    16. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    17. Yong Jee KIM & Brigitte WALDORF & Juan SESMERO, 2020. "Relocation, Retreat, and the Rising Sea Level: A Simulation of Aggregate Outcomes in Escambia County, Florida," Region et Developpement, Region et Developpement, LEAD, Universite du Sud - Toulon Var, vol. 51, pages 31-43.
    18. Laurent Linnemer & Michael Visser, 2016. "The Most Cited Articles from the Top-5 Journals (1991-2015)," CESifo Working Paper Series 5999, CESifo.
    19. Robert Tanton, 2018. "Spatial Microsimulation: Developments and Potential Future Directions," International Journal of Microsimulation, International Microsimulation Association, vol. 11(1), pages 143-161.
    20. Luis Miguel Pérez & Raul Oltra-Badenes & Juan Vicente Oltra Gutiérrez & Hermenegildo Gil-Gómez, 2020. "A Bibliometric Diagnosis and Analysis about Smart Cities," Sustainability, MDPI, vol. 12(16), pages 1-43, August.

    More about this item

    Keywords

    Validation; Bland Altman; Spatial Microsimulation;
    All these keywords.

    JEL classification:

    • C63 - Mathematical and Quantitative Methods - - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling - - - Computational Techniques

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ijm:journl:v:9:y:2016:i:2:p:106-122. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jinjing Li (email available below). General contact details of provider: http://www.microsimulation.pub .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.