IDEAS home Printed from https://ideas.repec.org/a/ijm/journl/v9y2016i2p106-122.html
   My bibliography  Save this article

Validation of Spatial Microsimulation Models: a Proposal to Adopt the Bland-Altman Method

Author

Listed:
  • Kate A Timmins

    () (School of Sport and Exercise Science, University of Lincoln, Lincoln LN6 7TS, United Kingdom)

  • Kimberley L Edwards

    () (Arthritis Research UK Centre for Sport Exercise and Osteoarthritis, University of Nottingham, C Floor, South Block, Queens Medical Centre, Nottingham, NG7 2UH, United Kingdom)

Abstract

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The Bland-Altman plot consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R², SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods.

Suggested Citation

  • Kate A Timmins & Kimberley L Edwards, 2016. "Validation of Spatial Microsimulation Models: a Proposal to Adopt the Bland-Altman Method," International Journal of Microsimulation, International Microsimulation Association, vol. 9(2), pages 106-122.
  • Handle: RePEc:ijm:journl:v:9:y:2016:i:2:p:106-122
    as

    Download full text from publisher

    File URL: http://www.microsimulation.org/IJM/V9_2/IJM_9_2_2016_Timmins.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Dimitris Ballas & Graham P Clarke, 2001. "Modelling the Local Impacts of National Social Policies: A Spatial Microsimulation Approach," Environment and Planning C, , vol. 19(4), pages 587-606, August.
    2. Cathal O'Donoghue & Karyn Morrissey & John Lennon, 2014. "Spatial Microsimulation Modelling: a Review of Applications and Methodological Choices," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 26-75.
    3. Robert Tanton & Paul Williamson & Ann Harding, 2014. "Comparing Two Methods of Reweighting a Survey File to Small Area Data," International Journal of Microsimulation, International Microsimulation Association, vol. 7(1), pages 76-99.
    4. Rahman, Azizur & Harding, Ann & Tanton, Robert & Liu, Shuangzhe, 2013. "Simulating the characteristics of populations at the small area level: New validation techniques for a spatial microsimulation model in Australia," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 149-165.
    5. P Williamson & M Birkin & P H Rees, 1998. "The Estimation of Population Microdata by Using Data from Small Area Statistics and Samples of Anonymised Records," Environment and Planning A, , vol. 30(5), pages 785-816, May.
    6. Edwards, Kimberley L. & Clarke, Graham P., 2009. "The design and validation of a spatial microsimulation model of obesogenic environments for children in Leeds, UK: SimObesity," Social Science & Medicine, Elsevier, vol. 69(7), pages 1127-1134, October.
    7. P Williamson & M Birkin & P H Rees, 1998. "The estimation of population microdata by using data from small area statistics and samples of anonymised records," Environment and Planning A, Pion Ltd, London, vol. 30(5), pages 785-816, May.
    8. Dimitris Ballas & Graham P Clarke, 2001. "Modelling the local impacts of national social policies: a spatial microsimulation approach," Environment and Planning C: Government and Policy, Pion Ltd, London, vol. 19(4), pages 587-606, August.
    9. Robert Tanton & Yogi Vidyattama, 2010. "Pushing It To The Edge: Extending Generalised Regression As A Spatial Microsimulation Method," International Journal of Microsimulation, International Microsimulation Association, vol. 3(2), pages 23-33.
    10. Adrian Mander, 2005. "BATPLOT: Stata module to produce Bland-Altman plots accounting for trend," Statistical Software Components S448703, Boston College Department of Economics, revised 17 Jun 2012.
    11. Robin Lovelace & Mark Birkin & Dimitris Ballas & Eveline van Leeuwen, 2015. "Evaluating the Performance of Iterative Proportional Fitting for Spatial Microsimulation: New Tests for an Established Technique," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-21.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    Validation; Bland Altman; Spatial Microsimulation;

    JEL classification:

    • C63 - Mathematical and Quantitative Methods - - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling - - - Computational Techniques

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ijm:journl:v:9:y:2016:i:2:p:106-122. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jinjing Li). General contact details of provider: http://www.microsimulation.org/ijm/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.