IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v44y1998i6p820-830.html
   My bibliography  Save this article

Active Nonlinear Tests (ANTs) of Complex Simulation Models

Author

Listed:
  • John H. Miller

    (Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213)

Abstract

Simulation models are becoming increasingly common in the analysis of critical scientific, policy, and management issues. Such models provide a way to analyze complex systems characterized by both large parameter spaces and nonlinear interactions. Unfortunately, these same characteristics make understanding such models using traditional testing techniques extremely difficult. Here we show how a model's structure and robustness can be validated via a simple, automatic, nonlinear search algorithm designed to actively "break" the model's implications. Using the active nonlinear tests (ANTs) developed here, one can easily probe for key weaknesses in a simulation's structure, and thereby begin to improve and refine its design. We demonstrate ANTs by testing a well-known model of global dynamics (World3), and show how this technique can be used to uncover small, but powerful, nonlinear effects that may highlight vulnerabilities in the original model.

Suggested Citation

  • John H. Miller, 1998. "Active Nonlinear Tests (ANTs) of Complex Simulation Models," Management Science, INFORMS, vol. 44(6), pages 820-830, June.
    • Handle: RePEc:inm:ormnsc:v:44:y:1998:i:6:p:820-830
    DOI: 10.1287/mnsc.44.6.820
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.44.6.820
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.44.6.820?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Holland, John H & Miller, John H, 1991. "Artificial Adaptive Agents in Economic Theory," American Economic Review, American Economic Association, vol. 81(2), pages 365-371, May.
    2. Steve Bankes, 1993. "Exploratory Modeling for Policy Analysis," Operations Research, INFORMS, vol. 41(3), pages 435-449, June.
    3. Nordhaus, William D, 1973. "World Dynamics: Measurement Without Data," Economic Journal, Royal Economic Society, vol. 83(332), pages 1156-1183, December.
    4. William D. Nordhaus, 1992. "Lethal Model 2: The Limits to Growth Revisited," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 23(2), pages 1-60.
    5. David C. Cox & Paul Baybutt, 1981. "Methods for Uncertainty Analysis: A Comparative Survey," Risk Analysis, John Wiley & Sons, vol. 1(4), pages 251-258, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Haruvy, Ernan & Roth, Alvin E. & Unver, M. Utku, 2006. "The dynamics of law clerk matching: An experimental and computational investigation of proposals for reform of the market," Journal of Economic Dynamics and Control, Elsevier, vol. 30(3), pages 457-486, March.
    2. Carrella, Ernesto & Saul, Steven & Marshall, Kristin & Burgess, Matthew G. & Cabral, Reniel B. & Bailey, Richard M. & Dorsett, Chris & Drexler, Michael & Madsen, Jens Koed & Merkl, Andreas, 2020. "Simple Adaptive Rules Describe Fishing Behaviour Better than Perfect Rationality in the US West Coast Groundfish Fishery," Ecological Economics, Elsevier, vol. 169(C).
    3. Gönenç Yücel & Yaman Barlas, 2011. "Automated parameter specification in dynamic feedback models based on behavior pattern features," System Dynamics Review, System Dynamics Society, vol. 27(2), pages 195-215, April.
    4. William Tracy, 2014. "Paradox Lost: The Evolution of Strategies in Selten’s Chain Store Game," Computational Economics, Springer;Society for Computational Economics, vol. 43(1), pages 83-103, January.
    5. Oliva, Rogelio, 2003. "Model calibration as a testing strategy for system dynamics models," European Journal of Operational Research, Elsevier, vol. 151(3), pages 552-568, December.
    6. Komeil Mahjori Karmozdi & Mohammad Reza Kohansal & Mohammad Ghorbani, 2020. "Sustainable economic rural development system pattern in Ghaemshahr: an application of the developed TOP-MARD core model," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 22(6), pages 5793-5817, August.
    7. Ma, Tieju & Nakamori, Yoshiteru, 2005. "Agent-based modeling on technological innovation as an evolutionary process," European Journal of Operational Research, Elsevier, vol. 166(3), pages 741-755, November.
    8. Margolis, David N. & Navarro, Lucas & Robalino, David A., 2012. "Unemployment Insurance, Job Search and Informal Employment," IZA Discussion Papers 6660, Institute of Labor Economics (IZA).
    9. Kwakkel, Jan H. & Pruyt, Erik, 2013. "Exploratory Modeling and Analysis, an approach for model-based foresight under deep uncertainty," Technological Forecasting and Social Change, Elsevier, vol. 80(3), pages 419-431.
    10. Eva Labro, 2015. "Using simulation methods in accounting research," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 26(2), pages 99-104, August.
    11. Benjamin L. Turner & Hector M. Menendez & Roger Gates & Luis O. Tedeschi & Alberto S. Atzori, 2016. "System Dynamics Modeling for Agricultural and Natural Resource Management Issues: Review of Some Past Cases and Forecasting Future Roles," Resources, MDPI, vol. 5(4), pages 1-24, November.
    12. Keiki Takadama & Takao Terano & Katsunori Shimohara, 2003. "Interpretation by Implementation for Understanding a Multiagent Organization," Computational and Mathematical Organization Theory, Springer, vol. 9(1), pages 19-35, May.
    13. Sibel Eker & Jill Slinger & Els Daalen & Gönenç Yücel, 2014. "Sensitivity analysis of graphical functions," System Dynamics Review, System Dynamics Society, vol. 30(3), pages 186-205, July.
    14. Prenkert, Frans & Følgesvold, Atle, 2014. "Relationship strength and network form: An agent-based simulation of interaction in a business network," Australasian marketing journal, Elsevier, vol. 22(1), pages 15-27.
    15. Scott F. Turner & Richard A. Bettis & Richard M. Burton, 2002. "Exploring Depth Versus Breadth in Knowledge Management Strategies," Computational and Mathematical Organization Theory, Springer, vol. 8(1), pages 49-73, May.
    16. Jan H. Kwakkel & Erik Pruyt, 2015. "Using System Dynamics for Grand Challenges: The ESDMA Approach," Systems Research and Behavioral Science, Wiley Blackwell, vol. 32(3), pages 358-375, May.
    17. Chaturvedi, Alok & Mehta, Shailendra & Dolk, Daniel & Ayer, Rick, 2005. "Agent-based simulation for computational experimentation: Developing an artificial labor market," European Journal of Operational Research, Elsevier, vol. 166(3), pages 694-716, November.
    18. Robalino, David A. & Voetberg, Albertus & Picazo, Oscar, 2002. "The macroeconomic impacts of AIDS in Kenya estimating optimal reduction targets for the HIV/AIDS incidence rate," Journal of Policy Modeling, Elsevier, vol. 24(2), pages 195-218, May.
    19. Busby, J.S., 2019. "The co-evolution of competition and parasitism in the resource-based view: A risk model of product counterfeiting," European Journal of Operational Research, Elsevier, vol. 276(1), pages 300-313.
    20. Azad M. Madni & Michael Sievers, 2018. "Model‐based systems engineering: Motivation, current status, and research opportunities," Systems Engineering, John Wiley & Sons, vol. 21(3), pages 172-190, May.
    21. Harald de Bruijn & Andreas Größler & Nuno Videira, 2020. "Antifragility as a design criterion for modelling dynamic systems," Systems Research and Behavioral Science, Wiley Blackwell, vol. 37(1), pages 23-37, January.
    22. Rand, William & Rust, Roland T., 2011. "Agent-based modeling in marketing: Guidelines for rigor," International Journal of Research in Marketing, Elsevier, vol. 28(3), pages 181-193.
    23. Erling Moxnes & Sergey Naumov, 2024. "Analysis of dynamic models by optimization," System Dynamics Review, System Dynamics Society, vol. 40(1), January.
    24. Peter Gerbrands & Brigitte Unger & Joras Ferwerda, 2022. "Bilateral responsive regulation and international tax competition: An agent‐based simulation," Regulation & Governance, John Wiley & Sons, vol. 16(3), pages 760-780, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John H. Miller, 1996. "Active Nonlinear Tests (ANTs) of Complex Simulation Models," Working Papers 96-03-011, Santa Fe Institute.
    2. Carson, Richard T. & McCubbin, Donald R., 1998. "Policy Paper 32: Emissions and Development in the United States: International Implications," Institute on Global Conflict and Cooperation, Working Paper Series qt02t32857, Institute on Global Conflict and Cooperation, University of California.
    3. King, Carey W., 2020. "An integrated biophysical and economic modeling framework for long-term sustainability analysis: the HARMONEY model," Ecological Economics, Elsevier, vol. 169(C).
    4. Vicknair, David & Tansey, Michael & O'Brien, Thomas E., 2022. "Measuring fossil fuel reserves: A simulation and review of the U.S. Securities and Exchange Commission approach," Resources Policy, Elsevier, vol. 79(C).
    5. Robert S. Pindyck, 2017. "The Use and Misuse of Models for Climate Policy," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 11(1), pages 100-114.
    6. Cameron Hepburn & Alex Bowen, 2013. "Prosperity with growth: economic growth, climate change and environmental limits," Chapters, in: Roger Fouquet (ed.), Handbook on Energy and Climate Change, chapter 29, pages 617-638, Edward Elgar Publishing.
    7. Jeroen C.J.M. van den Bergh & Peter Nijkamp, 1997. "Optimal Growth, Coordination and Sustainability in the Spatial Economy," Tinbergen Institute Discussion Papers 97-104/3, Tinbergen Institute.
    8. van den Bergh, Jeroen C. J. M., 2004. "Optimal climate policy is a utopia: from quantitative to qualitative cost-benefit analysis," Ecological Economics, Elsevier, vol. 48(4), pages 385-393, April.
    9. Pogany, Peter, 2013. "Thermodynamic Isolation and the New World Order," MPRA Paper 49924, University Library of Munich, Germany.
    10. Smulders, Sjak & Gradus, Raymond, 1996. "Pollution abatement and long-term growth," European Journal of Political Economy, Elsevier, vol. 12(3), pages 505-532, November.
    11. Ondřej Šíma, 2020. "Reálná ekonomika jako zdroj nerovnováhy obchodní bilance - základní přístup [Real Economy as a Source of Trade Balance Disequilibrium - Basic Approach]," Politická ekonomie, Prague University of Economics and Business, vol. 2020(3), pages 322-347.
    12. Guido A Veldhuis & Nico M de Reus & Bas MJ Keijser, 2020. "Concept development for comprehensive operations support with modeling and simulation," The Journal of Defense Modeling and Simulation, , vol. 17(1), pages 99-116, January.
    13. Joshua M. Epstein, 2007. "Agent-Based Computational Models and Generative Social Science," Introductory Chapters, in: Generative Social Science Studies in Agent-Based Computational Modeling, Princeton University Press.
    14. Marco Casari, 2002. "Can genetic algorithms explain experimental anomalies? An application to common property resources," UFAE and IAE Working Papers 542.02, Unitat de Fonaments de l'Anàlisi Econòmica (UAB) and Institut d'Anàlisi Econòmica (CSIC).
    15. Benoît Desmarchelier & Faridah Djellal & Faïz Gallouj, 2018. "Public Service Innovation Networks (PSINs): Collaborating for Innovation and Value Creation," Working Papers halshs-01934275, HAL.
    16. Pezzey, John C.V. & Burke, Paul J., 2014. "Towards a more inclusive and precautionary indicator of global sustainability," Ecological Economics, Elsevier, vol. 106(C), pages 141-154.
    17. Casari, Marco, 2008. "Markets in equilibrium with firms out of equilibrium: A simulation study," Journal of Economic Behavior & Organization, Elsevier, vol. 65(2), pages 261-276, February.
    18. R駩s Chenavaz & Octavio Escobar, 2015. "Population distribution, effective area and economic growth," Applied Economics, Taylor & Francis Journals, vol. 47(53), pages 5776-5790, November.
    19. Ian McCarthy, 2008. "Simulating Sequential Search Models with Genetic Algorithms: Analysis of Price Ceilings, Taxes, Advertising and Welfare," CAEPR Working Papers 2008-010, Center for Applied Economics and Policy Research, Department of Economics, Indiana University Bloomington.
    20. Alicia Vidler & Toby Walsh, 2025. "Shifting Power: Leveraging LLMs to Simulate Human Aversion in ABMs of Bilateral Financial Exchanges, A bond market study," Papers 2503.00320, arXiv.org, revised Mar 2025.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:44:y:1998:i:6:p:820-830. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.