IDEAS home Printed from https://ideas.repec.org/a/spr/joheur/v31y2025i1d10.1007_s10732-025-09549-2.html
   My bibliography  Save this article

Measuring the effectiveness and efficiency of simulation optimization metaheuristic algorithms

Author

Listed:
  • Benjamin G. Thengvall

    (OptTek Systems, Inc.)

  • Shane N. Hall

    (Montana State University)

  • Michael P. Deskevich

    (OptTek Systems, Inc.)

Abstract

Metaheuristic algorithms have proven capable as general-purpose algorithms for solving simulation optimization problems. Researchers and practitioners often compare different metaheuristic algorithms by examining one or more measures that are derived through empirical analysis. This paper presents a single measure that can be used to empirically compare different metaheuristic algorithms for optimization problems. This measure incorporates both the effectiveness and efficiency of the metaheuristic algorithm, which is especially important in simulation optimization applications because the number of simulation runs available to the analyst (i.e., the run budget) can vary significantly with each simulation study. Therefore, the trade-off between the effectiveness and efficiency of a metaheuristic algorithm must be examined. This single measure is especially useful for multi-objective optimization problems; however, determining this measure is non-trivial for two or more objective functions. Additional details for calculating this measure for multi-objective optimization problems are provided as well as a procedure for comparing two or more metaheuristic algorithms. Finally, computational results are presented and analyzed to compare the performance of metaheuristic algorithms using knapsack problems, pure binary integer programs, traveling salesman problems, and the average results obtained across a diverse set of optimization problems that include simulation and multi-objective optimization problems.

Suggested Citation

  • Benjamin G. Thengvall & Shane N. Hall & Michael P. Deskevich, 2025. "Measuring the effectiveness and efficiency of simulation optimization metaheuristic algorithms," Journal of Heuristics, Springer, vol. 31(1), pages 1-21, March.
    • Handle: RePEc:spr:joheur:v:31:y:2025:i:1:d:10.1007_s10732-025-09549-2
    DOI: 10.1007/s10732-025-09549-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10732-025-09549-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10732-025-09549-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. J. N. Hooker, 1994. "Needed: An Empirical Science of Algorithms," Operations Research, INFORMS, vol. 42(2), pages 201-212, April.
    2. Marie Coffin & Matthew J. Saltzman, 2000. "Statistical Analysis of Computational Tests of Algorithms and Heuristics," INFORMS Journal on Computing, INFORMS, vol. 12(1), pages 24-44, February.
    3. Brian M Wade, 2019. "A multi-objective optimization of ballistic and cruise missile fire plans based on damage calculations from missile impacts on an airfield defended by an air defense artillery network," The Journal of Defense Modeling and Simulation, , vol. 16(2), pages 103-117, April.
    4. Sheldon H. Jacobson & Shane N. Hall & Laura A. McLay & Jeffrey E. Orosz, 2005. "Performance Analysis of Cyclical Simulated Annealing Algorithms," Methodology and Computing in Applied Probability, Springer, vol. 7(2), pages 183-201, June.
    5. Harvey J. Greenberg, 1990. "Computational Testing: Why, How and How Much," INFORMS Journal on Computing, INFORMS, vol. 2(1), pages 94-97, February.
    6. Audet, Charles & Bigeon, Jean & Cartier, Dominique & Le Digabel, Sébastien & Salomon, Ludovic, 2021. "Performance indicators in multiobjective optimization," European Journal of Operational Research, Elsevier, vol. 292(2), pages 397-422.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicholas G. Hall & Marc E. Posner, 2001. "Generating Experimental Data for Computational Testing with Machine Scheduling Applications," Operations Research, INFORMS, vol. 49(6), pages 854-865, December.
    2. Mesquita-Cunha, Mariana & Figueira, José Rui & Barbosa-Póvoa, Ana Paula, 2023. "New ϵ−constraint methods for multi-objective integer linear programming: A Pareto front representation approach," European Journal of Operational Research, Elsevier, vol. 306(1), pages 286-307.
    3. Mastrolilli, Monaldo & Bianchi, Leonora, 2005. "Core instances for testing: A case study," European Journal of Operational Research, Elsevier, vol. 166(1), pages 51-62, October.
    4. Felipe Campelo & Elizabeth F. Wanner, 2020. "Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances," Journal of Heuristics, Springer, vol. 26(6), pages 851-883, December.
    5. Belarmino Adenso-Díaz & Manuel Laguna, 2006. "Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search," Operations Research, INFORMS, vol. 54(1), pages 99-114, February.
    6. Marie Coffin & Matthew J. Saltzman, 2000. "Statistical Analysis of Computational Tests of Algorithms and Heuristics," INFORMS Journal on Computing, INFORMS, vol. 12(1), pages 24-44, February.
    7. de Freitas, Juliana Campos & Cantane, Daniela Renata & Rocha, Humberto & Dias, Joana, 2024. "A multiobjective beam angle optimization framework for intensity-modulated radiation therapy," European Journal of Operational Research, Elsevier, vol. 318(1), pages 286-296.
    8. Zandieh, Fatemeh & Ghannadpour, Seyed Farid, 2023. "A comprehensive risk assessment view on interval type-2 fuzzy controller for a time-dependent HazMat routing problem," European Journal of Operational Research, Elsevier, vol. 305(2), pages 685-707.
    9. Dalila B. M. M. Fontes & S. Mahdi Homayouni, 2023. "A bi-objective multi-population biased random key genetic algorithm for joint scheduling quay cranes and speed adjustable vehicles in container terminals," Flexible Services and Manufacturing Journal, Springer, vol. 35(1), pages 241-268, March.
    10. Gaggero, Mauro & Paolucci, Massimo & Ronco, Roberto, 2023. "Exact and heuristic solution approaches for energy-efficient identical parallel machine scheduling with time-of-use costs," European Journal of Operational Research, Elsevier, vol. 311(3), pages 845-866.
    11. Bibiana Granda & Begoña Vitoriano & José Rui Figueira, 2025. "A mathematical programming approach for a wildfire suppression problem," Operational Research, Springer, vol. 25(1), pages 1-27, March.
    12. Ducardo L. Molina & Juan Ricardo Vidal Medina & Alexis Sagastume Gutiérrez & Juan J. Cabello Eras & Jesús A. Lopez & Simón Hincapie & Enrique C. Quispe, 2023. "Multiobjective Optimization of the Energy Efficiency and the Steam Flow in a Bagasse Boiler," Sustainability, MDPI, vol. 15(14), pages 1-17, July.
    13. Hamed Khosravi & Taofeeq Olajire & Ahmed Shoyeb Raihan & Imtiaz Ahmed, 2024. "A data driven sequential learning framework to accelerate and optimize multi-objective manufacturing decisions," Journal of Intelligent Manufacturing, Springer, vol. 35(8), pages 4087-4112, December.
    14. Felipe Campelo & Fernanda Takahashi, 2019. "Sample size estimation for power and accuracy in the experimental comparison of algorithms," Journal of Heuristics, Springer, vol. 25(2), pages 305-338, April.
    15. Jean Bigeon & Sébastien Le Digabel & Ludovic Salomon, 2021. "DMulti-MADS: mesh adaptive direct multisearch for bound-constrained blackbox multiobjective optimization," Computational Optimization and Applications, Springer, vol. 79(2), pages 301-338, June.
    16. Ahuja, Ravindra K., 1956- & Orlin, James B., 1953-, 1992. "Use of representative operation counts in computational testings of algorithms," Working papers 3459-92., Massachusetts Institute of Technology (MIT), Sloan School of Management.
    17. Francisco Jonatas Siqueira Coelho & Allan Rivalles Souza Feitosa & André Luís Michels Alcântara & Kaifeng Li & Ronaldo Ferreira Lima & Victor Rios Silva & Abel Guilhermino da Silva-Filho, 2023. "HyMOTree: Automatic Hyperparameters Tuning for Non-Technical Loss Detection Based on Multi-Objective and Tree-Based Algorithms," Energies, MDPI, vol. 16(13), pages 1-22, June.
    18. Pablo Moscato & Luke Mathieson & Mohammad Nazmul Haque, 2021. "Augmented intuition: a bridge between theory and practice," Journal of Heuristics, Springer, vol. 27(4), pages 497-547, August.
    19. Marins, Fernando A. S. & Senne, Edson L. F. & Darby-Dowman, Ken & Machado, Arlene F. & Perin, Clovis, 1997. "Algorithms for network piecewise-linear programs: A comparative study," European Journal of Operational Research, Elsevier, vol. 97(1), pages 183-199, February.
    20. Mausser, Helmut E. & Laguna, Manuel, 1999. "A heuristic to minimax absolute regret for linear programs with interval objective function coefficients," European Journal of Operational Research, Elsevier, vol. 117(1), pages 157-174, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:joheur:v:31:y:2025:i:1:d:10.1007_s10732-025-09549-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.