IDEAS home Printed from https://ideas.repec.org/a/wsi/ijitdm/v14y2015i06ns021962201550025x.html
   My bibliography  Save this article

When is a Decision-Making Method Trustworthy? Criteria for Evaluating Multi-Criteria Decision-Making Methods

Author

Listed:
  • Thomas L. Saaty

    (Graduate School of Business, University of Pittsburgh, Pittsburgh, PA 15260, USA)

  • Daji Ergu

    (Southwest University for Nationalities, Chengdu 610041, China)

Abstract

Decision makers often face complicated decision problems with intangible and conflicting criteria. Numerous multi-criteria decision-making (MCDM) methods have been proposed to handle the measurement of the priorities of conflicting tangible/intangible criteria and in turn use them to choose the best alternative for a decision. However, the presence of many MCDM methods bewilders users. The existence of these methods becomes a decision problem in itself, and decision makers may be uncertain about which one to use. Thus the comparative analysis and evaluation of various MCDM methods has come under scrutiny by both researchers and practitioners in order to discover if there are logical, mathematical, social or practical reasons why one method is better than another. Criteria for their evaluation are the first important issue that needs to be resolved. In this paper, 16 criteria are introduced that may be used to judge and evaluate various MCDM methods. The criteria proposed and some guidelines for their evaluation are given to help readers evaluate these MCDM methods.

Suggested Citation

  • Thomas L. Saaty & Daji Ergu, 2015. "When is a Decision-Making Method Trustworthy? Criteria for Evaluating Multi-Criteria Decision-Making Methods," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 14(06), pages 1171-1187, November.
    • Handle: RePEc:wsi:ijitdm:v:14:y:2015:i:06:n:s021962201550025x
    DOI: 10.1142/S021962201550025X
    as

    Download full text from publisher

    File URL: http://www.worldscientific.com/doi/abs/10.1142/S021962201550025X
    Download Restriction: Access to full text is restricted to subscribers
    File URL: https://libkey.io/10.1142/S021962201550025X?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Peng, Yi & Kou, Gang & Wang, Guoxun & Shi, Yong, 2011. "FAMCDM: A fusion approach of MCDM methods to rank multiclass classification algorithms," Omega, Elsevier, vol. 39(6), pages 677-689, December.
    2. Park, C Whan, 1978. "A Conflict Resolution Choice Model," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 5(2), pages 124-137, Se.
    3. Greco, Salvatore & Matarazzo, Benedetto & Slowinski, Roman, 1999. "Rough approximation of a preference relation by dominance relations," European Journal of Operational Research, Elsevier, vol. 117(1), pages 63-83, August.
    4. Salminen, Pekka & Hokkanen, Joonas & Lahdelma, Risto, 1998. "Comparing multicriteria methods in the context of environmental problems," European Journal of Operational Research, Elsevier, vol. 104(3), pages 485-496, February.
    5. Xu, Xiaozhan, 2001. "The SIR method: A superiority and inferiority ranking method for multiple criteria decision making," European Journal of Operational Research, Elsevier, vol. 131(3), pages 587-602, June.
    6. C. West Churchman & Russell L. Ackoff, 1954. "An Approximate Measure of Value," Operations Research, INFORMS, vol. 2(2), pages 172-187, May.
    7. Opricovic, Serafim & Tzeng, Gwo-Hshiung, 2004. "Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS," European Journal of Operational Research, Elsevier, vol. 156(2), pages 445-455, July.
    8. Thomas L. Saaty, 2005. "The Analytic Hierarchy and Analytic Network Processes for the Measurement of Intangible Criteria and for Decision-Making," International Series in Operations Research & Management Science, in: Multiple Criteria Decision Analysis: State of the Art Surveys, chapter 0, pages 345-405, Springer.
    9. Tam, C.M. & Tong, Thomas K.L. & Chiu, Gerald W.C., 2006. "Comparing non-structural fuzzy decision support system and analytical hierarchy process in decision-making for construction problems," European Journal of Operational Research, Elsevier, vol. 174(2), pages 1317-1324, October.
    10. Bertrand Mareschal & Jean Pierre Brans & Philippe Vincke, 1986. "How to select and how to rank projects: the Prométhée method," ULB Institutional Repository 2013/9307, ULB -- Universite Libre de Bruxelles.
    11. Imran S. Currim & Rakesh K. Sarin, 1984. "A Comparative Evaluation of Multiattribute Consumer Preference Models," Management Science, INFORMS, vol. 30(5), pages 543-561, May.
    12. Gang Kou & Yanqun Lu & Yi Peng & Yong Shi, 2012. "Evaluation Of Classification Algorithms Using Mcdm And Rank Correlation," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 11(01), pages 197-225.
    13. Peter C. Fishburn, 1967. "Letter to the Editor—Additive Utilities with Incomplete Product Sets: Application to Priorities and Assignments," Operations Research, INFORMS, vol. 15(3), pages 537-542, June.
    14. Brans, J. P. & Vincke, Ph. & Mareschal, B., 1986. "How to select and how to rank projects: The method," European Journal of Operational Research, Elsevier, vol. 24(2), pages 228-238, February.
    15. Aczel, Janos & Roberts, Fred S., 1989. "On the possible merging functions," Mathematical Social Sciences, Elsevier, vol. 17(3), pages 205-243, June.
    16. Chithambaranathan, P. & Subramanian, Nachiappan & Gunasekaran, Angappa & Palaniappan, PL.K., 2015. "Service supply chain environmental performance evaluation using grey based hybrid MCDM approach," International Journal of Production Economics, Elsevier, vol. 166(C), pages 163-176.
    17. Hajkowicz, Stefan & Higgins, Andrew, 2008. "A comparison of multiple criteria analysis techniques for water resource management," European Journal of Operational Research, Elsevier, vol. 184(1), pages 255-265, January.
    18. Opricovic, Serafim & Tzeng, Gwo-Hshiung, 2007. "Extended VIKOR method in comparison with outranking methods," European Journal of Operational Research, Elsevier, vol. 178(2), pages 514-529, April.
    19. Harvey M. Wagner, 1995. "Global Sensitivity Analysis," Operations Research, INFORMS, vol. 43(6), pages 948-969, December.
    20. Kou, Gang & Lin, Changsheng, 2014. "A cosine maximization method for the priority vector derivation in AHP," European Journal of Operational Research, Elsevier, vol. 235(1), pages 225-232.
    21. Yoram Wind & Thomas L. Saaty, 1980. "Marketing Applications of the Analytic Hierarchy Process," Management Science, INFORMS, vol. 26(7), pages 641-658, July.
    22. Olson, David L., 2001. "Comparison of three multicriteria methods to predict known outcomes," European Journal of Operational Research, Elsevier, vol. 130(3), pages 576-587, May.
    23. Zanakis, Stelios H. & Solomon, Anthony & Wishart, Nicole & Dublish, Sandipa, 1998. "Multi-attribute decision making: A simulation comparison of select methods," European Journal of Operational Research, Elsevier, vol. 107(3), pages 507-529, June.
    24. Guitouni, Adel & Martel, Jean-Marc, 1998. "Tentative guidelines to help choosing an appropriate MCDA method," European Journal of Operational Research, Elsevier, vol. 109(2), pages 501-521, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eduardo Fernandez & Jorge Navarro & Rafael Olmedo, 2018. "Characterization of the Effectiveness of Several Outranking-Based Multi-Criteria Sorting Methods," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 17(04), pages 1047-1084, July.
    2. Behzadian, Majid & Kazemzadeh, R.B. & Albadvi, A. & Aghdasi, M., 2010. "PROMETHEE: A comprehensive literature review on methodologies and applications," European Journal of Operational Research, Elsevier, vol. 200(1), pages 198-215, January.
    3. Roman Vavrek, 2019. "Evaluation of the Impact of Selected Weighting Methods on the Results of the TOPSIS Technique," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 18(06), pages 1821-1843, November.
    4. Mulliner, Emma & Smallbone, Kieran & Maliene, Vida, 2013. "An assessment of sustainable housing affordability using a multiple criteria decision making method," Omega, Elsevier, vol. 41(2), pages 270-279.
    5. Daniel R. Georgiadis & Thomas A. Mazzuchi & Shahram Sarkani, 2013. "Using multi criteria decision making in analysis of alternatives for selection of enabling technology," Systems Engineering, John Wiley & Sons, vol. 16(3), pages 287-303, September.
    6. Mimica R. Milošević & Dušan M. Milošević & Ana D. Stanojević & Dragan M. Stević & Dušan J. Simjanović, 2021. "Fuzzy and Interval AHP Approaches in Sustainable Management for the Architectural Heritage in Smart Cities," Mathematics, MDPI, vol. 9(4), pages 1-29, February.
    7. Rezaei, Jafar, 2015. "Best-worst multi-criteria decision-making method," Omega, Elsevier, vol. 53(C), pages 49-57.
    8. Mulliner, Emma & Malys, Naglis & Maliene, Vida, 2016. "Comparative analysis of MCDM methods for the assessment of sustainable housing affordability," Omega, Elsevier, vol. 59(PB), pages 146-156.
    9. Misbah Anjum & Vernika Agarwal & P. K. Kapur & Sunil Kumar Khatri, 2020. "Two-phase methodology for prioritization and utility assessment of software vulnerabilities," International Journal of System Assurance Engineering and Management, Springer;The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden, vol. 11(2), pages 289-300, July.
    10. Claudia Margarita Acuña-Soto & Vicente Liern & Blanca Pérez-Gladish, 2020. "Multiple criteria performance evaluation of YouTube mathematical educational videos by IS-TOPSIS," Operational Research, Springer, vol. 20(4), pages 2017-2039, December.
    11. Yi Peng, 2015. "Regional earthquake vulnerability assessment using a combination of MCDM methods," Annals of Operations Research, Springer, vol. 234(1), pages 95-110, November.
    12. Kokaraki, Nikoleta & Hopfe, Christina J. & Robinson, Elaine & Nikolaidou, Elli, 2019. "Testing the reliability of deterministic multi-criteria decision-making methods using building performance simulation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 112(C), pages 991-1007.
    13. Kuang-Hua Hu & Wei Jianguo & Gwo-Hshiung Tzeng, 2017. "Risk Factor Assessment Improvement for China’s Cloud Computing Auditing Using a New Hybrid MADM Model," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 16(03), pages 737-777, May.
    14. Hajkowicz, Stefan & Higgins, Andrew, 2008. "A comparison of multiple criteria analysis techniques for water resource management," European Journal of Operational Research, Elsevier, vol. 184(1), pages 255-265, January.
    15. Zheng Yuan & Baohua Wen & Cheng He & Jin Zhou & Zhonghua Zhou & Feng Xu, 2022. "Application of Multi-Criteria Decision-Making Analysis to Rural Spatial Sustainability Evaluation: A Systematic Review," IJERPH, MDPI, vol. 19(11), pages 1-31, May.
    16. Ioannis Sitaridis & Fotis Kitsios, 2020. "Competitiveness analysis and evaluation of entrepreneurial ecosystems: a multi-criteria approach," Annals of Operations Research, Springer, vol. 294(1), pages 377-399, November.
    17. Ormerod, Richard J. & Ulrich, Werner, 2013. "Operational research and ethics: A literature review," European Journal of Operational Research, Elsevier, vol. 228(2), pages 291-307.
    18. Emre Çalişkan & Erdem Aksakal & Saliha Çetinyokuş & Tahsin Çetinyokuş, 2019. "Hybrid Use of Likert Scale-Based AHP and PROMETHEE Methods for Hazard Analysis and Consequence Modeling (HACM) Software Selection," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 18(05), pages 1689-1715, September.
    19. Willem Brauers, 2013. "Multi-objective seaport planning by MOORA decision making," Annals of Operations Research, Springer, vol. 206(1), pages 39-58, July.
    20. Hassan, Mohammad Nurul & Hawas, Yaser E. & Ahmed, Kamran, 2013. "A multi-dimensional framework for evaluating the transit service performance," Transportation Research Part A: Policy and Practice, Elsevier, vol. 50(C), pages 47-61.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wsi:ijitdm:v:14:y:2015:i:06:n:s021962201550025x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Tai Tone Lim (email available below). General contact details of provider: http://www.worldscinet.com/ijitdm/ijitdm.shtml .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.