IDEAS home Printed from https://ideas.repec.org/a/spr/aqjoor/v21y2023i1d10.1007_s10288-022-00530-4.html
   My bibliography  Save this article

Multiple criteria sorting models and methods—Part I: survey of the literature

Author

Listed:
  • Khaled Belahcène

    (Heudiasyc, Université de Technologie de Compiègne)

  • Vincent Mousseau

    (MICS, CentraleSupélec, Université Paris-Saclay)

  • Wassila Ouerdane

    (MICS, CentraleSupélec, Université Paris-Saclay)

  • Marc Pirlot

    (MATHRO, Faculté Polytechnique, Université de Mons)

  • Olivier Sobrie

    (MATHRO, Faculté Polytechnique, Université de Mons)

Abstract

Multiple criteria sorting methods assign objects into ordered categories while objects are characterized by a vector of n attributes values. Categories are ordered, and the assignment of the object is monotonic w.r.t. to some underlying order on the attributes scales (criteria). Our goal is to offer a survey of the literature on multiple criteria sorting methods, since the origins, in the 1980s, focusing on the underlying models. Our proposal is organized into two parts. In Part I, we start by recalling two main models, one based on additive value functions (UTADIS) and the other on an outranking relation (Electre Tri). Then we draw a (structured) picture of multiple criteria sorting models and the methods designed for eliciting their parameters or learning them based on assignment examples. In Part II (to appear in a forthcoming issue of this journal), we attempt to provide a theoretical view of the field and position some existing models within it. We then discuss issues related to imperfect or insufficient information.

Suggested Citation

  • Khaled Belahcène & Vincent Mousseau & Wassila Ouerdane & Marc Pirlot & Olivier Sobrie, 2023. "Multiple criteria sorting models and methods—Part I: survey of the literature," 4OR, Springer, vol. 21(1), pages 1-46, March.
  • Handle: RePEc:spr:aqjoor:v:21:y:2023:i:1:d:10.1007_s10288-022-00530-4
    DOI: 10.1007/s10288-022-00530-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10288-022-00530-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10288-022-00530-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Francineide Morais Bezerra & Paulo Melo & João Paulo Costa, 2017. "Reaching Consensus with VICA-ELECTRE TRI: A Case Study," Group Decision and Negotiation, Springer, vol. 26(6), pages 1145-1171, November.
    2. Salvatore Greco & Benedetto Matarazzo & Roman Słowiński, 2016. "Decision Rule Approach," International Series in Operations Research & Management Science, in: Salvatore Greco & Matthias Ehrgott & José Rui Figueira (ed.), Multiple Criteria Decision Analysis, edition 2, chapter 0, pages 497-552, Springer.
    3. Greco, Salvatore & Mousseau, Vincent & Slowinski, Roman, 2010. "Multiple criteria sorting with a set of additive value functions," European Journal of Operational Research, Elsevier, vol. 207(3), pages 1455-1470, December.
    4. Araz, Ceyhun & Ozkarahan, Irem, 2007. "Supplier evaluation and management system for strategic sourcing based on a new multicriteria sorting procedure," International Journal of Production Economics, Elsevier, vol. 106(2), pages 585-606, April.
    5. James S. Dyer, 1990. "Remarks on the Analytic Hierarchy Process," Management Science, INFORMS, vol. 36(3), pages 249-258, March.
    6. Bous, Géraldine & Fortemps, Philippe & Glineur, François & Pirlot, Marc, 2010. "ACUTA: A novel method for eliciting additive value functions on the basis of holistic preference statements," European Journal of Operational Research, Elsevier, vol. 206(2), pages 435-444, October.
    7. Bouyssou, Denis & Marchant, Thierry, 2007. "An axiomatic approach to noncompensatory sorting methods in MCDM, I: The case of two categories," European Journal of Operational Research, Elsevier, vol. 178(1), pages 217-245, April.
    8. Greco, Salvatore & Mousseau, Vincent & Słowiński, Roman, 2014. "Robust ordinal regression for value functions handling interacting criteria," European Journal of Operational Research, Elsevier, vol. 239(3), pages 711-730.
    9. Cinelli, Marco & Kadziński, Miłosz & Miebs, Grzegorz & Gonzalez, Michael & Słowiński, Roman, 2022. "Recommending multiple criteria decision analysis methods with a new taxonomy-based decision support system," European Journal of Operational Research, Elsevier, vol. 302(2), pages 633-651.
    10. Doumpos, M. & Marinakis, Y. & Marinaki, M. & Zopounidis, C., 2009. "An evolutionary approach to construction of outranking models for multicriteria classification: The case of the ELECTRE TRI method," European Journal of Operational Research, Elsevier, vol. 199(2), pages 496-505, December.
    11. Zopounidis, Constantin & Doumpos, Michael, 2002. "Multicriteria classification and sorting methods: A literature review," European Journal of Operational Research, Elsevier, vol. 138(2), pages 229-246, April.
    12. Lahdelma, Risto & Hokkanen, Joonas & Salminen, Pekka, 1998. "SMAA - Stochastic multiobjective acceptability analysis," European Journal of Operational Research, Elsevier, vol. 106(1), pages 137-143, April.
    13. Costa, Ana Sara & Figueira, José Rui & Borbinha, José, 2018. "A multiple criteria nominal classification method based on the concepts of similarity and dissimilarity," European Journal of Operational Research, Elsevier, vol. 271(1), pages 193-209.
    14. Belacel, Nabil, 2000. "Multicriteria assignment method PROAFTN: Methodology and medical application," European Journal of Operational Research, Elsevier, vol. 125(1), pages 175-183, August.
    15. Patrice Perny, 1998. "Multicriteria filtering methods based onconcordance and non-discordance principles," Annals of Operations Research, Springer, vol. 80(0), pages 137-165, January.
    16. Almeida-Dias, J. & Figueira, J.R. & Roy, B., 2012. "A multiple criteria sorting method where each category is characterized by several reference actions: The Electre Tri-nC method," European Journal of Operational Research, Elsevier, vol. 217(3), pages 567-579.
    17. Greco, Salvatore & Mousseau, Vincent & Slowinski, Roman, 2008. "Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions," European Journal of Operational Research, Elsevier, vol. 191(2), pages 416-436, December.
    18. Denis Bouyssou & Thierry Marchant, 2009. "Ordered categories and additive conjoint measurement on connected sets," Post-Print hal-02361900, HAL.
    19. Dias, Luis & Mousseau, Vincent & Figueira, Jose & Climaco, Joao, 2002. "An aggregation/disaggregation approach to obtain robust conclusions with ELECTRE TRI," European Journal of Operational Research, Elsevier, vol. 138(2), pages 332-348, April.
    20. Ishizaka, Alessio & Lolli, Francesco & Balugani, Elia & Cavallieri, Rita & Gamberini, Rita, 2018. "DEASort: Assigning items with data envelopment analysis in ABC classes," International Journal of Production Economics, Elsevier, vol. 199(C), pages 7-15.
    21. Figueira, José Rui & Greco, Salvatore & Roy, Bernard, 2009. "ELECTRE methods with interaction between criteria: An extension of the concordance index," European Journal of Operational Research, Elsevier, vol. 199(2), pages 478-495, December.
    22. Michael Doumpos & Constantin Zopounidis & Emilios C. C Galariotis, 2014. "Inferring robust decision models in multicriteria classification problems: An experimental analysis," Post-Print hal-00961323, HAL.
    23. Bouyssou, Denis & Marchant, Thierry, 2007. "An axiomatic approach to noncompensatory sorting methods in MCDM, II: More than two categories," European Journal of Operational Research, Elsevier, vol. 178(1), pages 246-276, April.
    24. Costa, Ana Sara & Corrente, Salvatore & Greco, Salvatore & Figueira, José Rui & Borbinha, José, 2020. "A robust hierarchical nominal multicriteria classification method based on similarity and dissimilarity," European Journal of Operational Research, Elsevier, vol. 286(3), pages 986-1001.
    25. Pelegrina, Guilherme Dean & Duarte, Leonardo Tomazeli & Grabisch, Michel & Romano, João Marcos Travassos, 2020. "The multilinear model in multicriteria decision making: The case of 2-additive capacities and contributions to parameter identification," European Journal of Operational Research, Elsevier, vol. 282(3), pages 945-956.
    26. Liu, Jiapeng & Kadziński, Miłosz & Liao, Xiuwu & Mao, Xiaoxin & Wang, Yao, 2020. "A preference learning framework for multiple criteria sorting with diverse additive value models and valued assignment examples," European Journal of Operational Research, Elsevier, vol. 286(3), pages 963-985.
    27. Tlili, Ali & Belahcène, Khaled & Khaled, Oumaima & Mousseau, Vincent & Ouerdane, Wassila, 2022. "Learning non-compensatory sorting models using efficient SAT/MaxSAT formulations," European Journal of Operational Research, Elsevier, vol. 298(3), pages 979-1006.
    28. Doumpos, Michael & Zopounidis, Constantin, 2011. "Preference disaggregation and statistical learning for multicriteria decision support: A review," European Journal of Operational Research, Elsevier, vol. 209(3), pages 203-214, March.
    29. Michel Grabisch, 2016. "Set Functions, Games and Capacities in Decision Making," Theory and Decision Library C, Springer, number 978-3-319-30690-2, July.
    30. Kadziński, Miłosz & Ciomek, Krzysztof, 2021. "Active learning strategies for interactive elicitation of assignment examples for threshold-based multiple criteria sorting," European Journal of Operational Research, Elsevier, vol. 293(2), pages 658-680.
    31. Podinovskii, Vladislav V., 1994. "Criteria importance theory," Mathematical Social Sciences, Elsevier, vol. 27(3), pages 237-252, June.
    32. Fernandez, Eduardo & Navarro, Jorge, 2011. "A new approach to multi-criteria sorting based on fuzzy outranking relations: The THESEUS method," European Journal of Operational Research, Elsevier, vol. 213(2), pages 405-413, September.
    33. Mousseau, V. & Figueira, J. & Naux, J. -Ph., 2001. "Using assignment examples to infer weights for ELECTRE TRI method: Some experimental results," European Journal of Operational Research, Elsevier, vol. 130(2), pages 263-275, April.
    34. De Smet, Yves & Nemery, Philippe & Selvaraj, Ramkumar, 2012. "An exact algorithm for the multicriteria ordered clustering problem," Omega, Elsevier, vol. 40(6), pages 861-869.
    35. Almeida-Dias, J. & Figueira, J.R. & Roy, B., 2010. "Electre Tri-C: A multiple criteria sorting method based on characteristic reference actions," European Journal of Operational Research, Elsevier, vol. 204(3), pages 565-580, August.
    36. Selin Özpeynirci & Özgür Özpeynirci & Vincent Mousseau, 2018. "An interactive algorithm for multiple criteria constrained sorting problem," Annals of Operations Research, Springer, vol. 267(1), pages 447-466, August.
    37. Alessio Ishizaka & Maynard Gordon, 2017. "MACBETHSort: a multiple criteria decision aid procedure for sorting strategic products," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(1), pages 53-61, January.
    38. Arcidiacono, Sally Giuseppe & Corrente, Salvatore & Greco, Salvatore, 2021. "Robust stochastic sorting with interacting criteria hierarchically structured," European Journal of Operational Research, Elsevier, vol. 292(2), pages 735-754.
    39. Fernández, Eduardo & Figueira, José Rui & Navarro, Jorge, 2019. "An interval extension of the outranking approach and its application to multiple-criteria ordinal classification," Omega, Elsevier, vol. 84(C), pages 189-198.
    40. Leger, Julien & Martel, Jean-Marc, 2002. "A multicriteria assignment procedure for a nominal sorting problematic," European Journal of Operational Research, Elsevier, vol. 138(2), pages 349-364, April.
    41. Liu, Jiapeng & Liao, Xiuwu & Kadziński, Miłosz & Słowiński, Roman, 2019. "Preference disaggregation within the regularization framework for sorting problems with multiple potentially non-monotonic criteria," European Journal of Operational Research, Elsevier, vol. 276(3), pages 1071-1089.
    42. Opricovic, Serafim & Tzeng, Gwo-Hshiung, 2004. "Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS," European Journal of Operational Research, Elsevier, vol. 156(2), pages 445-455, July.
    43. Stefan Eppe & Julien Roland & Yves De Smet, 2014. "On the use of valued action profiles for relational multi-criteria clustering," International Journal of Multicriteria Decision Making, Inderscience Enterprises Ltd, vol. 4(3), pages 201-233.
    44. repec:dau:papers:123456789/1482 is not listed on IDEAS
    45. Fernández, Eduardo & Navarro, Jorge & Solares, Efrain, 2022. "A hierarchical interval outranking approach with interacting criteria," European Journal of Operational Research, Elsevier, vol. 298(1), pages 293-307.
    46. Denis Bouyssou & Thierry Marchant & Marc Pirlot & Alexis Tsoukiàs & Philippe Vincke, 2006. "Evaluation and Decision Models with Multiple Criteria," International Series in Operations Research and Management Science, Springer, number 978-0-387-31099-2, December.
    47. Doumpos, Michael & Zopounidis, Constantin & Galariotis, Emilios, 2014. "Inferring robust decision models in multicriteria classification problems: An experimental analysis," European Journal of Operational Research, Elsevier, vol. 236(2), pages 601-611.
    48. Kadziński, Miłosz & Tervonen, Tommi & Rui Figueira, José, 2015. "Robust multi-criteria sorting with the outranking preference model and characteristic profiles," Omega, Elsevier, vol. 55(C), pages 126-140.
    49. Liu, Jiapeng & Liao, Xiuwu & Zhao, Wenhong & Yang, Na, 2016. "A classification approach based on the outranking model for multiple criteria ABC analysis," Omega, Elsevier, vol. 61(C), pages 19-34.
    50. Tervonen, Tommi & Figueira, José Rui & Lahdelma, Risto & Dias, Juscelino Almeida & Salminen, Pekka, 2009. "A stochastic method for robustness analysis in sorting problems," European Journal of Operational Research, Elsevier, vol. 192(1), pages 236-242, January.
    51. De Smet, Yves & Montano Guzman, Linett, 2004. "Towards multicriteria clustering: An extension of the k-means algorithm," European Journal of Operational Research, Elsevier, vol. 158(2), pages 390-398, October.
    52. Karasakal, Esra & Aker, Pınar, 2017. "A multicriteria sorting approach based on data envelopment analysis for R&D project selection problem," Omega, Elsevier, vol. 73(C), pages 79-92.
    53. James S. Dyer, 1990. "A Clarification of "Remarks on the Analytic Hierarchy Process"," Management Science, INFORMS, vol. 36(3), pages 274-275, March.
    54. Angilella, Silvia & Greco, Salvatore & Matarazzo, Benedetto, 2010. "Non-additive robust ordinal regression: A multiple criteria decision model based on the Choquet integral," European Journal of Operational Research, Elsevier, vol. 201(1), pages 277-288, February.
    55. Corrente, Salvatore & Greco, Salvatore & Słowiński, Roman, 2016. "Multiple Criteria Hierarchy Process for ELECTRE Tri methods," European Journal of Operational Research, Elsevier, vol. 252(1), pages 191-203.
    56. Kadziński, Miłosz & Ciomek, Krzysztof & Słowiński, Roman, 2015. "Modeling assignment-based pairwise comparisons within integrated framework for value-driven multiple criteria sorting," European Journal of Operational Research, Elsevier, vol. 241(3), pages 830-841.
    57. R. Pelissari & M. C. Oliveira & S. Ben Amor & A. Kandakoglu & A. L. Helleno, 2020. "SMAA methods and their applications: a literature review and future research directions," Annals of Operations Research, Springer, vol. 293(2), pages 433-493, October.
    58. Jacquet-Lagreze, Eric & Siskos, Yannis, 2001. "Preference disaggregation: 20 years of MCDA experience," European Journal of Operational Research, Elsevier, vol. 130(2), pages 233-245, April.
    59. Greco, Salvatore & Matarazzo, Benedetto & Slowinski, Roman, 2002. "Rough sets methodology for sorting problems in presence of multiple attributes and criteria," European Journal of Operational Research, Elsevier, vol. 138(2), pages 247-259, April.
    60. Fernández, Eduardo & Figueira, José Rui & Navarro, Jorge, 2020. "Interval-based extensions of two outranking methods for multi-criteria ordinal classification," Omega, Elsevier, vol. 95(C).
    61. Siamak Kheybari & S. Ali Naji & Fariba Mahdi Rezaie & Reza Salehpour, 2019. "ABC classification according to Pareto’s principle: a hybrid methodology," OPSEARCH, Springer;Operational Research Society of India, vol. 56(2), pages 539-562, June.
    62. Murat Köksalan & Vincent Mousseau & Selin Özpeynirci, 2017. "Multi-Criteria Sorting with Category Size Restrictions," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 16(01), pages 5-23, January.
    63. Miłosz Kadziński & Magdalena Martyn, 2021. "Enriched preference modeling and robustness analysis for the ELECTRE Tri-B method," Annals of Operations Research, Springer, vol. 306(1), pages 173-207, November.
    64. Sébastien Damart & Luis Dias & Vincent Mousseau, 2007. "Supporting groups in sorting decisions: methodology and use of a multi-criteria aggregation-disaggregation DSS," Post-Print halshs-00170234, HAL.
    65. Roy, Bernard & Slowinski, Roman, 2008. "Handling effects of reinforced preference and counter-veto in credibility of outranking," European Journal of Operational Research, Elsevier, vol. 188(1), pages 185-190, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cinelli, Marco & Kadziński, Miłosz & Miebs, Grzegorz & Gonzalez, Michael & Słowiński, Roman, 2022. "Recommending multiple criteria decision analysis methods with a new taxonomy-based decision support system," European Journal of Operational Research, Elsevier, vol. 302(2), pages 633-651.
    2. Govindan, Kannan & Jepsen, Martin Brandt, 2016. "ELECTRE: A comprehensive literature review on methodologies and applications," European Journal of Operational Research, Elsevier, vol. 250(1), pages 1-29.
    3. Fernández, Eduardo & Figueira, José Rui & Navarro, Jorge & Solares, Efrain, 2023. "A generalized approach to ordinal classification based on the comparison of actions with either limiting or characteristic profiles," European Journal of Operational Research, Elsevier, vol. 305(3), pages 1309-1322.
    4. Ishizaka, Alessio & Lokman, Banu & Tasiou, Menelaos, 2021. "A Stochastic Multi-criteria divisive hierarchical clustering algorithm," Omega, Elsevier, vol. 103(C).
    5. Zhen Zhang & Zhuolin Li, 2023. "Consensus-based TOPSIS-Sort-B for multi-criteria sorting in the context of group decision-making," Annals of Operations Research, Springer, vol. 325(2), pages 911-938, June.
    6. Wang, Liang & Zhang, Zi-Xin & Ishizaka, Alessio & Wang, Ying-Ming & Martínez, Luis, 2023. "TODIMSort: A TODIM based method for sorting problems," Omega, Elsevier, vol. 115(C).
    7. Eduardo Fernández & José Rui Figueira & Jorge Navarro, 2023. "A theoretical look at ordinal classification methods based on comparing actions with limiting boundaries between adjacent classes," Annals of Operations Research, Springer, vol. 325(2), pages 819-843, June.
    8. Tlili, Ali & Belahcène, Khaled & Khaled, Oumaima & Mousseau, Vincent & Ouerdane, Wassila, 2022. "Learning non-compensatory sorting models using efficient SAT/MaxSAT formulations," European Journal of Operational Research, Elsevier, vol. 298(3), pages 979-1006.
    9. Arcidiacono, Sally Giuseppe & Corrente, Salvatore & Greco, Salvatore, 2021. "Robust stochastic sorting with interacting criteria hierarchically structured," European Journal of Operational Research, Elsevier, vol. 292(2), pages 735-754.
    10. Kadziński, Miłosz & Ghaderi, Mohammad & Dąbrowski, Maciej, 2020. "Contingent preference disaggregation model for multiple criteria sorting problem," European Journal of Operational Research, Elsevier, vol. 281(2), pages 369-387.
    11. Murat Köksalan & Vincent Mousseau & Selin Özpeynirci, 2017. "Multi-Criteria Sorting with Category Size Restrictions," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 16(01), pages 5-23, January.
    12. Ru, Zice & Liu, Jiapeng & Kadziński, Miłosz & Liao, Xiuwu, 2023. "Probabilistic ordinal regression methods for multiple criteria sorting admitting certain and uncertain preferences," European Journal of Operational Research, Elsevier, vol. 311(2), pages 596-616.
    13. Jindong Qin & Yingying Liang & Luis Martinez & Alessio Ishizaka & Witold Pedrycz, 2023. "ORESTE-SORT: a novel multiple criteria sorting method for sorting port group competitiveness," Annals of Operations Research, Springer, vol. 325(2), pages 875-909, June.
    14. Sarah Ben Amor & Fateh Belaid & Ramzi Benkraiem & Boumediene Ramdani & Khaled Guesmi, 2023. "Multi-criteria classification, sorting, and clustering: a bibliometric review and research agenda," Annals of Operations Research, Springer, vol. 325(2), pages 771-793, June.
    15. Díaz, Raymundo & Fernández, Eduardo & Figueira, José-Rui & Navarro, Jorge & Solares, Efrain, 2023. "A new hierarchical multiple criteria ordered clustering approach as a complementary tool for sorting and ranking problems," Omega, Elsevier, vol. 117(C).
    16. Costa, Ana Sara & Figueira, José Rui & Borbinha, José, 2018. "A multiple criteria nominal classification method based on the concepts of similarity and dissimilarity," European Journal of Operational Research, Elsevier, vol. 271(1), pages 193-209.
    17. Liu, Jiapeng & Liao, Xiuwu & Kadziński, Miłosz & Słowiński, Roman, 2019. "Preference disaggregation within the regularization framework for sorting problems with multiple potentially non-monotonic criteria," European Journal of Operational Research, Elsevier, vol. 276(3), pages 1071-1089.
    18. Vincent Mousseau & Özgür Özpeynirci & Selin Özpeynirci, 2018. "Inverse multiple criteria sorting problem," Annals of Operations Research, Springer, vol. 267(1), pages 379-412, August.
    19. Kadziński, Miłosz & Ciomek, Krzysztof, 2021. "Active learning strategies for interactive elicitation of assignment examples for threshold-based multiple criteria sorting," European Journal of Operational Research, Elsevier, vol. 293(2), pages 658-680.
    20. Eduardo Fernandez & Jorge Navarro & Rafael Olmedo, 2018. "Characterization of the Effectiveness of Several Outranking-Based Multi-Criteria Sorting Methods," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 17(04), pages 1047-1084, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:aqjoor:v:21:y:2023:i:1:d:10.1007_s10288-022-00530-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.