IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v185y2022i2p519-542.html
   My bibliography  Save this article

Pairwise comparisons as a scale development tool for composite measures

Author

Listed:
  • Ginevra Floridi
  • Benjamin E. Lauderdale

Abstract

Composite scales are widely used for measuring aggregate social science concepts. These often consist of linear indices obtained as the weighted sum of a set of relevant indicators. However, selecting coefficients (or weights) that reflect the substantive importance of each indicator towards the concept of interest is a difficult task. We propose a method for the generation of linear indices for aggregate concepts based on pairwise comparisons. Specifically, we ask a group of subject‐matter experts to perform a series of pairwise comparisons, with respect to the concept of interest, between profiles displaying different combinations of indicators. This allows us to estimate coefficients for each indicator that provide a linear approximation to how experts make the pairwise evaluations. As we show, the method makes it straightforward to assess intercoder reliability, while being a more accessible task than directly asking experts for coefficients. We demonstrate our method with an application to the concept of ‘productive ageing’, including a cross‐cultural comparison of weighting schemes derived from a group of Italian and a group of South Korean experts on this concept.

Suggested Citation

  • Ginevra Floridi & Benjamin E. Lauderdale, 2022. "Pairwise comparisons as a scale development tool for composite measures," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(2), pages 519-542, April.
    • Handle: RePEc:bla:jorssa:v:185:y:2022:i:2:p:519-542
    DOI: 10.1111/rssa.12790
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssa.12790
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssa.12790?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hardesty, David M. & Bearden, William O., 2004. "The use of expert judges in scale development: Implications for improving face validity of measures of unobservable constructs," Journal of Business Research, Elsevier, vol. 57(2), pages 98-107, February.
    2. Carlson, David & Montgomery, Jacob M., 2017. "A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political Texts," American Political Science Review, Cambridge University Press, vol. 111(4), pages 835-843, November.
    3. Benoit, Kenneth & Conway, Drew & Lauderdale, Benjamin E. & Laver, Michael & Mikhaylov, Slava, 2016. "Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data," American Political Science Review, Cambridge University Press, vol. 110(2), pages 278-295, May.
    4. Willem Saris & Irmtraud Gallhofer, 2004. "Operationalization of Social Science Concepts by Intuition," Quality & Quantity: International Journal of Methodology, Springer, vol. 38(3), pages 235-258, June.
    5. Iñaki Permanyer, 2011. "Assessing The Robustness Of Composite Indices Rankings," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 57(2), pages 306-326, June.
    6. Massimiliano Bratti & Tommaso Frattini & Francesco Scervini, 2018. "Grandparental availability for child care and maternal labor force participation: pension reform evidence from Italy," Journal of Population Economics, Springer;European Society for Population Economics, vol. 31(4), pages 1239-1277, October.
    7. Bansak, Kirk & Hainmueller, Jens & Hopkins, Daniel J. & Yamamoto, Teppei, 2021. "Beyond the breaking point? Survey satisficing in conjoint experiments," Political Science Research and Methods, Cambridge University Press, vol. 9(1), pages 53-71, January.
    8. Bryony Hoskins & Massimiliano Mascherini, 2009. "Measuring Active Citizenship through the Development of a Composite Indicator," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 90(3), pages 459-488, February.
    9. Angilella, Silvia & Corrente, Salvatore & Greco, Salvatore & Słowiński, Roman, 2016. "Robust Ordinal Regression and Stochastic Multiobjective Acceptability Analysis in multiple criteria hierarchy process for the Choquet integral preference model," Omega, Elsevier, vol. 63(C), pages 154-169.
    10. Bruno Arpino & Chiara Pronzato & Lara Tavares, 2014. "The Effect of Grandparental Support on Mothers’ Labour Market Participation: An Instrumental Variable Approach," European Journal of Population, Springer;European Association for Population Studies, vol. 30(4), pages 369-390, November.
    11. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    12. Lindsey Baker & Lawrence Cahalin & Kerstin Gerst & Jeffrey Burr, 2005. "Productive Activities And Subjective Well-Being Among Older Adults: The Influence Of Number Of Activities And Time Commitment," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 73(3), pages 431-458, September.
    13. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    14. Dorothée Rouzet & Aida Caldera Sánchez & Theodore Renault & Oliver Roehn, 2019. "Fiscal challenges and inclusive growth in ageing societies," OECD Economic Policy Papers 27, OECD Publishing.
    15. Bruno Arpino & Valeria Bordone, 2017. "Regular provision of grandchild care and participation in social activities," Review of Economics of the Household, Springer, vol. 15(1), pages 135-174, March.
    16. Salvatore Greco & Alessio Ishizaka & Benedetto Matarazzo & Gianpiero Torrisi, 2018. "Stochastic multi-attribute acceptability analysis (SMAA): an application to the ranking of Italian regions," Regional Studies, Taylor & Francis Journals, vol. 52(4), pages 585-600, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giuliano Resce & Fritz Schiltz, 2021. "Sustainable Development in Europe: A Multicriteria Decision Analysis," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 67(2), pages 509-529, June.
    2. Giuseppe Coco & Raffaele Lagravinese & Giuliano Resce, 2020. "Beyond the weights: a multicriteria approach to evaluate inequality in education," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 18(4), pages 469-489, December.
    3. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    4. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "The Ordinal Input for Cardinal Output Approach of Non-compensatory Composite Indicators: The PROMETHEE Scoring Method," MPRA Paper 95816, University Library of Munich, Germany.
    5. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2021. "The ordinal input for cardinal output approach of non-compensatory composite indicators: the PROMETHEE scoring method," European Journal of Operational Research, Elsevier, vol. 288(1), pages 225-246.
    6. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.
    7. Lagravinese, Raffaele & Liberati, Paolo & Resce, Giuliano, 2019. "Exploring health outcomes by stochastic multicriteria acceptability analysis: An application to Italian regions," European Journal of Operational Research, Elsevier, vol. 274(3), pages 1168-1179.
    8. Rajko Tomaš, 2022. "Measurement of the Concentration of Potential Quality of Life in Local Communities," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 163(1), pages 79-109, August.
    9. Aleksandra Maksimovska & Aleksandar Stojkov, 2019. "Composite Indicator of Social Responsiveness of Local Governments: An Empirical Mapping of the Networked Community Governance Paradigm," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 669-706, July.
    10. Daniela Boca & Daniela Piazzalunga & Chiara Pronzato, 2018. "The role of grandparenting in early childcare and child outcomes," Review of Economics of the Household, Springer, vol. 16(2), pages 477-512, June.
    11. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2018. "σ-µ efficiency analysis: A new methodology for evaluating units through composite indices," MPRA Paper 83569, University Library of Munich, Germany.
    12. Carmen García-Peña & Moneyba González-Medina & Jose Manuel Diaz-Sarachaga, 2021. "Assessment of the Governance Dimension in the Frame of the 2030 Agenda: Evidence from 100 Spanish Cities," Sustainability, MDPI, vol. 13(10), pages 1-21, May.
    13. Wolfgang Frimmel & Martin Halla & Bernhard Schmidpeter & Rudolf Winter-Ebmer, 2022. "Grandmothers’ Labor Supply," Journal of Human Resources, University of Wisconsin Press, vol. 57(5), pages 1645-1689.
    14. Tsionas, Mike G., 2020. "A note on Sigma–Mu efficiency analysis as a methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 286(3), pages 1187-1196.
    15. R. Pelissari & M. C. Oliveira & S. Ben Amor & A. Kandakoglu & A. L. Helleno, 2020. "SMAA methods and their applications: a literature review and future research directions," Annals of Operations Research, Springer, vol. 293(2), pages 433-493, October.
    16. Ludovico Carrino, 2016. "Data Versus Survey-based Normalisation in a Multidimensional Analysis of Social Inclusion," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 2(3), pages 305-345, November.
    17. Suman Seth & Antonio Villar, 2014. "Human Development, Inequality and Poverty: empirical findings," Working Papers 14.11, Universidad Pablo de Olavide, Department of Economics.
    18. Edson Kogachi & Adonias Ferreira & Carlos Cavalcante & Marcelo Embiruçu, 2021. "Development of Performance Evaluation Indicators for Table Grape Packaging Units. 2. Global Indexes," Sustainability, MDPI, vol. 13(11), pages 1-16, June.
    19. James Foster & Mark McGillivray & Suman Seth, 2012. "Rank Robustness of Composite Indices: Dominance and Ambiguity," OPHI Working Papers 26b, Queen Elizabeth House, University of Oxford.
    20. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:185:y:2022:i:2:p:519-542. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.