IDEAS home Printed from https://ideas.repec.org/a/eee/wdevel/v74y2015icp184-196.html
   My bibliography  Save this article

Policy Evaluation via Composite Indexes: Qualitative Lessons from International Transparency Policy Indexes

Author

Listed:
  • Michener, Gregory

Abstract

International transparency policy indexes (ITPIs) help determine billions in investment and aid, influence “authoritative” scholarship, and shape policy choices. Are ITPIs valid yardsticks of transparency, or do they encourage dissimulation? Most scholarship on index-based evaluations focuses on “concept indexes” (e.g., governance) from quantitative approaches. This paper presents qualitative insights about ITPIs in specific and “policy indexes” in general, analyzing three measurement-related pitfalls and proposing countermeasures. Most significantly, it shows how indexes presuppose substitutability while policies contain nonsubstitutable ‘necessary’ policy provisions. This dilemma of “ontological compatibility” means that policies can rank favorably on indexes notwithstanding the absence of lynchpin policy provisions.

Suggested Citation

  • Michener, Gregory, 2015. "Policy Evaluation via Composite Indexes: Qualitative Lessons from International Transparency Policy Indexes," World Development, Elsevier, vol. 74(C), pages 184-196.
    • Handle: RePEc:eee:wdevel:v:74:y:2015:i:c:p:184-196
    DOI: 10.1016/j.worlddev.2015.04.016
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0305750X1500100X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.worlddev.2015.04.016?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Grimm, Michael & Harttgen, Kenneth & Klasen, Stephan & Misselhorn, Mark, 2008. "A Human Development Index by Income Groups," World Development, Elsevier, vol. 36(12), pages 2527-2546, December.
    2. Katherine Bersch & Sandra Botero, 2014. "Measuring Governance: Implications of Conceptual Choices," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 26(1), pages 124-141, January.
    3. Larsen, Rasmus Kløcker & Mamosso, Christiane Alzouma, 2014. "Aid with Blinkers: Environmental Governance of Uranium Mining in Niger," World Development, Elsevier, vol. 56(C), pages 62-76.
    4. Fox, Jonathan A, 2007. "The uncertain relationship between transparency and accountability," Center for Global, International and Regional Studies, Working Paper Series qt8c25c3z4, Center for Global, International and Regional Studies, UC Santa Cruz.
    5. Grupp, Hariolf & Mogee, Mary Ellen, 2004. "Indicators for national science and technology policy: how robust are composite indicators?," Research Policy, Elsevier, vol. 33(9), pages 1373-1384, November.
    6. Matt Andrews, 2008. "The Good Governance Agenda: Beyond Indicators without Theory," Oxford Development Studies, Taylor & Francis Journals, vol. 36(4), pages 379-407.
    7. Knoll, Martin & Zloczysti, Petra, 2012. "The Good Governance Indicators of the Millennium Challenge Account: How Many Dimensions are Really Being Measured?," World Development, Elsevier, vol. 40(5), pages 900-915.
    8. Pritchett, Lant & Woolcock, Michael, 2004. "Solutions When the Solution is the Problem: Arraying the Disarray in Development," World Development, Elsevier, vol. 32(2), pages 191-212, February.
    9. Sartori, Giovanni, 1970. "Concept Misformation in Comparative Politics," American Political Science Review, Cambridge University Press, vol. 64(4), pages 1033-1053, December.
    10. Matt Andrews, 2014. "An Ends-Means Approach to Looking at Governance," CID Working Papers 281, Center for International Development at Harvard University.
    11. Nelson, Paul J., 2001. "Transparency Mechanisms at the Multilateral Development Banks," World Development, Elsevier, vol. 29(11), pages 1835-1847, November.
    12. Heywood, Paul M. & Rose, Jonathan, 2014. "“Close but no Cigar†: the measurement of corruption," Journal of Public Policy, Cambridge University Press, vol. 34(3), pages 507-529, December.
    13. Chong Ju Choi & Carla C. J. M. Millar & Caroline Y. L. Wong, 2005. "Knowledge and the State," Palgrave Macmillan Books, in: Knowledge Entanglements, chapter 0, pages 19-38, Palgrave Macmillan.
    14. Koen Decancq & María Ana Lugo, 2013. "Weights in Multidimensional Indices of Wellbeing: An Overview," Econometric Reviews, Taylor & Francis Journals, vol. 32(1), pages 7-34, January.
    15. Gisselquist, Rachel M., 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series 068, World Institute for Development Economic Research (UNU-WIDER).
    16. Rachel M. Gisselquist, 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series wp-2013-068, World Institute for Development Economic Research (UNU-WIDER).
    17. Ragin, Charles C., 2000. "Fuzzy-Set Social Science," University of Chicago Press Economics Books, University of Chicago Press, edition 1, number 9780226702773, September.
    18. Gordon Mitchell, 1996. "Problems And Fundamentals Of Sustainable Development Indicators," Sustainable Development, John Wiley & Sons, Ltd., vol. 4(1), pages 1-11.
    19. Knack, Stephen & Rogers, F. Halsey & Eubank, Nicholas, 2011. "Aid Quality and Donor Rankings," World Development, Elsevier, vol. 39(11), pages 1907-1917.
    20. Charles P. Oman & Christiane Arndt, 2006. "Governance Indicators for Development," OECD Development Centre Policy Insights 33, OECD Publishing.
    21. Alkire, Sabina & Santos, Maria Emma, 2014. "Measuring Acute Poverty in the Developing World: Robustness and Scope of the Multidimensional Poverty Index," World Development, Elsevier, vol. 59(C), pages 251-274.
    22. Virginia Haufler, 2010. "Disclosure as Governance: The Extractive Industries Transparency Initiative and Resource Management in the Developing World," Global Environmental Politics, MIT Press, vol. 10(3), pages 53-73, August.
    23. Frederik Booysen, 2002. "An Overview and Evaluation of Composite Indices of Development," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 59(2), pages 115-151, August.
    24. Judith G. Kelley & Beth A. Simmons, 2015. "Politics by Number: Indicators as Social Pressure in International Relations," American Journal of Political Science, John Wiley & Sons, vol. 59(1), pages 55-70, January.
    25. Staffan Andersson & Paul M. Heywood, 2009. "The Politics of Perception: Use and Abuse of Transparency International's Approach to Measuring Corruption," Political Studies, Political Studies Association, vol. 57, pages 746-767, December.
    26. Hollyer, James R. & Rosendorff, B. Peter & Vreeland, James Raymond, 2014. "Measuring Transparency," Political Analysis, Cambridge University Press, vol. 22(4), pages 413-434.
    27. repec:dau:papers:123456789/4352 is not listed on IDEAS
    28. Razafindrakoto, Mireille & Roubaud, François, 2010. "Are International Databases on Corruption Reliable? A Comparison of Expert Opinion Surveys and Household Surveys in Sub-Saharan Africa," World Development, Elsevier, vol. 38(8), pages 1057-1069, August.
    29. Barrett, Christopher B. & Lee, David R. & McPeak, John G., 2005. "Institutional Arrangements for Rural Poverty Reduction and Resource Conservation," World Development, Elsevier, vol. 33(2), pages 193-197, February.
    30. Grupp, Hariolf & Schubert, Torben, 2010. "Review and new evidence on composite innovation indicators for evaluating national performance," Research Policy, Elsevier, vol. 39(1), pages 67-78, February.
    31. Knack, Stephen & Kugler, Mark & Manning, Nick, 2002. "Second generation governance indicators," MPRA Paper 25029, University Library of Munich, Germany, revised 2003.
    32. Harttgen, Kenneth & Klasen, Stephan, 2012. "A Household-Based Human Development Index," World Development, Elsevier, vol. 40(5), pages 878-899.
    33. Staffan Andersson & Paul M. Heywood, 2009. "The Politics of Perception: Use and Abuse of Transparency International's Approach to Measuring Corruption," Political Studies, Political Studies Association, vol. 57(4), pages 746-767, December.
    34. Ghosh, Anirban & Kharas, Homi, 2011. "The Money Trail: Ranking Donor Transparency in Foreign Aid," World Development, Elsevier, vol. 39(11), pages 1918-1929.
    35. Gingerich, Daniel W., 2013. "Governance Indicators and the Level of Analysis Problem: Empirical Findings from South America," British Journal of Political Science, Cambridge University Press, vol. 43(3), pages 505-540, July.
    36. Kaufmann, Daniel & Bellver, Ana, 2005. "Transparenting Transparency: Intial Empirics and Policy Applications," MPRA Paper 8188, University Library of Munich, Germany.
    37. Andrews, Matt, 2014. "An Ends-Means Approach to Looking at Governance," Working Paper Series rwp14-022, Harvard University, John F. Kennedy School of Government.
    38. Bermeo, Sarah Blodgett, 2011. "Foreign Aid and Regime Change: A Role for Donor Intent," World Development, Elsevier, vol. 39(11), pages 2021-2031.
    39. Collier, David & Mahon, James E., 1993. "Conceptual “Stretching†Revisited: Adapting Categories in Comparative Analysis," American Political Science Review, Cambridge University Press, vol. 87(4), pages 845-855, December.
    40. repec:ucp:bkecon:9780226702766 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Relly, Jeannine E. & Rabbi, Md. Fazle & Sabharwal, Meghna & Pakanati, Rajdeep & Schwalbe, Ethan H., 2020. "More than a decade in the making: A study of the implementation of India's Right to Information Act," World Development, Elsevier, vol. 136(C).
    2. Carpantier, J.-F. & Vermeulen, W.N., 2018. "Emergence of sovereign wealth funds," Journal of Commodity Markets, Elsevier, vol. 11(C), pages 1-21.
    3. Marta Kuc-Czarnecka & Samuele Lo Piano & Andrea Saltelli, 2020. "Quantitative Storytelling in the Making of a Composite Indicator," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 775-802, June.
    4. Gregory A. PORUMBESCU & Nicola BELLE & Maria CUCCINIELLO & Greta NASI, 2019. "Does Transparency Lead To Coproduction?," TAD 14 The disciplines and the study of Public Administration: Transatlantic perspectives in the margin of the 14th Administration and Public Management International Conference, Bucharest, June 6-18 6, Research Centre in Public Administration and Public Services, Bucharest, Romania.
    5. Biggeri, Mario & Clark, David A. & Ferrannini, Andrea & Mauro, Vincenzo, 2019. "Tracking the SDGs in an ‘integrated’ manner: A proposal for a new index to capture synergies and trade-offs between and within goals," World Development, Elsevier, vol. 122(C), pages 628-647.
    6. Lnenicka, Martin & Nikiforova, Anastasija & Luterek, Mariusz & Milic, Petar & Rudmark, Daniel & Neumaier, Sebastian & Santoro, Caterina & Flores, Cesar Casiano & Janssen, Marijn & Rodríguez Bolívar, M, 2023. "Identifying patterns and recommendations of and for sustainable open data initiatives: a benchmarking-driven analysis of open government data initiatives among European countries," SocArXiv v7msn, Center for Open Science.
    7. Martin Lnenicka & Anastasija Nikiforova & Mariusz Luterek & Petar Milic & Daniel Rudmark & Sebastian Neumaier & Caterina Santoro & Cesar Casiano Flores & Marijn Janssen & Manuel Pedro Rodr'iguez Bol'i, 2023. "Identifying patterns and recommendations of and for sustainable open data initiatives: a benchmarking-driven analysis of open government data initiatives among European countries," Papers 2312.00551, arXiv.org, revised Dec 2023.
    8. Ludger Niemann & Thomas Hoppe, 2021. "How to Sustain Sustainability Monitoring in Cities: Lessons from 49 Community Indicator Initiatives across 10 Latin American Countries," Sustainability, MDPI, vol. 13(9), pages 1-16, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gisselquist, Rachel M., 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series 068, World Institute for Development Economic Research (UNU-WIDER).
    2. Rachel M. Gisselquist, 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series wp-2013-068, World Institute for Development Economic Research (UNU-WIDER).
    3. Rachel M. Gisselquist, 2012. "Good Governance as a Concept, and Why This Matters for Development Policy," WIDER Working Paper Series wp-2012-030, World Institute for Development Economic Research (UNU-WIDER).
    4. Ward, Hugh & Dorussen, Han, 2015. "Public Information and Performance: The Role of Spatial Dependence in the Worldwide Governance Indicators among African Countries," World Development, Elsevier, vol. 74(C), pages 253-263.
    5. José-Miguel Bello y Villarino, 2021. "Measuring Corruption: A Critical Analysis of the Existing Datasets and Their Suitability for Diachronic Transnational Research," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 157(2), pages 709-747, September.
    6. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    7. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    8. Gisselquist, Rachel M., 2012. "Good Governance as a Concept, and Why This Matters for Development Policy," WIDER Working Paper Series 030, World Institute for Development Economic Research (UNU-WIDER).
    9. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2018. "σ-µ efficiency analysis: A new methodology for evaluating units through composite indices," MPRA Paper 83569, University Library of Munich, Germany.
    10. Khatab Alqararah, 2023. "Assessing the robustness of composite indicators: the case of the Global Innovation Index," Journal of Innovation and Entrepreneurship, Springer, vol. 12(1), pages 1-22, December.
    11. Gaoussou Diarra & Patrick Plane, 2014. "Assessing the World Bank's Influence on the Good Governance Paradigm," Oxford Development Studies, Taylor & Francis Journals, vol. 42(4), pages 473-487, December.
    12. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.
    13. Panagiotis Artelaris, 2022. "A development index for the Greek regions," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(3), pages 1261-1281, June.
    14. Tobias Heinrich & Yoshiharu Kobayashi, 2022. "Evaluating explanations for poverty selectivity in foreign aid," Kyklos, Wiley Blackwell, vol. 75(1), pages 30-47, February.
    15. Ricard Giné‐Garriga & Agustí Pérez‐Foguet, 2019. "Monitoring and targeting the sanitation poor: A multidimensional approach," Natural Resources Forum, Blackwell Publishing, vol. 43(2), pages 82-94, May.
    16. Carmelo León & Jorge Araña & Javier León, 2013. "Correcting for Scale Perception Bias in Measuring Corruption: an Application to Chile and Spain," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 114(3), pages 977-995, December.
    17. Gutmann, Jerg & Padovano, Fabio & Voigt, Stefan, 2020. "Perception vs. experience: Explaining differences in corruption measures using microdata," European Journal of Political Economy, Elsevier, vol. 65(C).
    18. Ferreira, Paulo Jorge Silveira & Dionísio, Andreia Teixeira Marques, 2016. "What are the conditions for good innovation results? A fuzzy-set approach for European Union," Journal of Business Research, Elsevier, vol. 69(11), pages 5396-5400.
    19. Gustav Lidén, 2013. "What about theory? The consequences on a widened perspective of social theory," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(1), pages 213-225, January.
    20. Eugen Dimant & Guglielmo Tosato, 2018. "Causes And Effects Of Corruption: What Has Past Decade'S Empirical Research Taught Us? A Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 32(2), pages 335-356, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:wdevel:v:74:y:2015:i:c:p:184-196. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/worlddev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.