IDEAS home Printed from https://ideas.repec.org/a/spr/revint/v18y2023i4d10.1007_s11558-023-09489-1.html
   My bibliography  Save this article

The performance of international organizations: a new measure and dataset based on computational text analysis of evaluation reports

Author

Listed:
  • Steffen Eckhard

    (Zeppelin University)

  • Vytautas Jankauskas

    (Zeppelin University)

  • Elena Leuschner

    (University of Gothenburg)

  • Ian Burton

    (Zeppelin University)

  • Tilman Kerl

    (Vienna University of Technology)

  • Rita Sevastjanova

    (University of Konstanz)

Abstract

International organizations (IOs) of the United Nations (UN) system publish around 750 evaluation reports per year, offering insights on their performance across project, program, institutional, and thematic activities. So far, it was not feasible to extract quantitative performance measures from these text-based reports. Using deep learning, this article presents a novel text-based performance metric: We classify individual sentences as containing a negative, positive, or neutral assessment of the evaluated IO activity and then compute the share of positive sentences per report. Content validation yields that the measure adequately reflects the underlying concept of performance; convergent validation finds high correlation with human-provided performance scores by the World Bank; and construct validation shows that our measure has theoretically expected results. Based on this, we present a novel dataset with performance measures for 1,082 evaluated activities implemented by nine UN system IOs and discuss avenues for further research.

Suggested Citation

  • Steffen Eckhard & Vytautas Jankauskas & Elena Leuschner & Ian Burton & Tilman Kerl & Rita Sevastjanova, 2023. "The performance of international organizations: a new measure and dataset based on computational text analysis of evaluation reports," The Review of International Organizations, Springer, vol. 18(4), pages 753-776, October.
    • Handle: RePEc:spr:revint:v:18:y:2023:i:4:d:10.1007_s11558-023-09489-1
    DOI: 10.1007/s11558-023-09489-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11558-023-09489-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11558-023-09489-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Feeny, Simon & Vuong, Vu, 2017. "Explaining Aid Project and Program Success: Findings from Asian Development Bank Interventions," World Development, Elsevier, vol. 90(C), pages 329-343.
    2. Honig, Dan & Weaver, Catherine, 2019. "A Race to the Top? The Aid Transparency Index and the Social Power of Global Performance Indicators – ERRATUM," International Organization, Cambridge University Press, vol. 73(3), pages 709-711, July.
    3. Tamar Gutner & Alexander Thompson, 2010. "The politics of IO performance: A framework," The Review of International Organizations, Springer, vol. 5(3), pages 227-248, September.
    4. Mette Eilstrup-Sangiovanni, 2020. "Death of international organizations. The organizational ecology of intergovernmental organizations, 1815–2015," The Review of International Organizations, Springer, vol. 15(2), pages 339-370, April.
    5. Axel Dreher & Stephan Klasen & James Raymond Vreeland & Eric Werker, 2013. "The Costs of Favoritism: Is Politically Driven Aid Less Effective?," Economic Development and Cultural Change, University of Chicago Press, vol. 62(1), pages 157-191.
    6. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    7. Denizer, Cevdet & Kaufmann, Daniel & Kraay, Aart, 2013. "Good countries or good projects? Macro and micro correlates of World Bank project performance," Journal of Development Economics, Elsevier, vol. 105(C), pages 288-302.
    8. Silvia Marchesi & Tania Masi, 2021. "Delegation of implementation in project aid," The Review of International Organizations, Springer, vol. 16(3), pages 655-687, July.
    9. David Bulman & Walter Kolkma & Aart Kraay, 2017. "Good countries or good projects? Comparing macro and micro correlates of World Bank and Asian Development Bank project performance," The Review of International Organizations, Springer, vol. 12(3), pages 335-363, September.
    10. Inken Borzyskowski & Felicity Vabulas, 2019. "Hello, goodbye: When do states withdraw from international organizations?," The Review of International Organizations, Springer, vol. 14(2), pages 335-366, June.
    11. Mirko Heinzel & Andrea Liese, 2021. "Managing performance and winning trust: how World Bank staff shape recipient performance," The Review of International Organizations, Springer, vol. 16(3), pages 625-653, July.
    12. Mitchell Watkins, 2022. "Undermining conditionality? The effect of Chinese development assistance on compliance with World Bank project agreements," The Review of International Organizations, Springer, vol. 17(4), pages 667-690, October.
    13. Honig, Dan & Weaver, Catherine, 2019. "A Race to the Top? The Aid Transparency Index and the Social Power of Global Performance Indicators," International Organization, Cambridge University Press, vol. 73(3), pages 579-610, July.
    14. Lall, Ranjit, 2017. "Beyond Institutional Design: Explaining the Performance of International Organizations," International Organization, Cambridge University Press, vol. 71(2), pages 245-280, April.
    15. Mark T. Buntaine & Bradley C. Parks, 2013. "When Do Environmentally Focused Assistance Projects Achieve their Objectives? Evidence from World Bank Post-Project Evaluations," Global Environmental Politics, MIT Press, vol. 13(2), pages 65-88, May.
    16. Thomas Flores & Irfan Nooruddin, 2009. "Financing the peace: Evaluating World Bank post-conflict assistance programs," The Review of International Organizations, Springer, vol. 4(1), pages 1-27, March.
    17. Honig, Dan, 2019. "When Reporting Undermines Performance: The Costs of Politically Constrained Organizational Autonomy in Foreign Aid Implementation," International Organization, Cambridge University Press, vol. 73(1), pages 171-201, January.
    18. Lührmann, Anna & Marquardt, Kyle L. & Mechkova, Valeriya, 2020. "Constraining Governments: New Indices of Vertical, Horizontal, and Diagonal Accountability," American Political Science Review, Cambridge University Press, vol. 114(3), pages 811-820, August.
    19. Catherine Weaver, 2010. "The politics of performance evaluation: Independent evaluation at the International Monetary Fund," The Review of International Organizations, Springer, vol. 5(3), pages 365-385, September.
    20. Michael Lipson, 2010. "Performance under ambiguity: International organization performance in UN peacekeeping," The Review of International Organizations, Springer, vol. 5(3), pages 249-284, September.
    21. Thomas Stubbs & Bernhard Reinsberg & Alexander Kentikelenis & Lawrence King, 2020. "How to evaluate the effects of IMF conditionality," The Review of International Organizations, Springer, vol. 15(1), pages 29-73, January.
    22. Ben Cormier & Mark S. Manger, 2022. "Power, ideas, and World Bank conditionality," The Review of International Organizations, Springer, vol. 17(3), pages 397-425, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mirko Heinzel, 2023. "Vytautas Jankauskas and Steffen Eckhard. 2023. The Politics of Evaluation in International Organizations (Oxford: Oxford University Press)," The Review of International Organizations, Springer, vol. 18(4), pages 777-781, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Heinzel, Mirko & Reinsberg, Bernhard, 2024. "Trust funds and the sub-national effectiveness of development aid: Evidence from the World Bank," World Development, Elsevier, vol. 179(C).
    2. Silvia Marchesi & Tania Masi, 2021. "Delegation of implementation in project aid," The Review of International Organizations, Springer, vol. 16(3), pages 655-687, July.
    3. Philip Keefer & Christopher Kilby, 2021. "Introduction to the special issue: In memoriam Stephen Knack," The Review of International Organizations, Springer, vol. 16(3), pages 473-493, July.
    4. Mirko Heinzel, 2023. "Vytautas Jankauskas and Steffen Eckhard. 2023. The Politics of Evaluation in International Organizations (Oxford: Oxford University Press)," The Review of International Organizations, Springer, vol. 18(4), pages 777-781, October.
    5. Mirko Heinzel & Andrea Liese, 2021. "Managing performance and winning trust: how World Bank staff shape recipient performance," The Review of International Organizations, Springer, vol. 16(3), pages 625-653, July.
    6. Silvia Marchesi & Tania Masi & Saumik Paul, 2021. "Project Aid and Firm Performance," Working Papers 479, University of Milano-Bicocca, Department of Economics, revised Sep 2021.
    7. Eilers, Yota & Kluve, Jochen & Langbein, Jörg & Reiners, Lennart, 2023. "Volume, Risk, Complexity: What Makes Development Finance Projects Succeed or Fail?," IZA Discussion Papers 16691, Institute of Labor Economics (IZA).
    8. Silvia, Marchesi & Tania, Masi, 2019. "Allocation of implementing power: Evidence from World Bank projects," Working Papers 399, University of Milano-Bicocca, Department of Economics, revised Jan 2019.
    9. Bedasso, Biniam, 2024. "Ministerial musical chairs: Does leadership turnover undermine the effectiveness of World Bank education aid?," World Development Perspectives, Elsevier, vol. 33(C).
    10. Hauke Feil, 2021. "The cancer of corruption and World Bank project performance: Is there a connection?," Development Policy Review, Overseas Development Institute, vol. 39(3), pages 381-397, May.
    11. Yoram Z. Haftel & Tobias Lenz, 2022. "Measuring institutional overlap in global governance," The Review of International Organizations, Springer, vol. 17(2), pages 323-347, April.
    12. Andrew Lugg, 2024. "Re-contracting intergovernmental organizations: Membership change and the creation of linked intergovernmental organizations," The Review of International Organizations, Springer, vol. 19(3), pages 545-577, September.
    13. Bernhard Reinsberg & Christian Siauwijaya, 2024. "Does earmarked funding affect the performance of international organisations?," Global Policy, London School of Economics and Political Science, vol. 15(1), pages 23-39, February.
    14. Heidi Hardt, 2018. "Who matters for memory: Sources of institutional memory in international organization crisis management," The Review of International Organizations, Springer, vol. 13(3), pages 457-482, September.
    15. Heinzel, Mirko & Reinsberg, Bernhard, 2024. "Trust funds and the sub-national effectiveness of development aid: evidence from the World Bank," LSE Research Online Documents on Economics 122593, London School of Economics and Political Science, LSE Library.
    16. Terence Wood & Sabit Otor & Matthew Dornan, 2020. "Australian aid projects: What works, where projects work and how Australia compares," Asia and the Pacific Policy Studies, Wiley Blackwell, vol. 7(2), pages 171-186, May.
    17. Bernhard Reinsberg & Oliver Westerwinter, 2023. "Institutional Overlap in Global Governance and the Design of Intergovernmental Organizations," The Review of International Organizations, Springer, vol. 18(4), pages 693-724, October.
    18. Manfred Elsig, 2010. "The World Trade Organization at work: Performance in a member-driven milieu," The Review of International Organizations, Springer, vol. 5(3), pages 345-363, September.
    19. Tamar Gutner & Alexander Thompson, 2010. "The politics of IO performance: A framework," The Review of International Organizations, Springer, vol. 5(3), pages 227-248, September.
    20. Christopher Kilby & Katharina Michaelowa, 2019. "What Influences World Bank Project Evaluations?," Springer Books, in: Nabamita Dutta & Claudia R. Williamson (ed.), Lessons on Foreign Aid and Economic Development, chapter 0, pages 109-150, Springer.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:revint:v:18:y:2023:i:4:d:10.1007_s11558-023-09489-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.