IDEAS home Printed from https://ideas.repec.org/p/ajk/ajkdps/276.html
   My bibliography  Save this paper

No More Cost in Translation: Validating Open-Source Machine Translation for Quantitative Text Analysis

Author

Listed:
  • Hauke Licht

    (University of Cologne)

  • Ronja Sczepanski

    (Sciences Po Paris)

  • Moritz Laurer

    (Hugging Face; Vrije Universiteit Amsterdam)

  • Ayjeren Bekmuratovna

    (DHL)

Abstract

As more and more scholars apply computational text analysis methods to multilingual corpora, machine translation has become an indispensable tool. However, relying on commercial services for machine translation, such as Google Translate or DeepL, limits reproducibility and can be expensive. This paper assesses the viability of a reproducible and affordable alternative: free and open-source machine translation models. We ask whether researchers who use an open-source model instead of a commercial service for machine translation would obtain substantially different measurements from their multilingual corpora. We address this question by replicating and extending an influential study by de Vries et al. (2018) on the use of machine translation in cross-lingual topic modeling, and an original study of its use in supervised text classification with Transformer-based classifiers. We find only minor differences between the measurements generated by these methods when applied to corpora translated with open-source models and commercial services, respectively. We conclude that “free” machine translation is a very valuable addition to researchers’ multilingual text analysis toolkit. Our study adds to a growing body of work on multilingual text analysis methods and has direct practical implications for applied researchers.

Suggested Citation

  • Hauke Licht & Ronja Sczepanski & Moritz Laurer & Ayjeren Bekmuratovna, 2024. "No More Cost in Translation: Validating Open-Source Machine Translation for Quantitative Text Analysis," ECONtribute Discussion Papers Series 276, University of Bonn and University of Cologne, Germany.
  • Handle: RePEc:ajk:ajkdps:276
    as

    Download full text from publisher

    File URL: https://www.econtribute.de/RePEc/ajk/ajkdps/ECONtribute_276_2024.pdf
    File Function: First version, 2024
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. de Vries, Erik & Schoonvelde, Martijn & Schumacher, Gijs, 2018. "No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications," Political Analysis, Cambridge University Press, vol. 26(4), pages 417-430, October.
    2. Lind, Fabienne & Heidenreich, Tobias & Kralj, Christoph & Boomgaarden, Hajo G., 2021. "Greasing the wheels for comparative communication research: Supervised text classification for multilingual corpora," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 3(3), pages 1-30.
    3. Düpont, Nils & Rachuj, Martin, 2022. "The Ties That Bind: Text Similarities and Conditional Diffusion among Parties," British Journal of Political Science, Cambridge University Press, vol. 52(2), pages 613-630, April.
    4. Zobel, Malisa & Lehmann, Pola, 2018. "Positions and saliency of immigration in party manifestos: A novel dataset using crowd coding," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 57(4), pages 1056-1083.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fraccaroli, Nicolò & Giovannini, Alessandro & Jamet, Jean-François & Persson, Eric, 2022. "Ideology and monetary policy. The role of political parties’ stances in the European Central Bank’s parliamentary hearings," European Journal of Political Economy, Elsevier, vol. 74(C).
    2. Konstantinos G. Papaspyropoulos & Harikleia Liakou & Panayotis Dimopoulos, 2023. "Climate Change in the Biodiversity and Forest Strategies in Greece Using Discourse Analysis and Text Mining: Is an Integration into a Cost-Efficient Natural Resources Policy Feasible?," Sustainability, MDPI, vol. 15(7), pages 1-12, April.
    3. Ferrara, Federico M. & Masciandaro, Donato & Moschella, Manuela & Romelli, Davide, 2022. "Political voice on monetary policy: Evidence from the parliamentary hearings of the European Central Bank," European Journal of Political Economy, Elsevier, vol. 74(C).
    4. Mónica D. Oliveira & Inês Mataloto & Panos Kanavos, 2019. "Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 891-918, August.
    5. Christopher Wratil & Sara B Hobolt, 2019. "Public deliberations in the Council of the European Union: Introducing and validating DICEU," European Union Politics, , vol. 20(3), pages 511-531, September.
    6. Seraphine F. Maerz & Carsten Q. Schneider, 2020. "Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 517-545, April.
    7. Nicolò Fraccaroli & Alessandro Giovannini & Jean-François Jamet & Eric Persson, 2023. "Central Banks in Parliaments: A Text Analysis of the Parliamentary Hearings of the Bank of England, the European Central Bank, and the Federal Reserve," International Journal of Central Banking, International Journal of Central Banking, vol. 19(2), pages 543-600, June.
    8. Laia Subirats & Jordi Conesa & Manuel Armayones, 2020. "Biomedical Holistic Ontology for People with Rare Diseases," IJERPH, MDPI, vol. 17(17), pages 1-11, August.
    9. Woohyuk Kim & Sung-Bum Kim & Eunhye Park, 2021. "Mapping Tourists’ Destination (Dis)Satisfaction Attributes with User-Generated Content," Sustainability, MDPI, vol. 13(22), pages 1-16, November.
    10. Markus Gastinger & Henning Schmidtke, 2023. "Measuring precision precisely: A dictionary-based measure of imprecision," The Review of International Organizations, Springer, vol. 18(3), pages 553-571, July.
    11. Luis Aue & Florian Börgel, 2023. "From “Bangtan Boys” to “International Relations Professor”: Mapping Self‐Identifications in the UN’s Twitter Public," Politics and Governance, Cogitatio Press, vol. 11(3), pages 120-133.
    12. Heidenreich, Tobias & Eberl, Jakob-Moritz & Lind, Fabienne & Boomgaarden, Hajo G., 2022. "Discontentment trumps Euphoria: Interacting with European Politicians’ migration-related messages on social media," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, issue OnlineFir, pages 1-1.
    13. Bes, Bart Joachim & Schoonvelde, Martijn & Rauh, Christian, 2020. "Undermining, defusing or defending European integration? Assessing public communication of European executives in times of EU politicisation," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 59(2), pages 397-423.
    14. Fraccaroli, Nicolò & Giovannini, Alessandro & Jamet, Jean-Francois & Persson, Eric, 2022. "Ideology and monetary policy: the role of political parties’ stances in the ECB’s parliamentary hearings," Working Paper Series 2655, European Central Bank.
    15. Winslet Ting Yan Chan & Chi Hong Leung, 2021. "Mind the Gap: Discrepancy Between Customer Expectation and Perception on Commercial Chatbots Usage," Asian Journal of Empirical Research, Asian Economic and Social Society, vol. 11(1), pages 1-10, January.
    16. Damien Pennetreau & Thomas Laloux, 2021. "Talkin’ ‘bout a Negotiation: (Un)Transparent Rapporteurs’ Speeches in the European Parliament," Politics and Governance, Cogitatio Press, vol. 9(1), pages 248-260.
    17. Donato Masciandaro & Oana Peia & Davide Romelli, 2024. "Central bank communication and social media: From silence to Twitter," Journal of Economic Surveys, Wiley Blackwell, vol. 38(2), pages 365-388, April.
    18. Lazaros Vrysis & Nikolaos Vryzas & Rigas Kotsakis & Theodora Saridou & Maria Matsiola & Andreas Veglis & Carlos Arcila-Calderón & Charalampos Dimoulas, 2021. "A Web Interface for Analyzing Hate Speech," Future Internet, MDPI, vol. 13(3), pages 1-18, March.
    19. Ueli Reber & Manuel Fischer & Karin Ingold & Felix Kienast & Anna M. Hersperger & Rolf Grütter & Robin Benz, 2022. "Integrating biodiversity: a longitudinal and cross-sectoral analysis of Swiss politics," Policy Sciences, Springer;Society of Policy Sciences, vol. 55(2), pages 311-335, June.
    20. Andrea Häuptli & Daniel Vogler, 2024. "Assessing the leeway of state-led strategic communication abroad: a comparison of news coverage on Austria, Germany, and Switzerland in Arabic," Place Branding and Public Diplomacy, Palgrave Macmillan, vol. 20(1), pages 12-23, March.

    More about this item

    Keywords

    machine translation; multilingual topic modeling; multilingual Transformers;
    All these keywords.

    JEL classification:

    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ajk:ajkdps:276. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ECONtribute Office (email available below). General contact details of provider: https://www.econtribute.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.