IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2402.13604.html

Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE

Author

Listed:
  • Christian M{o}ller Dahl
  • Torben Johansen
  • Christian Vedel

Abstract

This paper introduces OccCANINE, an open-source tool that maps occupational descriptions to HISCO codes. Manual coding is slow and error-prone; OccCANINE replaces weeks of work with results in minutes. We fine-tune CANINE on 15.8 million description-code pairs from 29 sources in 13 languages. The model achieves 96 percent accuracy, precision, and recall. We also show that the approach generalizes to three systems - OCC1950, OCCICEM, and ISCO-68 - and release them open source. By breaking the "HISCO barrier," OccCANINE democratizes access to high-quality occupational coding, enabling broader research in economics, economic history, and related disciplines.

Suggested Citation

  • Christian M{o}ller Dahl & Torben Johansen & Christian Vedel, 2024. "Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE," Papers 2402.13604, arXiv.org, revised Feb 2026.
    • Handle: RePEc:arx:papers:2402.13604
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2402.13604
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Allen,Robert C., 2009. "The British Industrial Revolution in Global Perspective," Cambridge Books, Cambridge University Press, number 9780521868273, January.
    2. Berger, Thor, 2019. "Railroads and Rural Industrialization: evidence from a Historical Policy Experiment," Explorations in Economic History, Elsevier, vol. 74(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tom Gorges & Magnus {O}rberg Rove & Paul Sharp & Christian Vedel, 2025. "Tracks to Modernity: Railroads, Growth, and Social Movements in Denmark," Papers 2502.21141, arXiv.org, revised Dec 2025.
    2. Jeanet Sinding Bentzen & Nina Boberg-Fazlić & Paul Sharp & Christian Volmar Skovsgaard & Christian Vedel, 2024. "Assimilate for God: The Impact of Religious Divisions on Danish American Communities," Working Papers 0253, European Historical Economics Society (EHES).
    3. repec:ehl:lserod:128849 is not listed on IDEAS
    4. Andersson, Jonatan, 2025. "Ascending from the bottom rung: The labor market assimilation of rural-urban migrants in Sweden, 1880–1910," Explorations in Economic History, Elsevier, vol. 97(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Haris Kitsikopoulos, 2024. "Steam power diffusion in the British cotton and woolen industries, 1774–1800: the role of firm size," Review of Evolutionary Political Economy, Springer, vol. 5(3), pages 559-580, October.
    2. Schreiner, Lena & Madlener, Reinhard, 2022. "Investing in power grid infrastructure as a flexibility option: A DSGE assessment for Germany," Energy Economics, Elsevier, vol. 107(C).
    3. Thomas Baudin & Robert Stelter, 2022. "The rural exodus and the rise of Europe," Journal of Economic Growth, Springer, vol. 27(3), pages 365-414, September.
    4. Zamparelli, Luca, 2022. "On Labor Productivity Growth and the Wage Share with Endogenous Size and Direction of Technical Change," MPRA Paper 112684, University Library of Munich, Germany.
    5. Pamfili Antipa & Vincent Bignon, 2018. "Whither Economic History? Between Narratives and Quantification," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(3), pages 17-36.
    6. Claridge, Jordan & Delabastita, Vincent & Gibbs, Spike, 2024. "(In-kind) Wages and labour relations in the Middle Ages: It’s not (all) about the money," Explorations in Economic History, Elsevier, vol. 94(C).
    7. Ferreira, Pedro Cavalcanti & Pessôa, Samuel & Santos, Marcelo Rodrigues, 2011. "Globalization and the Industrial Revolution," Insper Working Papers wpe_253, Insper Working Paper, Insper Instituto de Ensino e Pesquisa.
    8. Huning, Thilo R. & Wahl, Fabian, 2017. "Lord of the lemons: Origin and dynamics of state capacity," Hohenheim Discussion Papers in Business, Economics and Social Sciences 22-2017, University of Hohenheim, Faculty of Business, Economics and Social Sciences.
    9. Julius Koschnick, 2025. "Teacher-directed scientific change:The case of the English Scientific Revolution," Working Papers 0274, European Historical Economics Society (EHES).
    10. repec:ehl:lserod:119284 is not listed on IDEAS
    11. Klas Rönnbäck, 2014. "Slave ownership and fossil fuel usage: a commentary," Climatic Change, Springer, vol. 122(1), pages 1-9, January.
    12. O'Brien, Patrick, 2018. "Cosmographies for the discovery, development and diffusion of useful and reliable knowledge in pre-industrial Europe and Late imperial China: a survey and speculation," Economic History Working Papers 90534, London School of Economics and Political Science, Department of Economic History.
    13. Daron Acemoglu & James A. Robinson, 2015. "The Rise and Decline of General Laws of Capitalism," Journal of Economic Perspectives, American Economic Association, vol. 29(1), pages 3-28, Winter.
    14. Dakpogan, Arnaud & Smit, Eon, 2018. "The effect of electricity losses on GDP in Benin," MPRA Paper 89545, University Library of Munich, Germany.
    15. Allen, Robert C., 2012. "Backward into the future: The shift to coal and implications for the next energy transition," Energy Policy, Elsevier, vol. 50(C), pages 17-23.
    16. Boško Mijatović & Branko Milanović, 2021. "The real urban wage in an agricultural economy without landless farmers: Serbia, 1862–1910," Economic History Review, Economic History Society, vol. 74(2), pages 424-448, May.
    17. Ewout Frankema & Marlous van Waijenburg, 2011. "African Real Wages in Asian Perspective, 1880-1940," Working Papers 0002, Utrecht University, Centre for Global Economic History.
    18. Stephen Broadberry & Bruce Campbell & Alexander Klein & Mark Overton, 2010. "British economic growth, 1300-1850: some preliminary estimates," Working Papers 10009, Economic History Society.
    19. Cédric Chambru & Paul Maneuvrier‐Hervieu, 2023. "The evolution of wages in early modern Normandy (1600–1850)," Economic History Review, Economic History Society, vol. 76(3), pages 917-940, August.
    20. Vincent Geloso & Peter Lindert, 2020. "Relative costs of living, for richer and poorer, 1688–1914," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 14(3), pages 417-442, September.
    21. Edwyna Harris & Sumner La Croix, 2019. "Prices, Wages, and Welfare in Early Colonial South Australia, 1836-1850," Monash Economics Working Papers 07-19, Monash University, Department of Economics.

    More about this item

    JEL classification:

    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J1 - Labor and Demographic Economics - - Demographic Economics
    • N01 - Economic History - - General - - - Development of the Discipline: Historiographical; Sources and Methods
    • N3 - Economic History - - Labor and Consumers, Demography, Education, Health, Welfare, Income, Wealth, Religion, and Philanthropy
    • N6 - Economic History - - Manufacturing and Construction
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2402.13604. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.