IDEAS home Printed from https://ideas.repec.org/p/hes/wpaper/0298.html

Automated historical census digitization using image augmentation and transformer-based methods

Author

Listed:
  • Leonardo Costa Ribeiro

    (Federal University of Minas Gerais)

  • Jonatan Andersson

    (Uppsala University)

  • William Skoglund

    (Lund University)

  • Jakob Molinder

    (Uppsala University)

  • Martin Önnerfors

    (Uppsala University)

Abstract

A large literature in economic history uses digitized census data to study individual-level outcomes in history. Although many census records have been digitized manually, the process is extremely labor-intensive, and substantial material remains unprocessed in archives. Recent advances in machine learning offer the potential to automate large part of this work. We demonstrate an end-to-end digitization pipeline based on the transformer-based Donut model, trained on hand-annotated data and enhanced with image augmentation, to extract information from the 1955 Stockholm tax and census records. The resulting output attains high accuracy across multiple evaluation metrics.

Suggested Citation

  • Leonardo Costa Ribeiro & Jonatan Andersson & William Skoglund & Jakob Molinder & Martin Önnerfors, 2026. "Automated historical census digitization using image augmentation and transformer-based methods," Working Papers 0298, European Historical Economics Society (EHES).
    • Handle: RePEc:hes:wpaper:0298
    as

    Download full text from publisher

    File URL: https://ehes.org/wp/EHES_298.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Blomqvist, Christopher & Enflo, Kerstin & Jakobsson, Andreas & Åström, Kalle, 2023. "Reading the ransom: Methodological advancements in extracting the Swedish Wealth Tax of 1571," Explorations in Economic History, Elsevier, vol. 87(C).
    2. William J. Collins & Marianne H. Wanamaker, 2014. "Selection and Economic Gains in the Great Migration of African Americans: New Evidence from Linked Census Data," American Economic Journal: Applied Economics, American Economic Association, vol. 6(1), pages 220-252, January.
    3. Long, Jason, 2005. "Rural-Urban Migration and Socioeconomic Mobility in Victorian Britain," The Journal of Economic History, Cambridge University Press, vol. 65(1), pages 1-35, March.
    4. Joseph P. Ferrie, 1999. "Yankeys Now: Immigrants in the Antebellum US 1840-1860," NBER Books, National Bureau of Economic Research, Inc, number ferr99-1, August.
    5. Christian M. Dahl & Torben S. D. Johansen & Emil N. Sørensen & Christian E. Westermann & Simon Wittrock, 2023. "Applications of machine learning in tabular document digitisation," Historical Methods: A Journal of Quantitative and Interdisciplinary History, Taylor & Francis Journals, vol. 56(1), pages 34-48, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ward, Zachary, 2017. "Birds of passage: Return migration, self-selection and immigration quotas," Explorations in Economic History, Elsevier, vol. 64(C), pages 37-52.
    2. Martin Dribe & Christer Lundh, 2010. "Marriage choices and social reproduction," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 22(14), pages 347-382.
    3. Andersson, Jonatan & Molinder, Jakob, 2025. "Did cities increase skills during industrialization? Evidence from rural-urban migration," Journal of Urban Economics, Elsevier, vol. 148(C).
    4. Dribe, Martin & Eriksson, Björn & Scalone, Francesco, 2019. "Migration, marriage and social mobility: Women in Sweden 1880–1900," Explorations in Economic History, Elsevier, vol. 71(C), pages 93-111.
    5. Long, Jason & Siu, Henry, 2018. "Refugees from Dust and Shrinking Land: Tracking the Dust Bowl Migrants," The Journal of Economic History, Cambridge University Press, vol. 78(4), pages 1001-1033, December.
    6. Catherine G. Massey, 2016. "Playing with Matches: An Assessment of Accuracy in Linked Historical Data," CARRA Working Papers 2016-05, Center for Economic Studies, U.S. Census Bureau.
    7. Ran Abramitzky & Philipp Ager & Leah Boustan & Elior Cohen & Casper Hansen, 2021. "The Effect of Immigration on Local Labor Markets: Lessons from the 1920s Border Closure," Research Working Paper RWP 21-09, Federal Reserve Bank of Kansas City.
    8. Yacine Boujija & Marie Connolly & Xavier St-Denis, 2023. "Mobilité géographique et transmission intergénérationnelle du revenu au Québec," CIRANO Project Reports 2023rp-11, CIRANO.
    9. Zachary Ward, 2019. "Internal Migration, Education and Upward Rank Mobility:Evidence from American History," CEH Discussion Papers 04, Centre for Economic History, Research School of Economics, Australian National University.
    10. Andersson, Jonatan, 2025. "Ascending from the bottom rung: The labor market assimilation of rural-urban migrants in Sweden, 1880–1910," Explorations in Economic History, Elsevier, vol. 97(C).
    11. Bryan A. Stuart & Evan J. Taylor, 2021. "Migration Networks and Location Decisions: Evidence from US Mass Migration," American Economic Journal: Applied Economics, American Economic Association, vol. 13(3), pages 134-175, July.
    12. Richard Hornbeck, 2020. "Dust Bowl Migrants: Identifying an Archetype," Working Papers 2020-120, Becker Friedman Institute for Research In Economics.
    13. Bailey, Roy E. & Hatton, Timothy J. & Inwood, Kris, 2016. "Atmospheric Pollution and Child Health in Late Nineteenth Century Britain," IZA Discussion Papers 10428, IZA Network @ LISER.
    14. Barbara Broadway & Anna Zhu, 2023. "Spatial heterogeneity in welfare reform success," Melbourne Institute Working Paper Series wp2023n13, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    15. Krzysztof Karbownik & Anthony Wray, 2019. "Long-Run Consequences of Exposure to Natural Disasters," Journal of Labor Economics, University of Chicago Press, vol. 37(3), pages 949-1007.
    16. Chiswick, Barry R. & Robinson, RaeAnn H., 2023. "The Occupational Attainment of American Jewish Men in the Mid-19th Century," GLO Discussion Paper Series 1256, Global Labor Organization (GLO).
    17. Michael A. Clemens & Claudio Montenegro & Lant Pritchett, 2016. "Bounding the Price Equivalent of Migration Barriers," Growth Lab Working Papers 67, Harvard's Growth Lab.
    18. Chiswick, Barry R. & Robinson, RaeAnn Halenda, 2024. "The Labor Market Attainment of Immigrants in the Antebellum United States," GLO Discussion Paper Series 1512, Global Labor Organization (GLO).
    19. Combes, Pierre-Philippe & Gobillon, Laurent & Zylberberg, Yanos, 2022. "Urban economics in a historical perspective: Recovering data with machine learning," Regional Science and Urban Economics, Elsevier, vol. 94(C).
    20. Broadberry, Stephen & Ghosal, Sayantan & Proto, Eugenio, 2017. "Anonymity, efficiency wages and technological progress," Journal of Development Economics, Elsevier, vol. 127(C), pages 379-394.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • N01 - Economic History - - General - - - Development of the Discipline: Historiographical; Sources and Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hes:wpaper:0298. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Christian Vedel (email available below). General contact details of provider: https://edirc.repec.org/data/ehessea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.