IDEAS home Printed from https://ideas.repec.org/a/eee/aosoci/v113y2024ics0361368224000278.html
   My bibliography  Save this article

Algorithmic self-referentiality: How machine learning pushes calculative practices to assess themselves

Author

Listed:
  • Millo, Yuval
  • Spence, Crawford
  • Xu, Ruowen

Abstract

Despite the growing importance of machine learning in today's organisations, we know relatively little about how machine learning operates and how it influences calculative practices and cultures. Based on 695 hours of ethnographic fieldwork in the team of credit modellers from a large internet company in China, this study analyses the calculative culture that underpins the development of credit models. We show that credit scoring methodologies develop progressively into a self-referential set of calculative practices where substantive concerns about loan default are supplanted by more insular concerns around the seamless operation of the model. Insofar as the latter can only be measured by the model itself, this reduces the role of calculative experts to facilitators of machine learning rather than the purposeful interpreters of machine learning produced data. In this regard, credit scoring experts focus more on ensuring that models have a robust conversation with themselves rather than with managers or credit scoring agents. This matters because machine learning-driven credit scoring models end up privileging access to credit for those whose data trails more readily pass through data preparation filters rather than those who are less likely to default. We thus contribute to an understanding of how machine learning-driven calculative cultures both enact algorithmic bias and operate beyond the ken of purposeful human actors.

Suggested Citation

  • Millo, Yuval & Spence, Crawford & Xu, Ruowen, 2024. "Algorithmic self-referentiality: How machine learning pushes calculative practices to assess themselves," Accounting, Organizations and Society, Elsevier, vol. 113(C).
  • Handle: RePEc:eee:aosoci:v:113:y:2024:i:c:s0361368224000278
    DOI: 10.1016/j.aos.2024.101567
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0361368224000278
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.aos.2024.101567?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. repec:spo:wpmain:info:hdl:2441/6cbt691h0h8o9q5rf0apko0pda is not listed on IDEAS
    2. Isabelle Guérin & Santosh Kumar, 2017. "Market, Freedom and the Illusions of Microcredit. Patronage, Caste, Class and Patriarchy in Rural South India," Journal of Development Studies, Taylor & Francis Journals, vol. 53(5), pages 741-754, May.
    3. Martin Messner, 2009. "The Limits of Accountability," Post-Print hal-00486747, HAL.
    4. repec:hal:spmain:info:hdl:2441/6cbt691h0h8o9q5rf0apko0pda is not listed on IDEAS
    5. Poon, Martha, 2009. "From new deal institutions to capital markets: Commercial consumer risk scores and the making of subprime mortgage finance," Accounting, Organizations and Society, Elsevier, vol. 34(5), pages 654-674, July.
    6. Boedker, Christina & Chong, Kar-Ming & Mouritsen, Jan, 2020. "The counter-performativity of calculative practices: Mobilising rankings of intellectual capital," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 72(C).
    7. Beunza, Daniel & Stark, David, 2012. "From dissonance to resonance: cognitive interdependence in quantitative finance," LSE Research Online Documents on Economics 45604, London School of Economics and Political Science, LSE Library.
    8. Michael McCanless, 2023. "Banking on alternative credit scores: Auditing the calculative infrastructure of U.S. consumer lending," Environment and Planning A, , vol. 55(8), pages 2128-2146, November.
    9. Michel Callon, 2010. "Performativity, Misfires And Politics," Journal of Cultural Economy, Taylor & Francis Journals, vol. 3(2), pages 163-169, July.
    10. Fabian Muniesa, 2014. "The Provoked Economy: Economic Reality and the Performative Turn," Post-Print halshs-00989576, HAL.
    11. George Salijeni & Anna Samsonova-Taddei & Stuart Turley, 2019. "Big Data and changes in audit technology: contemplating a research agenda," Accounting and Business Research, Taylor & Francis Journals, vol. 49(1), pages 95-119, January.
    12. Alan D. Meyer & Vibha Gaba & Kenneth A. Colwell, 2005. "Organizing Far from Equilibrium: Nonlinear Change in Organizational Fields," Organization Science, INFORMS, vol. 16(5), pages 456-473, October.
    13. Mehrpouya, Afshin & Samiolo, Rita, 2016. "Performance measurement in global governance: Ranking and the politics of variability," Accounting, Organizations and Society, Elsevier, vol. 55(C), pages 12-31.
    14. Tim D. Bauer & Cassandra Estep & Bertrand Malsch, 2019. "One Team or Two? Investigating Relationship Quality between Auditors and IT Specialists: Implications for Audit Team Identity and the Audit Process†," Contemporary Accounting Research, John Wiley & Sons, vol. 36(4), pages 2142-2177, December.
    15. Messner, Martin, 2009. "The limits of accountability," Accounting, Organizations and Society, Elsevier, vol. 34(8), pages 918-938, November.
    16. Fabian Muniesa & Dominique Chabert & Marceline Ducrocq-Grondin & Susan V. Scott, 2011. "Back-office intricacy: the description of financial objects in an investment bank," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 20(4), pages 1189-1213, August.
    17. George Salijeni & Anna Samsonova-Taddei & Stuart Turley, 2021. "Understanding How Big Data Technologies Reconfigure the Nature and Organization of Financial Statement Audits: A Sociomaterial Analysis," European Accounting Review, Taylor & Francis Journals, vol. 30(3), pages 531-555, May.
    18. Mikes, Anette, 2011. "From counting risk to making risk count: Boundary-work in risk management," Accounting, Organizations and Society, Elsevier, vol. 36(4), pages 226-245.
    19. Fourcade, Marion & Healy, Kieran, 2013. "Classification situations: Life-chances in the neoliberal era," Accounting, Organizations and Society, Elsevier, vol. 38(8), pages 559-572.
    20. Hall, Matthew & Mikes, Anette & Millo, Yuval, 2015. "How do risk managers become influential?: a field study of toolmaking in two financial institutions," LSE Research Online Documents on Economics 60485, London School of Economics and Political Science, LSE Library.
    21. Martha Poon, 2009. "From New Deal institutions to capital markets: commercial consumer risk scores and the making of subprime mortgage finance," Post-Print halshs-00359712, HAL.
    22. Martha Poon, 2009. "From New Deal institutions to capital markets: commercial consumer risk scores and the making of subprime mortgage finance," Working Papers halshs-00359712, HAL.
    23. Pollock, Neil & D’Adderio, Luciana, 2012. "Give me a two-by-two matrix and I will create the market: Rankings, graphic visualisations and sociomateriality," Accounting, Organizations and Society, Elsevier, vol. 37(8), pages 565-586.
    24. Marion Fourcade & Kieran Healy, 2013. "Classification situations: Life-chances in the neoliberal era," Post-Print hal-03470535, HAL.
    25. Martha Poon, 2009. "From New Deal institutions to capital markets: commercial consumer risk scores and the making of subprime mortgage finance," CSI Working Papers Series 014, Centre de Sociologie de l'Innovation (CSI), Mines ParisTech.
    26. Ronzani, Matteo & Gatzweiler, Marian Konstantin, 2022. "The lure of the visual: Multimodality, simplification, and performance measurement visualizations in a megaproject," Accounting, Organizations and Society, Elsevier, vol. 97(C).
    27. Karen Locke & Karen Golden-Biddle & Martha S. Feldman, 2008. "Perspective---Making Doubt Generative: Rethinking the Role of Doubt in the Research Process," Organization Science, INFORMS, vol. 19(6), pages 907-918, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kornberger, Martin & Pflueger, Dane & Mouritsen, Jan, 2017. "Evaluative infrastructures: Accounting for platform organization," Accounting, Organizations and Society, Elsevier, vol. 60(C), pages 79-95.
    2. Martinez, Daniel E. & Pflueger, Dane & Palermo, Tommaso, 2022. "Accounting and the territorialization of markets: A field study of the Colorado cannabis market," Accounting, Organizations and Society, Elsevier, vol. 102(C).
    3. repec:spo:wpmain:info:hdl:2441/4ff88coju39nk8b11b5ghfc1ff is not listed on IDEAS
    4. Lacan, Laure & Lazarus, Jeanne, 2015. "A relationship and a practice: On the French sociology of credit," MaxPo Discussion Paper Series 15/1, Max Planck Sciences Po Center on Coping with Instability in Market Societies (MaxPo).
    5. Olivier Godechot, 2019. "Conclusion: What finance manufactures," Post-Print hal-03393812, HAL.
    6. Cooper, Christine, 2015. "Entrepreneurs of the self: The development of management control since 1976," Accounting, Organizations and Society, Elsevier, vol. 47(C), pages 14-24.
    7. Kornberger Martin & Pflueger Dane & Mouritsen Jan, 2017. "Evaluative infrastructures : Accounting for platform organization," Post-Print hal-02276737, HAL.
    8. repec:hal:spmain:info:hdl:2441/4ff88coju39nk8b11b5ghfc1ff is not listed on IDEAS
    9. McFall, Liz, 2014. "Devising Consumption: cultural economies of insurance, credit and spending," OSF Preprints at2nv, Center for Open Science.
    10. Arena, Marika & Arnaboldi, Michela & Palermo, Tommaso, 2017. "The dynamics of (dis)integrated risk management: a comparative field study," LSE Research Online Documents on Economics 84285, London School of Economics and Political Science, LSE Library.
    11. Pinzur, David, 2016. "Making the grade: infrastructural semiotics and derivative market outcomes on the Chicago Board of Trade and New Orleans Cotton Exchange, 1856–1909," LSE Research Online Documents on Economics 102988, London School of Economics and Political Science, LSE Library.
    12. Ryan Bubb & Alex Kaufman, 2011. "Securitization and moral hazard: evidence from credit score cutoff rules," Public Policy Discussion Paper 11-6, Federal Reserve Bank of Boston.
    13. Savannah Cox, 2022. "Inscriptions of resilience: Bond ratings and the government of climate risk in Greater Miami, Florida," Environment and Planning A, , vol. 54(2), pages 295-310, March.
    14. Bernardo Bátiz-Lazo, 2017. "Between Novelty and Fashion: Risk Management and the Adoption of Computers in Retail Banking," Palgrave Studies in the History of Finance, in: Korinna Schönhärl (ed.), Decision Taking, Confidence and Risk Management in Banks from Early Modernity to the 20th Century, pages 189-207, Palgrave Macmillan.
    15. Kiviat, Barbara, 2019. "Credit Scoring in the United States," economic sociology. perspectives and conversations, Max Planck Institute for the Study of Societies, vol. 21(1), pages 33-42.
    16. Eglė Jakučionytė & Swapnil Singh, 2023. "Emergence of subprime lending in minority neighborhoods," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 51(6), pages 1547-1583, November.
    17. Alaimo, Cristina & Kallinikos, Jannis, 2022. "Organizations decentered: data objects, technology and knowledge," LSE Research Online Documents on Economics 112470, London School of Economics and Political Science, LSE Library.
    18. repec:spo:wpmain:info:hdl:2441/6cbt691h0h8o9q5rf0apko0pda is not listed on IDEAS
    19. Scott, Susan V., 2010. "Understanding the characteristics of techno-innovation in an era of self-regulated financial services," LSE Research Online Documents on Economics 37867, London School of Economics and Political Science, LSE Library.
    20. Godechot, Olivier, 2015. "Financialization is marketization! A study on the respective impact of various dimensions of financialization on the increase in global inequality," MaxPo Discussion Paper Series 15/3, Max Planck Sciences Po Center on Coping with Instability in Market Societies (MaxPo).
    21. Tordjman, Hélène, 2011. "La crise contemporaine, une crise de la modernité technique," Revue de la Régulation - Capitalisme, institutions, pouvoirs, Association Recherche et Régulation, vol. 10.
    22. Olivier Godechot, 2015. "Financialization Is Marketization!," SciencePo Working papers Main hal-03459520, HAL.
    23. Lei Ding & Jackelyn Hwang, 2016. "The Consequences of Gentrification: A Focus on Residents’ Financial Health in Philadelphia," Working Papers 16-22, Federal Reserve Bank of Philadelphia.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:aosoci:v:113:y:2024:i:c:s0361368224000278. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/aos .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.