IDEAS home Printed from https://ideas.repec.org/a/inm/ormoor/v47y2022i1p120-152.html

Mean Field Analysis of Deep Neural Networks

Author

Listed:
  • Justin Sirignano

    (Mathematical Institute, University of Oxford, Oxford OX2 6GG, United Kingdom)

  • Konstantinos Spiliopoulos

    (Department of Mathematics and Statistics, Boston University, Boston, Massachusetts 02215)

Abstract

We analyze multilayer neural networks in the asymptotic regime of simultaneously (a) large network sizes and (b) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multilayer neural network output. The limit procedure is valid for any number of hidden layers, and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function).

Suggested Citation

  • Justin Sirignano & Konstantinos Spiliopoulos, 2022. "Mean Field Analysis of Deep Neural Networks," Mathematics of Operations Research, INFORMS, vol. 47(1), pages 120-152, February.
    • Handle: RePEc:inm:ormoor:v:47:y:2022:i:1:p:120-152
    DOI: 10.1287/moor.2020.1118
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/moor.2020.1118
    Download Restriction: no
    File URL: https://libkey.io/10.1287/moor.2020.1118?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Justin Sirignano & Rama Cont, 2018. "Universal features of price formation in financial markets: perspectives from Deep Learning," Papers 1803.06917, arXiv.org.
    2. Justin Sirignano & Rama Cont, 2018. "Universal features of price formation in financial markets: perspectives from Deep Learning," Working Papers hal-01754054, HAL.
    3. Justin Sirignano & Konstantinos Spiliopoulos, 2017. "DGM: A deep learning algorithm for solving partial differential equations," Papers 1708.07469, arXiv.org, revised Sep 2018.
    4. Sirignano, Justin & Spiliopoulos, Konstantinos, 2020. "Mean field analysis of neural networks: A central limit theorem," Stochastic Processes and their Applications, Elsevier, vol. 130(3), pages 1820-1852.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ivan Peñaloza & Pablo Padilla, 2022. "A Pricing Method in a Constrained Market with Differential Informational Frameworks," Computational Economics, Springer;Society for Computational Economics, vol. 60(3), pages 1055-1100, October.
    2. Ymir Makinen & Juho Kanniainen & Moncef Gabbouj & Alexandros Iosifidis, 2018. "Forecasting of Jump Arrivals in Stock Prices: New Attention-based Network Architecture using Limit Order Book Data," Papers 1810.10845, arXiv.org.
    3. Sebastian Jaimungal, 2022. "Reinforcement learning and stochastic optimisation," Finance and Stochastics, Springer, vol. 26(1), pages 103-129, January.
    4. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2018. "DeepLOB: Deep Convolutional Neural Networks for Limit Order Books," Papers 1808.03668, arXiv.org, revised Jan 2020.
    5. Eduard Silantyev, 2019. "Order flow analysis of cryptocurrency markets," Digital Finance, Springer, vol. 1(1), pages 191-218, November.
    6. Christa Cuchiero & Wahid Khosrawi & Josef Teichmann, 2020. "A Generative Adversarial Network Approach to Calibration of Local Stochastic Volatility Models," Risks, MDPI, vol. 8(4), pages 1-31, September.
    7. Omer Berat Sezer & Mehmet Ugur Gudelek & Ahmet Murat Ozbayoglu, 2019. "Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019," Papers 1911.13288, arXiv.org.
    8. Takuya Shintate & Lukáš Pichl, 2019. "Trend Prediction Classification for High Frequency Bitcoin Time Series with Deep Learning," JRFM, MDPI, vol. 12(1), pages 1-15, January.
    9. Qi Zhao, 2020. "A Deep Learning Framework for Predicting Digital Asset Price Movement from Trade-by-trade Data," Papers 2010.07404, arXiv.org.
    10. Baptiste Barreau & Laurent Carlier & Damien Challet, 2019. "Deep Prediction of Investor Interest: a Supervised Clustering Approach," Papers 1909.05289, arXiv.org, revised Feb 2021.
    11. Bryan Lim & Stefan Zohren & Stephen Roberts, 2019. "Enhancing Time Series Momentum Strategies Using Deep Neural Networks," Papers 1904.04912, arXiv.org, revised Sep 2020.
    12. Paul Bilokon & Yitao Qiu, 2023. "Transformers versus LSTMs for electronic trading," Papers 2309.11400, arXiv.org.
    13. Kieran Wood & Samuel Kessler & Stephen J. Roberts & Stefan Zohren, 2023. "Few-Shot Learning Patterns in Financial Time-Series for Trend-Following Strategies," Papers 2310.10500, arXiv.org, revised Mar 2024.
    14. Dimitrios Nikolaidis & Michalis Doumpos, 2022. "Credit Scoring with Drift Adaptation Using Local Regions of Competence," SN Operations Research Forum, Springer, vol. 3(4), pages 1-28, December.
    15. Arthur le Calvez & Dave Cliff, 2018. "Deep Learning can Replicate Adaptive Traders in a Limit-Order-Book Financial Market," Papers 1811.02880, arXiv.org.
    16. Allison Koenecke & Amita Gajewar, 2019. "Curriculum Learning in Deep Neural Networks for Financial Forecasting," Papers 1904.12887, arXiv.org, revised Jul 2019.
    17. Hyungjun Park & Min Kyu Sim & Dong Gu Choi, 2019. "An intelligent financial portfolio trading strategy using deep Q-learning," Papers 1907.03665, arXiv.org, revised Nov 2019.
    18. Ymir Mäkinen & Juho Kanniainen & Moncef Gabbouj & Alexandros Iosifidis, 2019. "Forecasting jump arrivals in stock prices: new attention-based network architecture using limit order book data," Quantitative Finance, Taylor & Francis Journals, vol. 19(12), pages 2033-2050, December.
    19. Parley Ruogu Yang, 2021. "Forecasting high-frequency financial time series: an adaptive learning approach with the order book data," Papers 2103.00264, arXiv.org.
    20. Konark Jain & Nick Firoozye & Jonathan Kochems & Philip Treleaven, 2024. "Limit Order Book Simulations: A Review," Papers 2402.17359, arXiv.org, revised Mar 2024.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormoor:v:47:y:2022:i:1:p:120-152. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.