IDEAS home Printed from https://ideas.repec.org/a/eee/spapps/v130y2020i3p1820-1852.html
   My bibliography  Save this article

Mean field analysis of neural networks: A central limit theorem

Author

Listed:
  • Sirignano, Justin
  • Spiliopoulos, Konstantinos

Abstract

We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural network’s fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation. The proof relies upon weak convergence methods from stochastic analysis. In particular, we prove relative compactness for the sequence of processes and uniqueness of the limiting process in a suitable Sobolev space.

Suggested Citation

  • Sirignano, Justin & Spiliopoulos, Konstantinos, 2020. "Mean field analysis of neural networks: A central limit theorem," Stochastic Processes and their Applications, Elsevier, vol. 130(3), pages 1820-1852.
    • Handle: RePEc:eee:spapps:v:130:y:2020:i:3:p:1820-1852
    DOI: 10.1016/j.spa.2019.06.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304414918306197
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.spa.2019.06.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Delarue, F. & Inglis, J. & Rubenthaler, S. & Tanré, E., 2015. "Particle systems with a singular mean-field self-excitation. Application to neuronal networks," Stochastic Processes and their Applications, Elsevier, vol. 125(6), pages 2451-2492.
    2. Dai Pra, Paolo & Tolotti, Marco, 2009. "Heterogeneous credit portfolios and the dynamics of the aggregate losses," Stochastic Processes and their Applications, Elsevier, vol. 119(9), pages 2913-2944, September.
    3. Kay Giesecke & Konstantinos Spiliopoulos & Richard B. Sowers & Justin A. Sirignano, 2015. "Large Portfolio Asymptotics For Loss From Default," Mathematical Finance, Wiley Blackwell, vol. 25(1), pages 77-114, January.
    4. Fernandez, Begoña & Méléard, Sylvie, 1997. "A Hilbertian approach for fluctuations on the McKean-Vlasov model," Stochastic Processes and their Applications, Elsevier, vol. 71(1), pages 33-53, October.
    5. Kay Giesecke & Konstantinos Spiliopoulos & Richard B. Sowers & Justin A. Sirignano, 2011. "Large Portfolio Asymptotics for Loss From Default," Papers 1109.1272, arXiv.org, revised Feb 2015.
    6. Paolo Dai Pra & Wolfgang J. Runggaldier & Elena Sartori & Marco Tolotti, 2007. "Large portfolio losses: A dynamic contagion model," Papers 0704.1348, arXiv.org, revised Mar 2009.
    7. Kay Giesecke & Konstantinos Spiliopoulos & Richard B. Sowers, 2011. "Default clustering in large portfolios: Typical events," Papers 1104.1773, arXiv.org, revised Feb 2013.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sebastian Jaimungal, 2022. "Reinforcement learning and stochastic optimisation," Finance and Stochastics, Springer, vol. 26(1), pages 103-129, January.
    2. Hirsch, Christian & Neumann, Matthias & Schmidt, Volker, 2023. "Asymptotic properties of one-layer artificial neural networks with sparse connectivity," Statistics & Probability Letters, Elsevier, vol. 193(C).
    3. Ruixiang Zhang & Ziyu Zhu & Meng Yuan & Yihan Guo & Jie Song & Xuanxuan Shi & Yu Wang & Yaojie Sun, 2023. "Regional Residential Short-Term Load-Interval Forecasting Based on SSA-LSTM and Load Consumption Consistency Analysis," Energies, MDPI, vol. 16(24), pages 1-17, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Konstantinos Spiliopoulos & Jia Yang, 2018. "Network effects in default clustering for large systems," Papers 1812.07645, arXiv.org, revised Feb 2020.
    2. Ben Hambly & Nikolaos Kolliopoulos, 2020. "Fast mean-reversion asymptotics for large portfolios of stochastic volatility models," Finance and Stochastics, Springer, vol. 24(3), pages 757-794, July.
    3. Justin Sirignano & Kay Giesecke, 2019. "Risk Analysis for Large Pools of Loans," Management Science, INFORMS, vol. 65(1), pages 107-121, January.
    4. Tang, Qihe & Tong, Zhiwei & Yang, Yang, 2021. "Large portfolio losses in a turbulent market," European Journal of Operational Research, Elsevier, vol. 292(2), pages 755-769.
    5. Xiaowei Zhang & Jose Blanchet & Kay Giesecke & Peter W. Glynn, 2015. "Affine Point Processes: Approximation and Efficient Simulation," Mathematics of Operations Research, INFORMS, vol. 40(4), pages 797-819, October.
    6. Ben Hambly & Andreas Sojmark, 2018. "An SPDE Model for Systemic Risk with Endogenous Contagion," Papers 1801.10088, arXiv.org, revised Sep 2018.
    7. Tang, Qihe & Tang, Zhaofeng & Yang, Yang, 2019. "Sharp asymptotics for large portfolio losses under extreme risks," European Journal of Operational Research, Elsevier, vol. 276(2), pages 710-722.
    8. Josselin Garnier & George Papanicolaou & Tzu-Wei Yang, 2015. "A risk analysis for a system stabilized by a central agent," Papers 1507.08333, arXiv.org, revised Aug 2015.
    9. Ben Hambly & Andreas Søjmark, 2019. "An SPDE model for systemic risk with endogenous contagion," Finance and Stochastics, Springer, vol. 23(3), pages 535-594, July.
    10. Agostino Capponi & Xu Sun & David D. Yao, 2020. "A Dynamic Network Model of Interbank Lending—Systemic Risk and Liquidity Provisioning," Mathematics of Operations Research, INFORMS, vol. 45(3), pages 1127-1152, August.
    11. Jean-David Fermanian, 2020. "On the Dependence between Default Risk and Recovery Rates in Structural Models," Annals of Economics and Statistics, GENES, issue 140, pages 45-82.
    12. Egami, M. & Kevkhishvili, R., 2017. "An analysis of simultaneous company defaults using a shot noise process," Journal of Banking & Finance, Elsevier, vol. 80(C), pages 135-161.
    13. Burzoni, Matteo & Campi, Luciano, 2023. "Mean field games with absorption and common noise with a model of bank run," Stochastic Processes and their Applications, Elsevier, vol. 164(C), pages 206-241.
    14. Erhan Bayraktar & Gaoyue Guo & Wenpin Tang & Yuming Paul Zhang, 2022. "Systemic robustness: a mean-field particle system approach," Papers 2212.08518, arXiv.org, revised Aug 2023.
    15. Frikha, Noufel & Li, Libo, 2021. "Well-posedness and approximation of some one-dimensional Lévy-driven non-linear SDEs," Stochastic Processes and their Applications, Elsevier, vol. 132(C), pages 76-107.
    16. Gao, Fuqing & Zhu, Lingjiong, 2018. "Some asymptotic results for nonlinear Hawkes processes," Stochastic Processes and their Applications, Elsevier, vol. 128(12), pages 4051-4077.
    17. Sergey Nadtochiy & Mykhaylo Shkolnikov, 2017. "Particle systems with singular interaction through hitting times: application in systemic risk modeling," Papers 1705.00691, arXiv.org.
    18. Konstantinos Spiliopoulos & Richard B. Sowers, 2013. "Default Clustering in Large Pools: Large Deviations," Papers 1311.0498, arXiv.org, revised Feb 2015.
    19. Konstantinos Spiliopoulos, 2014. "Systemic Risk and Default Clustering for Large Financial Systems," Papers 1402.5352, arXiv.org, revised Feb 2015.
    20. Zachary Feinstein & Andreas Sojmark, 2019. "A Dynamic Default Contagion Model: From Eisenberg-Noe to the Mean Field," Papers 1912.08695, arXiv.org.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:spapps:v:130:y:2020:i:3:p:1820-1852. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/505572/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.