IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2004.13135.html
   My bibliography  Save this paper

Local Lipschitz Bounds of Deep Neural Networks

Author

Listed:
  • Calypso Herrera
  • Florian Krach
  • Josef Teichmann

Abstract

The Lipschitz constant is an important quantity that arises in analysing the convergence of gradient-based optimization methods. It is generally unclear how to estimate the Lipschitz constant of a complex model. Thus, this paper studies an important problem that may be useful to the broader area of non-convex optimization. The main result provides a local upper bound on the Lipschitz constants of a multi-layer feed-forward neural network and its gradient. Moreover, lower bounds are established as well, which are used to show that it is impossible to derive global upper bounds for the Lipschitz constants. In contrast to previous works, we compute the Lipschitz constants with respect to the network parameters and not with respect to the inputs. These constants are needed for the theoretical description of many step size schedulers of gradient based optimization schemes and their convergence analysis. The idea is both simple and effective. The results are extended to a generalization of neural networks, continuously deep neural networks, which are described by controlled ODEs.

Suggested Citation

  • Calypso Herrera & Florian Krach & Josef Teichmann, 2020. "Local Lipschitz Bounds of Deep Neural Networks," Papers 2004.13135, arXiv.org, revised Feb 2023.
    • Handle: RePEc:arx:papers:2004.13135
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2004.13135
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Christa Cuchiero & Wahid Khosrawi & Josef Teichmann, 2020. "A Generative Adversarial Network Approach to Calibration of Local Stochastic Volatility Models," Risks, MDPI, vol. 8(4), pages 1-31, September.
    2. Christa Cuchiero & Wahid Khosrawi & Josef Teichmann, 2020. "A generative adversarial network approach to calibration of local stochastic volatility models," Papers 2005.02505, arXiv.org, revised Sep 2020.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Calypso Herrera & Florian Krach & Pierre Ruyssen & Josef Teichmann, 2021. "Optimal Stopping via Randomized Neural Networks," Papers 2104.13669, arXiv.org, revised Dec 2023.
    2. Calypso Herrera & Florian Krach & Josef Teichmann, 2020. "Neural Jump Ordinary Differential Equations: Consistent Continuous-Time Prediction and Filtering," Papers 2006.04727, arXiv.org, revised Apr 2021.
    3. Calypso Herrera & Florian Krach & Anastasis Kratsios & Pierre Ruyssen & Josef Teichmann, 2020. "Denise: Deep Robust Principal Component Analysis for Positive Semidefinite Matrices," Papers 2004.13612, arXiv.org, revised Jun 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vedant Choudhary & Sebastian Jaimungal & Maxime Bergeron, 2023. "FuNVol: A Multi-Asset Implied Volatility Market Simulator using Functional Principal Components and Neural SDEs," Papers 2303.00859, arXiv.org, revised Dec 2023.
    2. Blanka Horvath & Josef Teichmann & Žan Žurič, 2021. "Deep Hedging under Rough Volatility," Risks, MDPI, vol. 9(7), pages 1-20, July.
    3. Hans Buhler & Blanka Horvath & Terry Lyons & Imanol Perez Arribas & Ben Wood, 2020. "A Data-driven Market Simulator for Small Data Environments," Papers 2006.14498, arXiv.org.
    4. Sebastian Jaimungal, 2022. "Reinforcement learning and stochastic optimisation," Finance and Stochastics, Springer, vol. 26(1), pages 103-129, January.
    5. Christa Cuchiero & Luca Di Persio & Francesco Guida & Sara Svaluto-Ferro, 2022. "Measure-valued processes for energy markets," Papers 2210.09331, arXiv.org.
    6. Francesca Biagini & Lukas Gonon & Thomas Reitsam, 2021. "Neural network approximation for superhedging prices," Papers 2107.14113, arXiv.org.
    7. Nelson Vadori, 2022. "Calibration of Derivative Pricing Models: a Multi-Agent Reinforcement Learning Perspective," Papers 2203.06865, arXiv.org, revised Oct 2023.
    8. Blanka Horvath & Josef Teichmann & Zan Zuric, 2021. "Deep Hedging under Rough Volatility," Papers 2102.01962, arXiv.org.
    9. Fred Espen Benth & Nils Detering & Silvia Lavagnini, 2021. "Accuracy of deep learning in calibrating HJM forward curves," Digital Finance, Springer, vol. 3(3), pages 209-248, December.
    10. Fred Espen Benth & Nils Detering & Silvia Lavagnini, 2020. "Accuracy of Deep Learning in Calibrating HJM Forward Curves," Papers 2006.01911, arXiv.org, revised May 2021.
    11. Patryk Gierjatowicz & Marc Sabate-Vidales & David v{S}iv{s}ka & Lukasz Szpruch & v{Z}an v{Z}uriv{c}, 2020. "Robust pricing and hedging via neural SDEs," Papers 2007.04154, arXiv.org.
    12. Ariel Neufeld & Philipp Schmocker, 2022. "Chaotic Hedging with Iterated Integrals and Neural Networks," Papers 2209.10166, arXiv.org, revised Feb 2023.
    13. Christa Cuchiero & Philipp Schmocker & Josef Teichmann, 2023. "Global universal approximation of functional input maps on weighted spaces," Papers 2306.03303, arXiv.org, revised Feb 2024.
    14. Beatrice Acciaio & Anastasis Kratsios & Gudmund Pammer, 2022. "Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer," Papers 2201.13094, arXiv.org, revised Mar 2023.
    15. Lukas Gonon, 2022. "Deep neural network expressivity for optimal stopping problems," Papers 2210.10443, arXiv.org.
    16. Jorino van Rhijn & Cornelis W. Oosterlee & Lech A. Grzelak & Shuaiqiang Liu, 2021. "Monte Carlo Simulation of SDEs using GANs," Papers 2104.01437, arXiv.org.
    17. Christa Cuchiero & Sara Svaluto-Ferro & Josef Teichmann, 2023. "Signature SDEs from an affine and polynomial perspective," Papers 2302.01362, arXiv.org.
    18. Haoyang Cao & Xin Guo, 2021. "Generative Adversarial Network: Some Analytical Perspectives," Papers 2104.12210, arXiv.org, revised Sep 2021.
    19. Laurens Van Mieghem & Antonis Papapantoleon & Jonas Papazoglou-Hennig, 2023. "Machine learning for option pricing: an empirical investigation of network architectures," Papers 2307.07657, arXiv.org.
    20. Eva Lutkebohmert & Thorsten Schmidt & Julian Sester, 2021. "Robust deep hedging," Papers 2106.10024, arXiv.org, revised Nov 2021.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2004.13135. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.