IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0189326.html
   My bibliography  Save this article

Do neural nets learn statistical laws behind natural language?

Author

Listed:
  • Shuntaro Takahashi
  • Kumiko Tanaka-Ishii

Abstract

The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf’s law and Heaps’ law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf’s law and Heaps’ law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.

Suggested Citation

  • Shuntaro Takahashi & Kumiko Tanaka-Ishii, 2017. "Do neural nets learn statistical laws behind natural language?," PLOS ONE, Public Library of Science, vol. 12(12), pages 1-17, December.
    • Handle: RePEc:plo:pone00:0189326
    DOI: 10.1371/journal.pone.0189326
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189326
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0189326&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0189326?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ricardo Baeza–Yates & Gonzalo Navarro, 2000. "Block addressing indices for approximate text retrieval," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(1), pages 69-82.
    2. Santhanam, M.S. & Kantz, Holger, 2005. "Long-range correlations and rare events in boundary layer wind fields," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 345(3), pages 713-721.
    3. Eduardo G Altmann & Janet B Pierrehumbert & Adilson E Motter, 2009. "Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words," PLOS ONE, Public Library of Science, vol. 4(11), pages 1-7, November.
    4. Ebeling, Werner & Neiman, Alexander, 1995. "Long-range correlations between letters and sentences in texts," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 215(3), pages 233-241.
    5. Kumiko Tanaka-Ishii & Armin Bunde, 2016. "Long-Range Memory in Literary Texts: On the Universal Clustering of the Rare Words," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kumiko Tanaka-Ishii & Armin Bunde, 2016. "Long-Range Memory in Literary Texts: On the Universal Clustering of the Rare Words," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    2. Heng Chen & Haitao Liu, 2018. "Quantifying Evolution of Short and Long-Range Correlations in Chinese Narrative Texts across 2000 Years," Complexity, Hindawi, vol. 2018, pages 1-12, February.
    3. Vieira, Denner S. & Picoli, Sergio & Mendes, Renio S., 2018. "Robustness of sentence length measures in written texts," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 506(C), pages 749-754.
    4. Corrêa, Edilson A. & Marinho, Vanessa Q. & Amancio, Diego R., 2020. "Semantic flow in language networks discriminates texts by genre and publication date," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 557(C).
    5. Sen, Rituparna & Hsieh, Fushing, 2009. "A note on testing regime switching assumption based on recurrence times," Statistics & Probability Letters, Elsevier, vol. 79(24), pages 2443-2450, December.
    6. Eduardo G Altmann & Janet B Pierrehumbert & Adilson E Motter, 2011. "Niche as a Determinant of Word Fate in Online Groups," PLOS ONE, Public Library of Science, vol. 6(5), pages 1-12, May.
    7. Edoardo Magnone, 2014. "A novel graphical representation of sentence complexity: the description and its application," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1301-1329, February.
    8. Yue Yang & Changgui Gu & Qin Xiao & Huijie Yang, 2017. "Evolution of scaling behaviors embedded in sentence series from A Story of the Stone," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-14, February.
    9. Shan Li & Ruokuang Lin & Chunhua Bian & Qianli D Y Ma & Plamen Ch Ivanov, 2016. "Model of the Dynamic Construction Process of Texts and Scaling Laws of Words Organization in Language Systems," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-12, December.
    10. Chen, Yanguang, 2012. "Zipf’s law, 1/f noise, and fractal hierarchy," Chaos, Solitons & Fractals, Elsevier, vol. 45(1), pages 63-73.
    11. Zörnig, Peter, 2010. "Statistical simulation and the distribution of distances between identical elements in a random sequence," Computational Statistics & Data Analysis, Elsevier, vol. 54(10), pages 2317-2327, October.
    12. Yuan, Qianshun & Semba, Sherehe & Zhang, Jing & Weng, Tongfeng & Gu, Changgui & Yang, Huijie, 2021. "Multi-scale transition matrix approach to time series," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 578(C).
    13. Pyko, Nikita S. & Pyko, Svetlana A. & Markelov, Oleg A. & Karimov, Artur I. & Butusov, Denis N. & Zolotukhin, Yaroslav V. & Uljanitski, Yuri D. & Bogachev, Mikhail I., 2018. "Assessment of cooperativity in complex systems with non-periodical dynamics: Comparison of five mutual information metrics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 503(C), pages 1054-1072.
    14. Espitia, Diego & Larralde, Hernán, 2020. "Universal and non-universal text statistics: Clustering coefficient for language identification," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 553(C).
    15. Cui, Xue-Mei & Yoon, Chang No & Youn, Hyejin & Lee, Sang Hoon & Jung, Jean S. & Han, Seung Kee, 2017. "Dynamic burstiness of word-occurrence and network modularity in textbook systems," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 487(C), pages 103-110.
    16. Karain, Wael I., 2019. "Investigating large-amplitude protein loop motions as extreme events using recurrence interval analysis," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 520(C), pages 1-10.
    17. Ghosh, Dipak & Chakraborty, Sayantan & Samanta, Shukla, 2019. "Study of translational effect in Tagore’s Gitanjali using Chaos based Multifractal analysis technique," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 523(C), pages 1343-1354.
    18. Frank Emmert-Streib, 2013. "Structural Properties and Complexity of a New Network Class: Collatz Step Graphs," PLOS ONE, Public Library of Science, vol. 8(2), pages 1-14, February.
    19. Rashidisabet, Homa & Ajilore, Olusola & Leow, Alex & Demos, Alexander P., 2022. "Revisiting power-law estimation with applications to real-world human typing dynamics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 599(C).
    20. Criado-Alonso, Ángeles & Aleja, David & Romance, Miguel & Criado, Regino, 2022. "Derivative of a hypergraph as a tool for linguistic pattern analysis," Chaos, Solitons & Fractals, Elsevier, vol. 163(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0189326. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.