IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v553y2020ics0378437119321673.html
   My bibliography  Save this article

Universal and non-universal text statistics: Clustering coefficient for language identification

Author

Listed:
  • Espitia, Diego
  • Larralde, Hernán

Abstract

In this work we analyze statistical properties of 91 relatively small texts in 7 different languages (Spanish, English, French, German, Turkish, Russian, Icelandic) as well as texts with randomly inserted spaces. Despite the size (around 11260 different words), the well known universal statistical laws – namely Zipf and Herdan–Heap’s laws – are confirmed, and are in close agreement with results obtained elsewhere. We also construct a word co-occurrence network of each text. While the degree distribution is again universal, we note that the distribution of clustering coefficients, which depend strongly on the local structure of networks, can be used to differentiate between languages, as well as to distinguish natural languages from random texts.

Suggested Citation

  • Espitia, Diego & Larralde, Hernán, 2020. "Universal and non-universal text statistics: Clustering coefficient for language identification," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 553(C).
    • Handle: RePEc:eee:phsmap:v:553:y:2020:i:c:s0378437119321673
    DOI: 10.1016/j.physa.2019.123905
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437119321673
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000
    File URL: https://libkey.io/10.1016/j.physa.2019.123905?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marcelo A Montemurro & Damián H Zanette, 2013. "Keywords and Co-Occurrence Patterns in the Voynich Manuscript: An Information-Theoretic Analysis," PLOS ONE, Public Library of Science, vol. 8(6), pages 1-9, June.
    2. Ricardo Baeza–Yates & Gonzalo Navarro, 2000. "Block addressing indices for approximate text retrieval," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(1), pages 69-82.
    3. Petroni, Filippo & Serva, Maurizio, 2010. "Measures of lexical distance between languages," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 389(11), pages 2280-2283.
    4. Ramon Ferrer-i-Cancho & Brita Elvevåg, 2010. "Random Texts Do Not Exhibit the Real Zipf's Law-Like Rank Distribution," PLOS ONE, Public Library of Science, vol. 5(3), pages 1-10, March.
    5. Camilo Akimushkin & Diego Raphael Amancio & Osvaldo Novais Oliveira Jr., 2017. "Text Authorship Identified Using the Dynamics of Word Co-Occurrence Networks," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-15, January.
    6. Gamallo, Pablo & Pichel, José Ramom & Alegria, Iñaki, 2017. "From language identification to language distance," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 484(C), pages 152-162.
    7. Leo Egghe, 2007. "Untangling Herdan's law and Heaps' law: Mathematical and informetric arguments," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(5), pages 702-709, March.
    8. Naumis, G.G. & Cocho, G., 2008. "Tail universalities in rank distributions as an algebraic problem: The beta-like function," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(1), pages 84-96.
    9. Mark Newman, 2000. "The power of design," Nature, Nature, vol. 405(6785), pages 412-413, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mehri, Ali & Jamaati, Maryam, 2021. "Statistical metrics for languages classification: A case study of the Bible translations," Chaos, Solitons & Fractals, Elsevier, vol. 144(C).
    2. Li, Wentian, 2012. "Fitting Chinese syllable-to-character mapping spectrum by the beta rank function," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 391(4), pages 1515-1518.
    3. Ingo Eduard Isphording & Sebastian Otten, 2013. "The Costs of Babylon—Linguistic Distance in Applied Economics," Review of International Economics, Wiley Blackwell, vol. 21(2), pages 354-369, May.
    4. repec:zbw:rwirep:0337 is not listed on IDEAS
    5. Isphording, Ingo E. & Piopiunik, Marc & Rodríguez-Planas, Núria, 2016. "Speaking in numbers: The effect of reading performance on math performance among immigrants," Economics Letters, Elsevier, vol. 139(C), pages 52-56.
    6. Shakibian, Hadi & Charkari, Nasrollah Moghadam, 2018. "Statistical similarity measures for link prediction in heterogeneous complex networks," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 501(C), pages 248-263.
    7. Anna Tietze & Philip Hofmann, 2019. "The h-index and multi-author hm-index for individual researchers in condensed matter physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 171-185, April.
    8. Valerio Ficcadenti & Roy Cerqueti & Ciro Hosseini Varde’i, 2023. "A rank-size approach to analyse soccer competitions and teams: the case of the Italian football league “Serie A"," Annals of Operations Research, Springer, vol. 325(1), pages 85-113, June.
    9. Wei, Shelia X. & Tong, Tong & Rousseau, Ronald & Wang, Wanru & Ye, Fred Y., 2022. "Relations among the h-, g-, ψ-, and p-index and offset-ability," Journal of Informetrics, Elsevier, vol. 16(4).
    10. repec:old:wpaper:352 is not listed on IDEAS
    11. Ausloos, Marcel & Cerqueti, Roy & Lupi, Claudio, 2017. "Long-range properties and data validity for hydrogeological time series: The case of the Paglia river," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 470(C), pages 39-50.
    12. Liu, Yanyan & Li, Keping & Yan, Dongyang & Gu, Shuang, 2022. "A network-based CNN model to identify the hidden information in text data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 590(C).
    13. Nazim Choudhury & Shahadat Uddin, 2016. "Time-aware link prediction to explore network effects on temporal knowledge evolution," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 745-776, August.
    14. Cerqueti, Roy & Lupi, Claudio & Pietrovito, Filomena & Pozzolo, Alberto Franco, 2022. "Rank–size distributions for banks: A cross-country analysis," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 585(C).
    15. Ibrahim Bousmah & Gilles Grenier & David M. Gray, 2021. "Linguistic Distance, Languages of Work and Wages of Immigrants in Montreal," Journal of Labor Research, Springer, vol. 42(1), pages 1-28, March.
    16. Erkan Gören, 2013. "Economic Effects of Domestic and Neighbouring Countries' Cultural Diversity," ZenTra Working Papers in Transnational Studies 16 / 2013, ZenTra - Center for Transnational Studies, revised Apr 2013.
    17. Joseph, Simmi Marina & Citraro, Salvatore & Morini, Virginia & Rossetti, Giulio & Stella, Massimo, 2023. "Cognitive network neighborhoods quantify feelings expressed in suicide notes and Reddit mental health communities," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 610(C).
    18. Isphording, Ingo E. & Otten, Sebastian, 2014. "Linguistic barriers in the destination language acquisition of immigrants," Journal of Economic Behavior & Organization, Elsevier, vol. 105(C), pages 30-50.
    19. Shuntaro Takahashi & Kumiko Tanaka-Ishii, 2017. "Do neural nets learn statistical laws behind natural language?," PLOS ONE, Public Library of Science, vol. 12(12), pages 1-17, December.
    20. Samuel Zanferdini Oliva & Livia Oliveira-Ciabati & Denise Gazotto Dezembro & Mário Sérgio Adolfi Júnior & Maísa Carvalho Silva & Hugo Cesar Pessotti & Juliana Tarossi Pollettini, 2021. "Text structuring methods based on complex network: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1471-1493, February.
    21. Lorraine Wong, 2023. "The effect of linguistic proximity on the labour market outcomes of the asylum population," Journal of Population Economics, Springer;European Society for Population Economics, vol. 36(2), pages 609-652, April.
    22. Reed, William J., 2001. "The Pareto, Zipf and other power laws," Economics Letters, Elsevier, vol. 74(1), pages 15-19, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:553:y:2020:i:c:s0378437119321673. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.