IDEAS home Printed from https://ideas.repec.org/a/spr/jcsosc/v8y2025i3d10.1007_s42001-025-00406-7.html
   My bibliography  Save this article

Enhancing tokenization accuracy with dynamic patterns: cumulative logic for segmenting user-generated content in logographic languages

Author

Listed:
  • Yin Zhang

    (Hong Kong Baptist University)

  • Zhihuai Lin

    (University of North Carolina at Chapel Hill)

  • Castiel Chi-chiu Tong

    (Hong Kong University of Science and Technology)

  • Sam Wai-yeung Ho

    (Fasta.ai Ltd.)

Abstract

Despite the significant advancements of Large Language Models (LLMs) in recent years, tokenization remains a critical step in Natural Language Processing (NLP) for social scientific research. This study presents a simple but effective approach to enhance tokenization accuracy in segmenting user-generated content (UGC) in logographic languages, such as Chinese. Existing tokenization techniques often struggle to effectively handle the complexities of UGC on digital platforms, which include informal language, slang, and newly coined terms. To address this challenge, we developed a dynamic tokenization model that incorporates cumulative logic to recognize and adapt to evolving linguistic patterns in social media content. By analyzing large online discussion datasets from LIHKG, a Reddit-like forum in Hong Kong, the model’s effectiveness is demonstrated through its ability to accurately segment domain-specific terms and novel expressions over time. Our results show that the model outperforms traditional tokenizers in recognizing contextually relevant tokens. This innovative approach offers practical advantages for analyzing large-scale UGC data, and has the potential to improve the performance of downstream NLP tasks.

Suggested Citation

  • Yin Zhang & Zhihuai Lin & Castiel Chi-chiu Tong & Sam Wai-yeung Ho, 2025. "Enhancing tokenization accuracy with dynamic patterns: cumulative logic for segmenting user-generated content in logographic languages," Journal of Computational Social Science, Springer, vol. 8(3), pages 1-24, August.
    • Handle: RePEc:spr:jcsosc:v:8:y:2025:i:3:d:10.1007_s42001-025-00406-7
    DOI: 10.1007/s42001-025-00406-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s42001-025-00406-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s42001-025-00406-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Kawaljeet Kaur Kapoor & Kuttimani Tamilmani & Nripendra P. Rana & Pushp Patil & Yogesh K. Dwivedi & Sridhar Nerur, 2018. "Advances in Social Media Research: Past, Present and Future," Information Systems Frontiers, Springer, vol. 20(3), pages 531-558, June.
    2. Kaplan, Andreas M. & Haenlein, Michael, 2010. "Users of the world, unite! The challenges and opportunities of Social Media," Business Horizons, Elsevier, vol. 53(1), pages 59-68, January.
    3. Yunya Song & Jonathon P. Schuldt & Yin Zhang & Ziwei Wang & Y. Connie Yuan & Sheng Zou & Jun Li, 2025. "Bridging ideologies: analyzing the use of moral language and framing in social media discourse on climate change by U.S. congress members through computational approaches," Climatic Change, Springer, vol. 178(3), pages 1-20, March.
    4. Grimmer, Justin & Stewart, Brandon M., 2013. "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts," Political Analysis, Cambridge University Press, vol. 21(3), pages 267-297, July.
    5. Denny, Matthew J. & Spirling, Arthur, 2018. "Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It," Political Analysis, Cambridge University Press, vol. 26(2), pages 168-189, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stefano Di Lauro & Aizhan Tursunbayeva & Gilda Antonelli & Marcello Martinez, 2021. "Organizational and Corporate Identity on Social Media: A Literature Review," International Journal of Business and Management, Canadian Center of Science and Education, vol. 15(4), pages 1-53, July.
    2. Irina Maiorescu & Mihaela Bucur & Bogdan Georgescu & Daniel Moise & Vasile Alecsandru Strat & Ion Daniel Zgură, 2020. "Social Media and IOT Wearables in Developing Marketing Strategies. Do SMEs Differ From Large Enterprises?," Sustainability, MDPI, vol. 12(18), pages 1-18, September.
    3. Mohamed M. Mostafa, 2023. "A one-hundred-year structural topic modeling analysis of the knowledge structure of international management research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3905-3935, August.
    4. Raed S. Algharabat & Nripendra P. Rana, 0. "Social Commerce in Emerging Markets and its Impact on Online Community Engagement," Information Systems Frontiers, Springer, vol. 0, pages 1-22.
    5. Purwoko Haryadi Santoso & Edi Istiyono & Haryanto & Wahyu Hidayatulloh, 2022. "Thematic Analysis of Indonesian Physics Education Research Literature Using Machine Learning," Data, MDPI, vol. 7(11), pages 1-41, October.
    6. Camilla Salvatore & Silvia Biffignandi & Annamaria Bianchi, 2022. "Corporate Social Responsibility Activities Through Twitter: From Topic Model Analysis to Indexes Measuring Communication Characteristics," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(3), pages 1217-1248, December.
    7. Jason Anastasopoulos & George J. Borjas & Gavin G. Cook & Michael Lachanski, 2018. "Job Vacancies, the Beveridge Curve, and Supply Shocks: The Frequency and Content of Help-Wanted Ads in Pre- and Post-Mariel Miami," NBER Working Papers 24580, National Bureau of Economic Research, Inc.
    8. Seraphine F. Maerz & Carsten Q. Schneider, 2020. "Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 517-545, April.
    9. Iasmin Goes, 2023. "Examining the effect of IMF conditionality on natural resource policy," Economics and Politics, Wiley Blackwell, vol. 35(1), pages 227-285, March.
    10. Erkan Gunes & Christoffer Koch Florczak, 2025. "Replacing or enhancing the human coder? Multiclass classification of policy documents with large language models," Journal of Computational Social Science, Springer, vol. 8(2), pages 1-20, May.
    11. Latifi, Albina & Naboka-Krell, Viktoriia & Tillmann, Peter & Winker, Peter, 2024. "Fiscal policy in the Bundestag: Textual analysis and macroeconomic effects," European Economic Review, Elsevier, vol. 168(C).
    12. Yeomans, Michael, 2021. "A concrete example of construct construction in natural language," Organizational Behavior and Human Decision Processes, Elsevier, vol. 162(C), pages 81-94.
    13. Mykola Makhortykh & Ernesto León & Clara Christner & Maryna Sydorova & Aleksandra Urman & Silke Adam & Michaela Maier & Teresa Gil-Lopez, 2025. "Is a single model enough? The systematic comparison of computational approaches for detecting populist radical right content," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(2), pages 1163-1207, April.
    14. Bastiaan Bruinsma & Moa Johansson, 2024. "Finding the structure of parliamentary motions in the Swedish Riksdag 1971–2015," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(4), pages 3275-3301, August.
    15. Mohsin Javed & Zuzana Tučková & Abdul Bashiru Jibril, 2020. "The Role of Social Media on Tourists’ Behavior: An Empirical Analysis of Millennials from the Czech Republic," Sustainability, MDPI, vol. 12(18), pages 1-19, September.
    16. Justyna Klejdysz & Robin L. Lumsdaine, 2023. "Shifts in ECB Communication: A Textual Analysis of the Press Conference," International Journal of Central Banking, International Journal of Central Banking, vol. 19(2), pages 473-542, June.
    17. Nisar, Tahir M. & Prabhakar, Guru & Patil, Pushp P., 2018. "Sports clubs’ use of social media to increase spectator interest," International Journal of Information Management, Elsevier, vol. 43(C), pages 188-195.
    18. Kayser, Victoria & Blind, Knut, 2017. "Extending the knowledge base of foresight: The contribution of text mining," Technological Forecasting and Social Change, Elsevier, vol. 116(C), pages 208-215.
    19. Tiwary, Nishant Kumar & Kumar, Rishi Kant & Sarraf, Shagun & Kumar, Prashant & Rana, Nripendra P., 2021. "Impact assessment of social media usage in B2B marketing: A review of the literature and a way forward," Journal of Business Research, Elsevier, vol. 131(C), pages 121-139.
    20. Rebeca Soler-Costa & Pablo Lafarga-Ostáriz & Marta Mauri-Medrano & Antonio-José Moreno-Guerrero, 2021. "Netiquette: Ethic, Education, and Behavior on Internet—A Systematic Literature Review," IJERPH, MDPI, vol. 18(3), pages 1-15, January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcsosc:v:8:y:2025:i:3:d:10.1007_s42001-025-00406-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.