IDEAS home Printed from https://ideas.repec.org/a/gam/jijerp/v19y2022i10p5893-d814188.html
   My bibliography  Save this article

Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering

Author

Listed:
  • Khishigsuren Davagdorj

    (School of Electrical and Computer Engineering, Chungbuk National University, Cheongju 28644, Korea)

  • Ling Wang

    (School of Computer Science, Northeast Electric Power University, Jilin 132013, China)

  • Meijing Li

    (College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China)

  • Van-Huy Pham

    (Data Science Laboratory, Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh City 700000, Vietnam)

  • Keun Ho Ryu

    (Data Science Laboratory, Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh City 700000, Vietnam
    Biomedical Engineering Institute, Chiang Mai University, Chiang Mai 50200, Thailand)

  • Nipon Theera-Umpon

    (Biomedical Engineering Institute, Chiang Mai University, Chiang Mai 50200, Thailand
    Department of Electrical Engineering, Faculty of Engineering, Chiang Mai University, Chiang Mai 50200, Thailand)

Abstract

The increasing expansion of biomedical documents has increased the number of natural language textual resources related to the current applications. Meanwhile, there has been a great interest in extracting useful information from meaningful coherent groupings of textual content documents in the last decade. However, it is challenging to discover informative representations and define relevant articles from the rapidly growing biomedical literature due to the unsupervised nature of document clustering. Moreover, empirical investigations demonstrated that traditional text clustering methods produce unsatisfactory results in terms of non-contextualized vector space representations because that neglect the semantic relationship between biomedical texts. Recently, pre-trained language models have emerged as successful in a wide range of natural language processing applications. In this paper, we propose the Gaussian Mixture Model-based efficient clustering framework that incorporates substantially pre-trained (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) BioBERT domain-specific language representations to enhance the clustering accuracy. Our proposed framework consists of main three phases. First, classic text pre-processing techniques are used biomedical document data, which crawled from the PubMed repository. Second, representative vectors are extracted from a pre-trained BioBERT language model for biomedical text mining. Third, we employ the Gaussian Mixture Model as a clustering algorithm, which allows us to assign labels for each biomedical document. In order to prove the efficiency of our proposed model, we conducted a comprehensive experimental analysis utilizing several clustering algorithms while combining diverse embedding techniques. Consequently, the experimental results show that the proposed model outperforms the benchmark models by reaching performance measures of Fowlkes mallows score, silhouette coefficient, adjusted rand index, Davies-Bouldin score of 0.7817, 0.3765, 0.4478, 1.6849, respectively. We expect the outcomes of this study will assist domain specialists in comprehending thematically cohesive documents in the healthcare field.

Suggested Citation

  • Khishigsuren Davagdorj & Ling Wang & Meijing Li & Van-Huy Pham & Keun Ho Ryu & Nipon Theera-Umpon, 2022. "Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering," IJERPH, MDPI, vol. 19(10), pages 1-21, May.
  • Handle: RePEc:gam:jijerp:v:19:y:2022:i:10:p:5893-:d:814188
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1660-4601/19/10/5893/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1660-4601/19/10/5893/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Leacky Muchene & Wende Safari, 2021. "Two-stage topic modelling of scientific publications: A case study of University of Nairobi, Kenya," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-20, January.
    2. Joaquim Fernando Pinto da Costa & Manuel Cabral, 2022. "Statistical Methods with Applications in Data Mining: A Review of the Most Recent Works," Mathematics, MDPI, vol. 10(6), pages 1-22, March.
    3. Tserenpurev Chuluunsaikhan & Ga-Ae Ryu & Kwan-Hee Yoo & HyungChul Rah & Aziz Nasridinov, 2020. "Incorporating Deep Learning and News Topic Modeling for Forecasting Pork Prices: The Case of South Korea," Agriculture, MDPI, vol. 10(11), pages 1-22, October.
    4. Erdenebileg Batbaatar & Keun Ho Ryu, 2019. "Ontology-Based Healthcare Named Entity Recognition from Twitter Messages Using a Recurrent Neural Network Approach," IJERPH, MDPI, vol. 16(19), pages 1-19, September.
    5. Kabil Boukhari & Mohamed Nazih Omri, 2020. "Approximate matching-based unsupervised document indexing approach: application to biomedical domain," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 903-924, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ga-Ae Ryu & Tserenpurev Chuluunsaikhan & Aziz Nasridinov & HyungChul Rah & Kwan-Hee Yoo, 2023. "SCE-LSTM: Sparse Critical Event-Driven LSTM Model with Selective Memorization for Agricultural Time-Series Prediction," Agriculture, MDPI, vol. 13(11), pages 1-21, October.
    2. Zhou, Yusheng & Wang, Xueqin & Yuen, Kum Fai, 2021. "Sustainability disclosure for container shipping: A text-mining approach," Transport Policy, Elsevier, vol. 110(C), pages 465-477.
    3. Wuyue An & Lin Wang & Yu‐Rong Zeng, 2023. "Text‐based soybean futures price forecasting: A two‐stage deep learning approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 312-330, March.
    4. Adelaida Ojeda-Beltrán & Andrés Solano-Barliza & Wilson Arrubla-Hoyos & Danny Daniel Ortega & Dora Cama-Pinto & Juan Antonio Holgado-Terriza & Miguel Damas & Gilberto Toscano-Vanegas & Alejandro Cama-, 2023. "Characterisation of Youth Entrepreneurship in Medellín-Colombia Using Machine Learning," Sustainability, MDPI, vol. 15(13), pages 1-19, June.
    5. Xiaohong Yu & Bin Liu & Yongzeng Lai, 2024. "Monthly Pork Price Prediction Applying Projection Pursuit Regression: Modeling, Empirical Research, Comparison, and Sustainability Implications," Sustainability, MDPI, vol. 16(4), pages 1-26, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jijerp:v:19:y:2022:i:10:p:5893-:d:814188. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.