IDEAS home Printed from https://ideas.repec.org/a/gvu/jscp00/v4y2022i4id1546.html

Implications of Tokenizers in BERT Model for Low-Resource Indian Language

Author

Listed:
  • N. Venkatesan
  • N. Arulanand

Abstract

For any deep learning language model, the initial tokens are prepared as a part of the text preparation process, Tokenization. Important de facto models like BERT and GPT de facto utilize WordPiece and Byte Pair Encoding (BPE) as approaches. Tokenization may have a distinct impact on models for low-resource languages, such as the south Indian Dravidian languages, where many words may be produced by adding prefixes and suffixes. In this paper, four tokenizers are compared at various granularity levels, i.e., their outputs range from the tiniest individual letters to words in their most basic form. Using the BERT pretraining process on the Tamil text, these tokenizers as well as the language models are trained. The model is then fine-tuned with numerous parameters adjusted for the improved performance for a subsequent job in Tamil text categorization. The custom-built tokenizer for Tamil text is created and trained with BPE, WordPiece Vocabulary, Unigram, and WordLevel mechanisms and the compared results are presented after the downstream task of Tamil text categorization is performed using the BERT algorithm.

Suggested Citation

Handle: RePEc:gvu:jscp00:v:4:y:2022:i:4:id:1546
DOI: 10.36548/jscp.2022.4.005
as

Download full text from publisher

File URL: https://irojournals.com/jscp/article/view/1546
File Function: Abstract page
Download Restriction: no

File URL: https://irojournals.com/jscp/article/download/1546/1391
File Function: Full text
Download Restriction: no

File URL: https://libkey.io/10.36548/jscp.2022.4.005?utm_source=ideas
LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
---><---

More about this item

Keywords

;
;
;
;
;
;
;
;

Statistics

Access and download statistics

Corrections

All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gvu:jscp00:v:4:y:2022:i:4:id:1546. See general information about how to correct material in RePEc.

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

We have no bibliographic references for this item. You can help adding them by using this form .

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Journal Manager (email available below). General contact details of provider: https://irojournals.com/jscp .

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.