A research project implementing transformer-based sequence-to-sequence models for Nepali language spelling correction, comparing mT5-small, mBART, and varta-T5 models.
As Nepal is embracing digitization across various sectors, the need for robust language tools like spelling correction tool for Nepali language is very important. Spelling errors, the fundamental errors in any language, hinder effective communication, knowledge creation, and further linguistic studies. The complex nature of its Devanagari script and nuanced grammar poses challenges in terms of spelling accuracy. While a previous research has employed sequence-to-sequence technique using GRU-based model with Levenshtein distance for Nepali spelling correction. This project takes a significant step forward by leveraging the power of transfer learning of transformer based sequence-to-sequence pre-trained models and explores their effectiveness by employing and comparing the performance of three distinct transformer-based models of different sizes. A parallel dataset is prepared from A Large Scale Nepali Text Corpus, using pseudo-random synthetic data generation technique, to fine tune these models. The three models varta-t5, mt5-small and mbart obtained accuracy of 75.47%, 76% and 75% on their test set, and 48.22%, 39.11% and 51.67% on the test set of different domain, showing the higher performance of the mt5-small on specific domain and better generalization capacity of mbart on dataset of other domain.