Evaluation of Arabic Large Language Models on Moroccan Dialect
Received: 23 January 2025 | Revised: 28 February 2025 | Accepted: 13 March 2025 | Online: 26 March 2025
Corresponding author: Tawfeeq Alsanoosy
Abstract
Large Language Models (LLMs) have shown outstanding performance in many Natural Language Processing (NLP) tasks for high-resource languages, especially English, primarily because most of them were trained on widely available text resources. As a result, many low-resource languages, such as Arabic and African languages and their dialects, are not well studied, raising concerns about whether LLMs can perform fairly across them. Therefore, evaluating the performance of LLMs for low-resource languages and diverse dialects is crucial. This study investigated the performance of LLMs in Moroccan Arabic, a low-resource dialect spoken by approximately 30 million people. The performance of 14 Arabic pre-trained models was evaluated on the Moroccan dialect, employing 11 datasets across various NLP tasks such as text classification, sentiment analysis, and offensive language detection. The evaluation results showed that MARBERTv2 achieved the highest overall average F1-score of 83.47, while the second-best model, DarijaBERT-mix, had an average F1-score of 83.38. These findings provide valuable insights into the effectiveness of current LLMs for low-resource languages, particularly the Moroccan dialect.
Keywords:
LLMs, Arabic language, transformers, NLP, Moroccan dialect, distributed computingDownloads
References
M. D. Alahmadi, M. Alharbi, A. Tayeb, and M. Alshangiti, "Evaluating Large Language Models’ Proficiency in Answering Arabic GAT Exam Questions," Engineering, Technology & Applied Science Research, vol. 14, no. 6, pp. 17774–17780, Dec. 2024.
W. Lei, N. A. S. Abdullah, and S. R. S. Aris, "A Systematic Literature Review on Automatic Sexism Detection in Social Media," Engineering, Technology & Applied Science Research, vol. 14, no. 6, pp. 18178–18188, Dec. 2024.
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proceedings of the 2019 Conference of the North, Minneapolis, MN, USA, 2019, pp. 4171–4186.
L. Martin et al., "CamemBERT: a Tasty French Language Model," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7203–7219.
J. Cañete, G. Chaperon, R. Fuentes, J. H. Ho, H. Kang, and J. Pérez, "Spanish Pre-trained BERT Model and Evaluation Data." arXiv, Aug. 06, 2023.
A. Slim, A. Melouah, U. Faghihi, and K. Sahib, "Improving Neural Machine Translation for Low Resource Algerian Dialect by Transductive Transfer Learning Strategy," Arabian Journal for Science and Engineering, vol. 47, no. 8, pp. 10411–10418, Aug. 2022.
F. Qarah, "EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora." arXiv, Aug. 07, 2024.
H. Haddad et al., "TunBERT: Pretraining BERT for Tunisian Dialect Understanding," SN Computer Science, vol. 4, no. 2, Feb. 2023, Art. no. 194.
I. Guellil, H. Saâdane, F. Azouaou, B. Gueni, and D. Nouvel, "Arabic natural language processing: An overview," Journal of King Saud University - Computer and Information Sciences, vol. 33, no. 5, pp. 497–507, Jun. 2021.
A. Farghaly and K. Shaalan, "Arabic Natural Language Processing: Challenges and Solutions," ACM Transactions on Asian Language Information Processing, vol. 8, no. 4, Sep. 2009.
M. Ennaji, Multilingualism, Cultural Identity, and Education in Morocco. Springer Science & Business Media, 2005.
M. T. I. Khondaker, A. Waheed, E. M. B. Nagoudi, and M. Abdul-Mageed, "GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP." arXiv, Oct. 21, 2023.
A. Maxutov, A. Myrzakhmet, and P. Braslavski, "Do LLMs Speak Kazakh? A Pilot Evaluation of Seven Models," in Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), Bangkok, Thailand and Online, Dec. 2024, pp. 81–91.
A. Üstün et al., "Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model." arXiv, Feb. 12, 2024.
S. Arif, A. H. Azeemi, A. A. Raza, and A. Athar, "Generalists vs. Specialists: Evaluating Large Language Models for Urdu." arXiv, Oct. 03, 2024.
M. H. Tahir, S. Shams, L. Fiaz, F. Adeeba, and S. Hussain, "Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks." arXiv, Dec. 31, 2024.
T. Wolf et al., "Transformers: State-of-the-Art Natural Language Processing," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, Jul. 2020, pp. 38–45.
M. Jbel, M. Jabrane, I. Hafidi, and A. Metrane, "Sentiment analysis dataset in Moroccan dialect: bridging the gap between Arabic and Latin scripted dialect," Language Resources and Evaluation, Sep. 2024.
J. Mouad, "MouadJb/MYC." Mar. 18, 2025, [Online]. Available: https://github.com/MouadJb/MYC.
K. Essefar, H. Ait Baha, A. El Mahdaouy, A. El Mekki, and I. Berrada, "OMCD: Offensive Moroccan Comments Dataset," Language Resources and Evaluation, vol. 57, no. 4, pp. 1745–1765, Dec. 2023.
K. Essefar, "kabilessefar/OMCD-Offensive-Moroccan-Comments-Dataset." Mar. 07, 2025, [Online]. Available: https://github.com/
kabilessefar/OMCD-Offensive-Moroccan-Comments-Dataset.
K. Gaanoun, A. M. Naira, A. Allak, and I. Benelallam, "DarijaBERT: a step forward in NLP for the written Moroccan dialect," International Journal of Data Science and Analytics, Jan. 2024.
"DBert/Data at main AIOXLABS/DBert," GitHub. https://github.com/AIOXLABS/DBert/tree/main/Data.
M. Garouani and J. Kharroubi, "MAC: An Open and Free Moroccan Arabic Corpus for Sentiment Analysis," in Innovations in Smart Cities Applications Volume 5, 2022, pp. 849–858.
LeMGarouani, "LeMGarouani/MAC." Mar. 18, 2025, [Online]. Available: https://github.com/LeMGarouani/MAC.
A. Elouardighi, M. Maghfour, and H. Hammia, "Collecting and Processing Arabic Facebook Comments for Sentiment Analysis," in Model and Data Engineering, 2017, pp. 262–274.
"sentiprojects/ElecMorocco2016." Mar. 18, 2025, [Online]. Available: https://github.com/sentiprojects/ElecMorocco2016.
R. M. Duwairi, R. Marji, N. Sha’ban, and S. Rushaidat, "Sentiment Analysis in Arabic tweets," in 2014 5th International Conference on Information and Communication Systems (ICICS), Irbid, Jordan, Apr. 2014, pp. 1–6.
"AbderrahmanSkiredj1/MSAC_darija_sentiment_analysis," Hugging Face. https://huggingface.co/datasets/AbderrahmanSkiredj1/MSAC_
darija_sentiment_analysis.
A. Ibrahimi and A. Mourhir, "Moroccan Darija Offensive Language Detection Dataset." Mendeley Data, Oct. 24, 2023.
E. Boujou, H. Chataoui, A. E. Mekki, S. Benjelloun, I. Chairi, and I. Berrada, "An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction." arXiv, Feb. 07, 2021.
"Open Datasets - UM6P College of Computing." https://cc.um6p.ma/cc_datasets.
M. Maghfour and A. Elouardighi, "Standard and Dialectal Arabic Text Classification for Sentiment Analysis," in Model and Data Engineering, 2018, pp. 282–291.
"sentiprojects/MSA_MDA_comments." Aug. 07, 2022, [Online]. Available: https://github.com/sentiprojects/MSA_MDA_comments.
O. Lamine, "Tweet_Classification_Moroccan_Dataset." [Online]. Available: https://www.kaggle.com/datasets/omarlamine/tweet-classification-moroccan-dataset.
"ohidaoui/darija-reviews · Datasets at Hugging Face," Nov. 30, 2024. https://huggingface.co/datasets/ohidaoui/darija-reviews.
W. Antoun, F. Baly, and H. Hajj, "AraBERT: Transformer-based Model for Arabic Language Understanding." arXiv, Mar. 07, 2021.
M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, "ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic." arXiv, Jun. 07, 2021.
G. Inoue, B. Alhafni, N. Baimukan, H. Bouamor, and N. Habash, "The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models." arXiv, Sep. 04, 2021.
A. Abdelali, S. Hassan, H. Mubarak, K. Darwish, and Y. Samih, "Pre-Training BERT on Arabic Tweets: Practical Considerations." arXiv, Feb. 21, 2021.
F. Qarah, "SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora." arXiv, May 10, 2024.
A. Messaoudi et al., "TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect," in Intelligent Systems and Pattern Recognition, 2022, pp. 278–290.
A. Abdaoui, M. Berrimi, M. Oussalah, and A. Moussaoui, "DziriBERT: a Pre-trained Language Model for the Algerian Dialect." arXiv, Dec. 12, 2022.
O. Moussaoui and Y. E. Younnoussi, "Pre-training Two BERT-Like Models for Moroccan Dialect: MorRoBERTa and MorrBERT," MENDEL, vol. 29, no. 1, pp. 55–61, Jun. 2023.
Downloads
How to Cite
License
Copyright (c) 2025 Faisal Qarah, Tawfeeq Alsanoosy
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
