Towards Optimal NLP Solutions: Analyzing GPT and LLaMA-2 Models Across Model Scale, Dataset Size, and Task Diversity
Received: 16 March 2024 | Revised: 30 March 2024 | Accepted: 2 April 2024 | Online: 25 April 2024
Corresponding author: Richa Sharma
Abstract
This study carries out a comprehensive comparison of fine-tuned GPT models (GPT-2, GPT-3, GPT-3.5) and LLaMA-2 models (LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 70B) in text classification, addressing dataset sizes, model scales, and task diversity. Since its inception in 2018, the GPT series has been pivotal in advancing NLP, with each iteration introducing substantial enhancements. Despite its progress, detailed analyses, especially against competitive open-source models like the LLaMA-2 series in text classification, remain scarce. The current study fills this gap by fine-tuning these models across varied datasets, focusing on enhancing task-specific performance in hate speech and offensive language detection, fake news classification, and sentiment analysis. The learning efficacy and efficiency of the GPT and LLaMA-2 models were evaluated, providing a nuanced guide to choosing optimal models for NLP tasks based on architectural benefits and adaptation efficiency with limited data and resources. In particular, even with datasets as small as 1,000 rows per class, the F1 scores for the GPT-3.5 and LLaMA-2 models exceeded 0.9, reaching 0.99 with complete datasets. Additionally, the LLaMA-2 13B and 70B models outperformed GPT-3, demonstrating their superior efficiency and effectiveness in text classification. Both the GPT and LLaMA-2 series showed commendable performance on all three tasks, underscoring their ability to handle a diversity of tasks. Based on the size, performance, and resources required for fine-tuning the model, this study identifies LLaMA-2 13B as the most optimal model for NLP tasks.
Keywords:
natural language processing, large language models, GPT series, LLaMA-2 series, fine tuningDownloads
References
E. Yilmaz and O. Can, "Unveiling Shadows: Harnessing Artificial Intelligence for Insider Threat Detection," Engineering, Technology & Applied Science Research, vol. 14, no. 2, pp. 13341–13346, Apr. 2024.
A. Kazm, A. Ali, and H. Hashim, "Transformer Encoder with Protein Language Model for Protein Secondary Structure Prediction," Engineering, Technology & Applied Science Research, vol. 14, no. 2, pp. 13124–13132, Apr. 2024.
R. Sharma, S. Deol, U. Kaushish, P. Pandey, and V. Maurya, "DWAEF: a deep weighted average ensemble framework harnessing novel indicators for sarcasm detection 1," Data Science, vol. 6, no. 1–2, pp. 17–44, Jan. 2023.
K. A. Aldriwish, "Empowering Learning through Intelligent Data-Driven Systems," Engineering, Technology & Applied Science Research, vol. 14, no. 1, pp. 12844–12849, Feb. 2024.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Improving Language Understanding by Generative Pre-Training." [Online]. Available: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language Models are Unsupervised Multitask Learners," OpenAI, San Francisco, CA, USA.
T. Brown et al., "Language Models are Few-Shot Learners," in Advances in Neural Information Processing Systems, 2020, vol. 33, pp. 1877–1901.
"OpenAI Platform: GPT-3.5 Turbo." https://platform.openai.com/docs/models/gpt-3-5-turbo.
"GPT-4 Technical Report," OpenAI, San Francisco, CA, USA, 2023. [Online]. Available: https://cdn.openai.com/papers/gpt-4.pdf.
H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models." arXiv, Feb. 27, 2023.
H. Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv, Jul. 19, 2023.
B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, and A. Mukherjee, "HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 17, pp. 14867–14875, May 2021.
R. Socher et al., "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank," in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA, pp. 1631–1642.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, "XLNet: generalized autoregressive pretraining for language understanding," in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, Sep. 2019, pp. 5753–5763.
C. Bekar, K. Carlaw, and R. Lipsey, "General purpose technologies in theory, application and controversy: a review," Journal of Evolutionary Economics, vol. 28, no. 5, pp. 1005–1033, Dec. 2018.
"Fake News Challenge." http://www.fakenewschallenge.org/.
"Google Colaboratory." https://colab.research.google.com/.
"Fine-tuning - OpenAI API." https://platform.openai.com/docs/guides/fine-tuning.
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs." arXiv, May 23, 2023.
Downloads
How to Cite
License
Copyright (c) 2024 Ankit Kumar, Richa Sharma, Punam Bedi
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
