Advancing Sentiment Analysis: Evaluating RoBERTa against Traditional and Deep Learning Models
Received: 23 November 2024 | Revised: 15 December 2024 | Accepted: 1 January 2025 | Online: 2 February 2025
Corresponding author: Wirapong Chansanam
Abstract
This research evaluates the performance of various sentiment analysis models, including traditional machine learning approaches (Naive Bayes, KNN, CART), a deep learning model (LSTM), and the transformer-based model RoBERTa using an Amazon book reviews dataset. ROBERTa outperformed all other models, achieving an accuracy of 96.30% and an F1-score of 98.11%, underscoring its superior ability to process complex and semantically diverse textual data. Traditional models, while computationally efficient, demonstrated limitations in capturing nuanced textual relationships, and the LSTM model, although competitive, faced scalability challenges and overfitting issues. These results demonstrate how transformer-based architectures such as RoBERTa offer advantages in real-world applications, particularly in e-commerce and social media sentiment analysis. This study underscores the superior capabilities of RoBERTa for sentiment analysis, particularly in processing semantically diverse and context-rich textual data that traditional models struggle to capture. Future work will explore optimizing RoBERTa's computational efficiency and expanding its applications to multilingual and cross-domain sentiment analysis tasks.
Keywords:
sentiment analysis, RoBERTa, Amazon book reviews, deep learning, machine learning modelDownloads
References
A. Alsaeedi and M. Zubair, "A Study on Sentiment Analysis Techniques of Twitter Data," International Journal of Advanced Computer Science and Applications, vol. 10, no. 2, 2019.
W. Chansanam and K. Tuamsuk, "Thai Twitter Sentiment Analysis: Performance Monitoring of Politics in Thailand using Text Mining Techniques," International Journal of Innovation, vol. 11, no. 12, 2020.
S. Sweta, "Application of Sentiment Analysis in Diverse Domains," in Sentiment Analysis and its Application in Educational Data Mining, S. Sweta, Ed. Singapore: Springer Nature, 2024, pp. 19–46.
N. Shrestha and F. Nasoz, "Deep Learning Sentiment Analysis of Amazon.com Reviews and Ratings," International Journal on Soft Computing, Artificial Intelligence and Applications, vol. 8, no. 1, pp. 01–15, Feb. 2019.
K. L. Tan, C. P. Lee, K. S. M. Anbananthen, and K. M. Lim, "RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis With Transformer and Recurrent Neural Network," IEEE Access, vol. 10, pp. 21517–21525, 2022.
A. S. Lv, D. Babu.M, A.Manonmani, Y. M. Reeja, M. S. S, and A. R. Kumar, "An Efficient Approach in Selection of Information-Gaining Features Using Sentiment Analysis," Journal of Computational Analysis and Applications (JoCAAA), vol. 33, no. 05, pp. 719–725, Sep. 2024.
J. M. T. Habib and A. A. Poguda, "Comparison of Deep Learning Sentiment Analysis Methods, Including LSTM and Machine Learning," Open Education, vol. 27, no. 4, pp. 60–71, Aug. 2023.
K. L. Tan, C. P. Lee, and K. M. Lim, "RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis," Applied Sciences, vol. 13, no. 6, Jan. 2023, Art. no. 3915.
M. M. Rahman, A. I. Shiplu, Y. Watanobe, and M. A. Alam, "RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis." arXiv, Jun. 01, 2024.
C. Aspillaga, A. Carvallo, and V. Araujo, "Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks." arXiv, Mar. 27, 2020.
A. Rawat, H. Maheshwari, M. Khanduja, R. Kumar, M. Memoria, and S. Kumar, "Sentiment Analysis of Covid19 Vaccines Tweets Using NLP and Machine Learning Classifiers," in 2022 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COM-IT-CON), Faridabad, India, May 2022, pp. 225–230.
O. Iparraguirre-Villanueva et al., "The Public Health Contribution of Sentiment Analysis of Monkeypox Tweets to Detect Polarities Using the CNN-LSTM Model," Vaccines, vol. 11, no. 2, Feb. 2023, Art. no. 312.
K. K. Mohbey, G. Meena, S. Kumar, and K. Lokesh, "A CNN-LSTM-Based Hybrid Deep Learning Approach for Sentiment Analysis on Monkeypox Tweets," New Generation Computing, vol. 42, no. 1, pp. 89–107, Mar. 2024.
A. K. Laturiuw and Y. A. Singgalen, "Sentiment Analysis of Raja Ampat Tourism Destination Using CRISP-DM: SVM, NBC, DT, and k-NN Algorithm," Journal of Information Systems and Informatics, vol. 5, no. 2, pp. 518–535, May 2023.
D. Suryadi and J. T. Sabarman, "Analyzing Restaurants in Tourism Destinations Through Online Reviews Using Topic Modeling and Sentiment Analysis," in 2023 IEEE 9th Information Technology International Seminar (ITIS), Batu Malang, Indonesia, Oct. 2023, pp. 1–6.
"Amazon Books Reviews." Kaggle, Accessed: Jan. 04, 2025. [Online]. Available: https://www.kaggle.com/datasets/mohamedbakhet/amazon-books-reviews.
A. McCallum and K. Nigam, "A comparison of event models for naive bayes text classification," in AAAI-98 workshop on learning for text categorization, 1998, vol. 752, no. 1, pp. 41–48.
C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval, 1st ed. Cambridge University Press, 2008.
D. D. Lewis, "Naive (Bayes) at forty: The independence assumption in information retrieval," in Machine Learning: ECML-98, 1998, pp. 4–15.
S. L. Salzberg, "C4.5: Programs for Machine Learning," Machine Learning, vol. 16, no. 3, pp. 235–240, Sep. 1994.
L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification And Regression Trees, 1st ed. Routledge, 2017.
T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21–27, Jan. 1967.
N. S. Altman, "An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression," The American Statistician, vol. 46, no. 3, pp. 175–185, Aug. 1992.
L. Peterson, "K-nearest neighbor," Scholarpedia, vol. 4, no. 2, 2009, Art. no. 1883.
S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
F. A. Gers, J. Schmidhuber, and F. Cummins, "Learning to Forget: Continual Prediction with LSTM," Neural Computation, vol. 12, no. 10, pp. 2451–2471, Oct. 2000.
A. Graves and J. Schmidhuber, "Framewise phoneme classification with bidirectional LSTM networks," in Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., Montreal, Canada, 2005, vol. 4, pp. 2047–2052.
Y. Liu et al., "RoBERTa: A Robustly Optimized BERT Pretraining Approach." arXiv, Jul. 26, 2019.
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, May 24, 2019.
Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou, "MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020, pp. 2158–2170.
D. M. W. Powers, "Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation." arXiv, Oct. 11, 2020.
M. Sokolova and G. Lapalme, "A systematic analysis of performance measures for classification tasks," Information Processing & Management, vol. 45, no. 4, pp. 427–437, Jul. 2009.
H. M and S. M.N, "A Review on Evaluation Metrics for Data Classification Evaluations," International Journal of Data Mining & Knowledge Management Process, vol. 5, no. 2, pp. 01–11, Mar. 2015.
T. Ngootip, "Enhancing Network Intrusion Detection in Cloud Computing Using a Deep Boltzmann Machine and LightGBM Ensemble Model: A Performance Evaluation on the NSL-KDD Dataset," Sociolytics Journal, vol. 1, no. 1, pp. 1–7, Sep. 2024.
S. Pansayta and W. Chansanam, "Thai COVID-19 patient clustering for monitoring and prevention: data mining techniques," IAES International Journal of Artificial Intelligence (IJ-AI), vol. 13, no. 1, Mar. 2024, Art. no. 256.
P. Manorom, U. Detthamrong, and W. Chansanam, "Comparative Assessment of Fraudulent Financial Transactions using the Machine Learning Algorithms Decision Tree, Logistic Regression, Naïve Bayes, K-Nearest Neighbor, and Random Forest," Engineering, Technology & Applied Science Research, vol. 14, no. 4, pp. 15676–15680, Aug. 2024.
M. K. Myee, R. D. C. Rebekah, T. Deepa, G. D. Zion, and K. Lokesh, "Detection of Depression in Social Media Posts using Emotional Intensity Analysis," Engineering, Technology & Applied Science Research, vol. 14, no. 5, pp. 16207–16211, Oct. 2024.
Downloads
How to Cite
License
Copyright (c) 2025 Pongsathon Pookduang, Rapeepat Klangbunrueang, Wirapong Chansanam, Tassanee Lunrasri
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
