Multimodal Sentiment Analysis of Twitter Data Using Early Fusion Along with a Fully Connected Neural Network and Multilayer Perceptron Framework
Corresponding author: B. S. Harish
Abstract
Multimodal sentiment analysis of social media data remains challenging due to the complex integration of textual and visual information. A Fully Connected Neural Network and Multilayer Perceptron-based Sentiment Analysis (SA-FCNNMP) framework is proposed for effective multimodal sentiment classification of Twitter data. The approach combines textual and visual data to improve sentiment comprehension and is tested on the Multi-View Sentiment Analysis (MVSA) Single Twitter Dataset, which consists of 4,869 images labeled with their corresponding texts. The approach begins by cleaning textual data through stop-word removal using Bayesian Boundary Trend Filtering (BBTF), preserving only meaningful words. Subsequently, features are extracted from text and images independently to leverage their unique properties. The input textual data are represented using Word2Vec embeddings that encode semantic relationships, whereas image features are extracted using a ResNet-50 convolutional neural network. These features are then fused at an early stage using Hierarchical Multi-Scale Feature Fusion (HMSFF), which incorporates information across all scales and modalities into a consistent representation. The fused features are channeled through a Fully Connected Neural Network and Multilayer Perceptron (FCNNMP), which is optimized using the data to classify sentiment more effectively. The SA-FCNNMP model categorizes sentiments as positive, negative, or neutral. Performance is evaluated using accuracy, precision, recall, sensitivity, and computational time. To maximize learning and prediction robustness, the model uses both cross-entropy loss, which works well for multi-class classification problems, and Mean Squared Error (MSE) loss, which captures finer-grained variations in sentiment distribution. Compared with existing state-of-the-art methods, experimental results demonstrate that the proposed SA-FCNNMP model outperforms existing methods in multimodal sentiment analysis on Twitter data.
Keywords:
multimodal sentiment classification, Twitter, fully connected neural network, multilayer perceptron, Bayesian Boundary Trend Filtering (BBTF), Word2Vec, ResNet-50Downloads
References
W. Aljedaani et al., "Sentiment analysis on Twitter data integrating TextBlob and deep learning models: The case of US airline industry," Knowledge-Based Systems, vol. 255, Nov. 2022, Art. no. 109780. DOI: https://doi.org/10.1016/j.knosys.2022.109780
M. Bibi et al., "A novel unsupervised ensemble framework using concept-based linguistic methods and machine learning for twitter sentiment analysis," Pattern Recognition Letters, vol. 158, pp. 80–86, June 2022. DOI: https://doi.org/10.1016/j.patrec.2022.04.004
Md. M. Rahman and M. N. Islam, "Exploring the Performance of Ensemble Machine Learning Classifiers for Sentiment Analysis of COVID-19 Tweets," in Sentimental Analysis and Deep Learning: Proceedings of ICSADL 2021, Songkhla, Thailand, 2021, pp. 383–396. DOI: https://doi.org/10.1007/978-981-16-5157-1_30
A. Alwehaibi, M. Bikdash, M. Albogmi, and K. Roy, "A study of the performance of embedding methods for Arabic short-text sentiment analysis using deep learning approaches," Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 8, Part B, pp. 6140–6149, Sept. 2022. DOI: https://doi.org/10.1016/j.jksuci.2021.07.011
M. Qorib, T. Oladunni, M. Denis, E. Ososanya, and P. Cotae, "Covid-19 vaccine hesitancy: Text mining, sentiment analysis and machine learning on COVID-19 vaccination Twitter dataset," Expert Systems with Applications, vol. 212, Feb. 2023, Art. no. 118715. DOI: https://doi.org/10.1016/j.eswa.2022.118715
N. Aslam, F. Rustam, E. Lee, P. B. Washington, and I. Ashraf, "Sentiment Analysis and Emotion Detection on Cryptocurrency Related Tweets Using Ensemble LSTM-GRU Model," IEEE Access, vol. 10, pp. 39313–39324, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3165621
Md. A. Babu, M. Ahammad, M. Mahmud, and Md. S. Uddin, "Social Media as a Market Prophecy: Leveraging ML Algorithms for Predicting Market Trends and Demand," Transportation Research Procedia, vol. 84, pp. 137–144, Jan. 2025. DOI: https://doi.org/10.1016/j.trpro.2025.03.056
S. Tabinda Kokab, S. Asghar, and S. Naz, "Transformer-based deep learning models for the sentiment analysis of social media data," Array, vol. 14, July 2022, Art. no. 100157. DOI: https://doi.org/10.1016/j.array.2022.100157
G. Kaur and A. Sharma, "A deep learning-based model using hybrid feature extraction approach for consumer sentiment analysis," Journal of Big Data, vol. 10, no. 1, Jan. 2023, Art. no. 5. DOI: https://doi.org/10.1186/s40537-022-00680-6
E. Lee, F. Rustam, P. B. Washington, F. E. Barakaz, W. Aljedaani, and I. Ashraf, "Racism Detection by Analyzing Differential Opinions Through Sentiment Analysis of Tweets Using Stacked Ensemble GCR-NN Model," IEEE Access, vol. 10, pp. 9717–9728, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3144266
K. Pasupa and T. Seneewong Na Ayutthaya, "Hybrid Deep Learning Models for Thai Sentiment Analysis," Cognitive Computation, vol. 14, no. 1, pp. 167–193, Jan. 2022. DOI: https://doi.org/10.1007/s12559-020-09770-0
K. L. Tan, C. P. Lee, K. S. M. Anbananthen, and K. M. Lim, "RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis With Transformer and Recurrent Neural Network," IEEE Access, vol. 10, pp. 21517–21525, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3152828
A. P. Rodrigues and N. N. Chiplunkar, "A new big data approach for topic classification and sentiment analysis of Twitter data," Evolutionary Intelligence, vol. 15, no. 2, pp. 877–887, June 2022. DOI: https://doi.org/10.1007/s12065-019-00236-3
I. E. Fattoh, F. Kamal Alsheref, W. M. Ead, and A. M. Youssef, "Semantic Sentiment Classification for COVID-19 Tweets Using Universal Sentence Encoder," Computational Intelligence and Neuroscience, vol. 2022, no. 1, Oct. 2022, Art. no. 6354543. DOI: https://doi.org/10.1155/2022/6354543
T. H. Jaya Hidayat, Y. Ruldeviyani, A. R. Aditama, G. R. Madya, A. W. Nugraha, and M. W. Adisaputra, "Sentiment analysis of twitter data related to Rinca Island development using Doc2Vec and SVM and logistic regression as classifier," Procedia Computer Science, vol. 197, pp. 660–667, Jan. 2022. DOI: https://doi.org/10.1016/j.procs.2021.12.187
F. K. Sufi and I. Khalil, "Automated Disaster Monitoring From Social Media Posts Using AI-Based Location Intelligence and Sentiment Analysis," IEEE Transactions on Computational Social Systems, vol. 11, no. 4, pp. 4614–4624, Aug. 2024. DOI: https://doi.org/10.1109/TCSS.2022.3157142
A. Sarirete, "Sentiment analysis tracking of COVID-19 vaccine through tweets," Journal of Ambient Intelligence and Humanized Computing, vol. 14, no. 11, pp. 14661–14669, Nov. 2023. DOI: https://doi.org/10.1007/s12652-022-03805-0
L. G. Singh and S. R. Singh, "Sentiment analysis of tweets using text and graph multi-views learning," Knowledge and Information Systems, vol. 66, no. 5, pp. 2965–2985, May 2024. DOI: https://doi.org/10.1007/s10115-023-02053-8
P. Rakshit and A. Sarkar, "A supervised deep learning-based sentiment analysis by the implementation of Word2Vec and GloVe Embedding techniques," Multimedia Tools and Applications, vol. 84, no. 2, pp. 979–1012, Jan. 2025. DOI: https://doi.org/10.1007/s11042-024-19045-7
S. Zhang, Y. He, L. Li, and Y. Dou, "Multimodal sentiment analysis with BERT-ResNet50," in Second International Conference on Algorithms, Microchips, and Network Applications, Zhengzhou, China, 2023, vol. 12635, pp. 232–236. DOI: https://doi.org/10.1117/12.2679113
Y. Li, H. Ding, Y. Lin, X. Feng, and L. Chang, "Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis," Artificial Intelligence Review, vol. 57, no. 4, Mar. 2024, Art. no. 78. DOI: https://doi.org/10.1007/s10462-023-10685-z
Y. Qi and Z. Shabrina, "Sentiment analysis using Twitter data: a comparative application of lexicon- and machine-learning-based approach," Social Network Analysis and Mining, vol. 13, no. 1, Feb. 2023, Art. no. 31. DOI: https://doi.org/10.1007/s13278-023-01030-x
T. Niu, S. Zhu, L. Pang, and A. El Saddik, "Sentiment Analysis on Multi-View Social Data," in 22nd International Conference on Multimedia Modeling, Miami, FL, USA, 2016, pp. 15–27. DOI: https://doi.org/10.1007/978-3-319-27674-8_2
N. Xu and W. Mao, "MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis," in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 2017, pp. 2399–2402. DOI: https://doi.org/10.1145/3132847.3133142
A. Alsayat, "Improving Sentiment Analysis for Social Media Applications Using an Ensemble Deep Learning Language Model," Arabian Journal for Science and Engineering, vol. 47, no. 2, pp. 2499–2511, Feb. 2022. DOI: https://doi.org/10.1007/s13369-021-06227-w
X. Yang, S. Feng, D. Wang, and Y. Zhang, "Image-Text Multimodal Emotion Classification via Multi-View Attentional Network," IEEE Transactions on Multimedia, vol. 23, pp. 4014–4026, 2021. DOI: https://doi.org/10.1109/TMM.2020.3035277
T. Zhu, L. Li, J. Yang, S. Zhao, H. Liu, and J. Qian, "Multimodal Sentiment Analysis With Image-Text Interaction Network," IEEE Transactions on Multimedia, vol. 25, pp. 3375–3385, 2023. DOI: https://doi.org/10.1109/TMM.2022.3160060
M. A. M. Hamidi, A. Y. Taqa, and Y. I. Ibrahim, "A Systematic Review of Multimodal Sentiment Analysis Based on Text-Image Fusion: Trends, Models, and Research Gaps," Sinkron : jurnal dan penelitian teknik informatika, vol. 9, no. 2, pp. 987–999, Apr. 2025. DOI: https://doi.org/10.33395/sinkron.v9i2.14840
K. Wang and Y. Zhang, "Topic Sentiment Analysis in Online Learning Community from College Students," Journal of Data and Information Science, vol. 5, no. 2, pp. 33–61, May 2020. DOI: https://doi.org/10.2478/jdis-2020-0009
B. Bhavana, C. Chaitanya, and B. M. G, "Multimodal Question Answering with DenseNet and BERT for Improved User Interaction," in 2025 Fourth International Conference on Smart Technologies, Communication and Robotics, Sathyamangalam, India, 2025, pp. 1–7. DOI: https://doi.org/10.1109/STCR62650.2025.11019326
D. Wang, X. Guo, Y. Tian, J. Liu, L. He, and X. Luo, "TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis," Pattern Recognition, vol. 136, Apr. 2023, Art. no. 109259. DOI: https://doi.org/10.1016/j.patcog.2022.109259
Z. Liu, L. Cai, W. Yang, and J. Liu, "Sentiment analysis based on text information enhancement and multimodal feature fusion," Pattern Recognition, vol. 156, Dec. 2024, Art. no. 110847. DOI: https://doi.org/10.1016/j.patcog.2024.110847
L. Zhu, Z. Zhu, C. Zhang, Y. Xu, and X. Kong, "Multimodal sentiment analysis based on fusion methods: A survey," Information Fusion, vol. 95, pp. 306–325, July 2023. DOI: https://doi.org/10.1016/j.inffus.2023.02.028
W. Zou, J. Ding, and C. Wang, "Utilizing BERT Intermediate Layers for Multimodal Sentiment Analysis," in 2022 IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, 2022, pp. 1–6. DOI: https://doi.org/10.1109/ICME52920.2022.9860014
J. Li et al., "Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis," in Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, Lisbon, Portugal, 2022, pp. 81–88. DOI: https://doi.org/10.1145/3551876.3554809
K. L. Tan, C. P. Lee, K. M. Lim, and K. S. M. Anbananthen, "Sentiment Analysis With Ensemble Hybrid Deep Learning Model," IEEE Access, vol. 10, pp. 103694–103704, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3210182
S. Kumari and M. P. Singh, "A Deep Learning Multimodal Framework for Fake News Detection," Engineering, Technology & Applied Science Research, vol. 14, no. 5, pp. 16527–16533, Oct. 2024. DOI: https://doi.org/10.48084/etasr.8170
T. Onizuka, F. Iwashige, and Shintaro Hashimoto, "Bayesian boundary trend filtering," Computational Statistics & Data Analysis, vol. 191, Mar. 2024, Art. no. 107889. DOI: https://doi.org/10.1016/j.csda.2023.107889
M. Grohe, "word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Data," in Proceedings of the 39th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, Portland, OR, USA, 2020, pp. 1–16. DOI: https://doi.org/10.1145/3375395.3387641
B. Koonce, "ResNet 50," in Convolutional Neural Networks with Swift for Tensorflow: Image Recognition and Dataset Categorization, Berkeley, CA, USA: Apress, 2021, pp. 63–72. DOI: https://doi.org/10.1007/978-1-4842-6168-2_6
G. R. Kishore, B. S. Harish, and C. K. Roopa, "Unenhanced Sparse Vector-based Embedding Method for Sentiment Analysis," Engineering, Technology & Applied Science Research, vol. 15, no. 2, pp. 21225–21231, Apr. 2025. DOI: https://doi.org/10.48084/etasr.10098
X. Huo et al., "HiFuse: Hierarchical multi-scale feature fusion network for medical image classification," Biomedical Signal Processing and Control, vol. 87, Jan. 2024, Art. no. 105534. DOI: https://doi.org/10.1016/j.bspc.2023.105534
S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, "Language (Technology) is Power: A Critical Survey of ‘Bias’ in NLP," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020, pp. 5454–5476. DOI: https://doi.org/10.18653/v1/2020.acl-main.485
L. Gong, X. He, and J. Yang, "An Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning," Applied Artificial Intelligence, vol. 38, no. 1, Dec. 2024, Art. no. 2371712. DOI: https://doi.org/10.1080/08839514.2024.2371712
Downloads
How to Cite
License
Copyright (c) 2025 T. S. Kaveri, B. S. Harish, C. K. Roopa, M. S. Kendagannaswamy
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
