Deep Learning and Fusion Mechanism-based Multimodal Fake News Detection Methodologies: A Review
Received: 21 May 2024 | Revised: 1 June 2024 | Accepted: 3 June 2024 | Online: 2 August 2024
Corresponding author: Iman Qays Abduljaleel
Abstract
Today, detecting fake news has become challenging as anyone can interact by freely sending or receiving electronic information. Deep learning processes to detect multimodal fake news have achieved great success. However, these methods easily fuse information from different modality sources, such as concatenation and element-wise product, without considering how each modality affects the other, resulting in low accuracy. This study presents a focused survey on the use of deep learning approaches to detect multimodal visual and textual fake news on various social networks from 2019 to 2024. Several relevant factors are discussed, including a) the detection stage, which involves deep learning algorithms,
b) methods for analyzing various data types, and c) choosing the best fusion mechanism to combine multiple data sources. This study delves into the existing constraints of previous studies to provide future tips for addressing open challenges and problems.
Keywords:
Misinformation, Attention mechanism, Fusion methods, social media, Vision transformerDownloads
References
S. Hangloo and B. Arora, "Combating multimodal fake news on social media: methods, datasets, and future perspective," Multimedia Systems, vol. 28, no. 6, pp. 2391–2422, Dec. 2022.
L. Hu, S. Wei, Z. Zhao, and B. Wu, "Deep learning for fake news detection: A comprehensive survey," AI Open, vol. 3, pp. 133–155, Jan. 2022.
C. Comito, L. Caroprese, and E. Zumpano, "Multimodal fake news detection on social media: a survey of deep learning techniques," Social Network Analysis and Mining, vol. 13, no. 1, Aug. 2023, Art. no. 101.
D. Gifu, "An Intelligent System for Detecting Fake News," Procedia Computer Science, vol. 221, pp. 1058–1065, Jan. 2023.
J. Li and M. Lei, "A Brief Survey for Fake News Detection via Deep Learning Models," Procedia Computer Science, vol. 214, pp. 1339–1344, Jan. 2022.
A. Gandhi, K. Adhvaryu, S. Poria, E. Cambria, and A. Hussain, "Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions," Information Fusion, vol. 91, pp. 424–444, Mar. 2023.
M. Nirav Shah and A. Ganatra, "A systematic literature review and existing challenges toward fake news detection models," Social Network Analysis and Mining, vol. 12, no. 1, Nov. 2022, Art. no. 168.
A. Figueira, N. Guimaraes, and L. Torgo, "Current State of the Art to Detect Fake News in Social Media: Global Trendings and Next Challenges:," in Proceedings of the 14th International Conference on Web Information Systems and Technologies, Seville, Spain, 2018, pp. 332–339.
J. Xue, Y. Wang, Y. Tian, Y. Li, L. Shi, and L. Wei, "Detecting fake news by exploring the consistency of multimodal data," Information Processing & Management, vol. 58, no. 5, Sep. 2021, Art. no. 102610.
"Google Trends," Google Trends. https://trends.google.com/trends/explore?date=today%205-y&q=fake%20news&hl=en (accessed May 30, 2024).
J. Wang, J. Zheng, S. Yao, R. Wang, and H. Du, "TLFND: A Multimodal Fusion Model Based on Three-Level Feature Matching Distance for Fake News Detection," Entropy, vol. 25, no. 11, Nov. 2023, Art. no. 1533.
K. Liu and M. Hai, "Rumor Detection of Covid-19 Related Microblogs on Sina Weibo," Procedia Computer Science, vol. 221, pp. 386–393, Jan. 2023.
S. Ahmed, K. Hinkelmann, and F. Corradini, "Combining Machine Learning with Knowledge Engineering to detect Fake News in Social Networks - a survey." arXiv, Jan. 20, 2022.
Y. Shen, Q. Liu, N. Guo, J. Yuan, and Y. Yang, "Fake News Detection on Social Networks: A Survey," Applied Sciences, vol. 13, no. 21, Jan. 2023, Art. no. 11877.
Y. Guo, H. Ge, and J. Li, "A two-branch multimodal fake news detection model based on multimodal bilinear pooling and attention mechanism," Frontiers in Computer Science, vol. 5, Apr. 2023.
L. Qian, R. Xu, and Z. Zhou, "MRDCA: a multimodal approach for fine-grained fake news detection through integration of RoBERTa and DenseNet based upon fusion mechanism of co-attention," Annals of Operations Research, Dec. 2022.
A. M. Luvembe, W. Li, S. Li, F. Liu, and X. Wu, "CAF-ODNN: Complementary attention fusion with optimized deep neural network for multimodal fake news detection," Information Processing & Management, vol. 61, no. 3, May 2024, Art. no. 103653.
Z. Qu, Y. Meng, G. Muhammad, and P. Tiwari, "QMFND: A quantum multimodal fusion-based fake news detection model for social media," Information Fusion, vol. 104, Apr. 2024, Art. no. 102172.
F. A. O. Santos, K. L. Ponce-Guevara, D. Macêdo, and C. Zanchettin, "Improving Universal Language Model Fine-Tuning using Attention Mechanism," in 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, Jul. 2019, pp. 1–7.
P. Meel and D. K. Vishwakarma, "HAN, image captioning, and forensics ensemble multimodal fake news detection," Information Sciences, vol. 567, pp. 23–41, Aug. 2021.
C. Raj and P. Meel, "ConvNet frameworks for multi-modal fake news detection," Applied Intelligence, vol. 51, no. 11, pp. 8132–8148, Nov. 2021.
L. Wang, C. Zhang, H. Xu, Y. Xu, X. Xu, and S. Wang, "Cross-modal Contrastive Learning for Multimodal Fake News Detection," in Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, Canada, Nov. 2023, pp. 5696–5704.
R. Kumari and A. Ekbal, "AMFB: Attention based multimodal Factorized Bilinear Pooling for multimodal Fake News Detection," Expert Systems with Applications, vol. 184, Dec. 2021, Art. no. 115412.
J. Zeng, Y. Zhang, and X. Ma, "Fake news detection for epidemic emergencies via deep correlations between text and images," Sustainable Cities and Society, vol. 66, Mar. 2021, Art. no. 102652.
I. Segura-Bedmar and S. Alonso-Bartolome, "Multimodal Fake News Detection," Information, vol. 13, no. 6, Jun. 2022, Art. no. 284.
V. Kishore and M. Kumar, "Enhanced Multimodal Fake News Detection with Optimal Feature Fusion and Modified Bi-LSTM Architecture," Cybernetics and Systems, Jan. 2023.
Z. Liang, "Fake News Detection Based on Multimodal Inputs," Computers, Materials & Continua, vol. 75, no. 2, pp. 4519–4534, 2023.
X. Cui and Y. Li, "Fake News Detection in Social Media based on Multi-Modal Multi-Task Learning," International Journal of Advanced Computer Science and Applications (IJACSA), vol. 13, no. 7, 31 2022.
D. K. Sharma, B. Singh, S. Agarwal, H. Kim, and R. Sharma, "FakedBits- Detecting Fake Information on Social Platforms using Multi-Modal Features," KSII Transactions on Internet and Information Systems (TIIS), vol. 17, no. 1, pp. 51–73, Jan. 2023.
L. Wu, P. Liu, and Y. Zhang, "See How You Read? Multi-Reading Habits Fusion Reasoning for Multi-Modal Fake News Detection," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, pp. 13736–13744, Jun. 2023.
X. Wang, X. Li, X. Liu, and H. Cheng, "Using ALBERT and Multi-modal Circulant Fusion for Fake News Detection," in 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic, 2022, pp. 2936–2942.
N. M. Duc Tuan and P. Quang Nhat Minh, "Multimodal Fusion with BERT and Attention Mechanism for Fake News Detection," in 2021 RIVF International Conference on Computing and Communication Technologies (RIVF), Hanoi, Vietnam, Aug. 2021, pp. 1–6.
R. Jaiswal, U. P. Singh, and K. P. Singh, "Fake News Detection Using BERT-VGG19 Multimodal Variational Autoencoder," in 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), Dehradun, India, Nov. 2021, pp. 1–5.
H. Yang et al., "Multi-Modal fake news Detection on Social Media with Dual Attention Fusion Networks," in 2021 IEEE Symposium on Computers and Communications (ISCC), Athens, Greece, Sep. 2021, pp. 1–6.
L. Ying, H. Yu, J. Wang, Y. Ji, and S. Qian, "Multi-Level Multi-Modal Cross-Attention Network for Fake News Detection," IEEE Access, vol. 9, pp. 132363–132373, 2021.
Y. Guo and W. Song, "A Temporal-and-Spatial Flow Based Multimodal Fake News Detection by Pooling and Attention Blocks," IEEE Access, vol. 10, pp. 131498–131508, 2022.
Y. Liang, T. Tohti, and A. Hamdulla, "False Information Detection via Multimodal Feature Fusion and Multi-Classifier Hybrid Prediction," Algorithms, vol. 15, no. 4, Apr. 2022, Art. no. 119.
S. K. Uppada, P. Patel, and S. B., "An image and text-based multimodal model for detecting fake news in OSN’s," Journal of Intelligent Information Systems, vol. 61, no. 2, pp. 367–393, Oct. 2023.
S. K. Hamed, M. J. Ab Aziz, and M. R. Yaakub, "A review of fake news detection approaches: A critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusion," Heliyon, vol. 9, no. 10, Oct. 2023, Art. no. e20382.
D. S. Asudani, N. K. Nagwani, and P. Singh, "Impact of word embedding models on text analytics in deep learning environment: a review," Artificial Intelligence Review, vol. 56, no. 9, pp. 10345–10425, Sep. 2023.
J. Egger, A. Pepe, C. Gsaxner, Y. Jin, J. Li, and R. Kern, "Deep learning—a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact," PeerJ Computer Science, vol. 7, Nov. 2021, Art. no. e773.
K. Han et al., "A Survey on Visual Transformer," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87–110, Jan. 2023.
A. Choudhary and A. Arora, "Assessment of bidirectional transformer encoder model and attention based bidirectional LSTM language models for fake news detection," Journal of Retailing and Consumer Services, vol. 76, Jan. 2024, 103545.
S. F. N. Azizah, H. D. Cahyono, S. W. Sihwi, and W. Widiarto, "Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection." arXiv, Aug. 09, 2023.
D. Tomás, R. Ortega-Bueno, G. Zhang, P. Rosso, and R. Schifanella, "Transformer-based models for multimodal irony detection," Journal of Ambient Intelligence and Humanized Computing, vol. 14, no. 6, pp. 7399–7410, Jun. 2023.
J. Maurício, I. Domingues, and J. Bernardino, "Comparing Vision Transformers and Convolutional Neural Networks for Image Classification: A Literature Review," Applied Sciences, vol. 13, no. 9, Jan. 2023, Art. no. 5521.
Z. Jin, J. Cao, Y. Zhang, J. Zhou, and Q. Tian, "Novel Visual and Statistical Image Features for Microblogs News Verification," IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 598–608, Mar. 2017.
K. Shu, L. Cui, S. Wang, D. Lee, and H. Liu, "dEFEND: Explainable Fake News Detection," in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, Apr. 2019, pp. 395–405.
T. Chen, X. Li, H. Yin, and J. Zhang, "Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection," in Trends and Applications in Knowledge Discovery and Data Mining, Melbourne, Australia, 2018, pp. 40–52.
M. H. Goldani, S. Momtazi, and R. Safabakhsh, "Detecting fake news with capsule neural networks," Applied Soft Computing, vol. 101, Mar. 2021, Art. no. 106991.
F. Qian, C. Gong, K. Sharma, and Y. Liu, "Neural User Response Generator: Fake News Detection with Collective User Intelligence," in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, Jul. 2018, pp. 3834–3840.
N. Kausar, A. AliKhan, and M. Sattar, "Towards better representation learning using hybrid deep learning model for fake news detection," Social Network Analysis and Mining, vol. 12, no. 1, Nov. 2022, Art. no. 165.
S. Abdali, S. Shaham, and B. Krishnamachari, "Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities." arXiv, Mar. 27, 2024.
M.-H. Guo et al., "Attention mechanisms in computer vision: A survey," Computational Visual Media, vol. 8, no. 3, pp. 331–368, Sep. 2022.
B. Ahmed, G. Ali, A. Hussain, A. Baseer, and J. Ahmed, "Analysis of Text Feature Extractors using Deep Learning on Fake News," Engineering, Technology & Applied Science Research, vol. 11, no. 2, pp. 7001–7005, Apr. 2021.
H. M. Al-Dabbas, R. A. Azeez, and A. E. Ali, "Two Proposed Models for Face Recognition: Achieving High Accuracy and Speed with Artificial Intelligence," Engineering, Technology & Applied Science Research, vol. 14, no. 2, pp. 13706–13713, Apr. 2024.
T. Zhang et al., "BDANN: BERT-Based Domain Adaptation Neural Network for Multi-Modal Fake News Detection," in 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, Jul. 2020, pp. 1–8.
C. Song, N. Ning, Y. Zhang, and B. Wu, "A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks," Information Processing & Management, vol. 58, no. 1, Jan. 2021, Art. no. 102437.
P. Yang, J. Ma, Y. Liu, and M. Liu, "Multi-modal transformer for fake news detection," Mathematical biosciences and engineering, vol. 20, no. 8, pp. 14699–14717, Jul. 2023.
Downloads
How to Cite
License
Copyright (c) 2024 Iman Qays Abduljaleel , Israa H. Ali
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
