Transformer Encoder with Protein Language Model for Protein Secondary Structure Prediction
Received: 2 January 2024 | Revised: 15 January 2024 | Accepted: 17 January 2024 | Online: 27 January 2024
Corresponding author: Ammar Kazm
Abstract
In bioinformatics, protein secondary structure prediction plays a significant role in understanding protein function and interactions. This study presents the TE_SS approach, which uses a transformer encoder-based model and the Ankh protein language model to predict protein secondary structures. The research focuses on the prediction of nine classes of structures, according to the Dictionary of Secondary Structure of Proteins (DSSP) version 4. The model's performance was rigorously evaluated using various datasets. Additionally, this study compares the model with the state-of-the-art methods in the prediction of eight structure classes. The findings reveal that TE_SS excels in nine- and three-class structure predictions while also showing remarkable proficiency in the eight-class category. This is underscored by its performance in Qs and SOV evaluation metrics, demonstrating its capability to discern complex protein sequence patterns. This advancement provides a significant tool for protein structure analysis, thereby enriching the field of bioinformatics.
Keywords:
bioinformatics, transformer model, protein secondary structure prediction, nine-class protein prediction, Ankh protein language modelDownloads
References
S. Damodaran and K. L. Parkin, Eds., "Amino Acids, Peptides, and Proteins," in Fennema’s Food Chemistry, 5th ed., Boca Raton, FL, USA: CRC Press, 2017.
S. Tahzeeb and S. Hasan, "A Neural Network-Based Multi-Label Classifier for Protein Function Prediction," Engineering, Technology & Applied Science Research, vol. 12, no. 1, pp. 7974–7981, Feb. 2022.
M. Zubair et al., "A Deep Learning Approach for Prediction of Protein Secondary Structure," Computers, Materials & Continua, vol. 72, no. 2, pp. 3705–3718, Mar. 2022.
W. Yang, Y. Liu, and C. Xiao, "Deep metric learning for accurate protein secondary structure prediction," Knowledge-Based Systems, vol. 242, Apr. 2022, Art. no. 108356.
W. Yang, Z. Hu, L. Zhou, and Y. Jin, "Protein secondary structure prediction using a lightweight convolutional network and label distribution aware margin loss," Knowledge-Based Systems, vol. 237, Feb. 2022, Art. no. 107771.
W. Kabsch and C. Sander, "Dictionary of protein secondary structure: Pattern recognition of hydrogen-bonded and geometrical features," Biopolymers, vol. 22, no. 12, pp. 2577–2637, 1983.
W. Yang, C. Liu, and Z. Li, "Lightweight Fine-tuning a Pretrained Protein Language Model for Protein Secondary Structure Prediction." bioRxiv, Mar. 23, 2023.
D. T. Jones, "Protein secondary structure prediction based on position-specific scoring matrices11Edited by G. Von Heijne," Journal of Molecular Biology, vol. 292, no. 2, pp. 195–202, Sep. 1999.
S. R. Eddy, "Profile hidden Markov models.," Bioinformatics, vol. 14, no. 9, pp. 755–763, Jan. 1998.
A. Rives et al., "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences," Proceedings of the National Academy of Sciences, vol. 118, no. 15, Apr. 2021, Art. no. e2016239118.
A. Elnaggar et al., "ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 7112–7127, Jul. 2022.
Z. Lin et al., "Language models of protein sequences at the scale of evolution enable accurate structure prediction." bioRxiv, Jul. 21, 2022.
B. Ahmed, G. Ali, A. Hussain, A. Baseer, and J. Ahmed, "Analysis of Text Feature Extractors using Deep Learning on Fake News," Engineering, Technology & Applied Science Research, vol. 11, no. 2, pp. 7001–7005, Apr. 2021.
J. Singh, T. Litfin, J. Singh, K. Paliwal, and Y. Zhou, "SPOT-Contact-LM: improving single-sequence-based prediction of protein contact map using a transformer language model," Bioinformatics, vol. 38, no. 7, pp. 1888–1894, Mar. 2022.
H. Stark, C. Dallago, M. Heinzinger, and B. Rost, "Light attention predicts protein location from the language of life," Bioinformatics Advances, vol. 1, no. 1, Jan. 2021, Art. no. vbab035.
S. Pokharel, P. Pratyush, M. Heinzinger, R. H. Newman, and D. B. Kc, "Improving protein succinylation sites prediction using embeddings from protein language model," Scientific Reports, vol. 12, no. 1, Oct. 2022, Art. no. 16933.
A. Villegas-Morcillo, A. M. Gomez, and V. Sanchez, "An analysis of protein language model embeddings for fold prediction," Briefings in Bioinformatics, vol. 23, no. 3, May 2022, Art. no. bbac142.
M. H. Hoie et al., "NetSurfP-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning," Nucleic Acids Research, vol. 50, no. W1, pp. W510–W515, Jul. 2022.
J. Singh, K. Paliwal, T. Litfin, J. Singh, and Y. Zhou, "Reaching alignment-profile-based accuracy in predicting protein secondary and tertiary structural properties without alignment," Scientific Reports, vol. 12, no. 1, May 2022, Art. no. 7607.
M. Levitt and C. Chothia, "Structural patterns in globular proteins," Nature, vol. 261, no. 5561, pp. 552–558, Jun. 1976.
P. Kumar, S. Bankapur, and N. Patil, "An enhanced protein secondary structure prediction using deep learning framework on hybrid profile based features," Applied Soft Computing, vol. 86, Jan. 2020, Art. no. 105926.
J. Selbig, T. Mevissen, and T. Lengauer, "Decision tree-based formation of consensus protein secondary structure prediction," Bioinformatics, vol. 15, no. 12, pp. 1039–1046, Dec. 1999.
B. Yang, Q. Wu, Z. Ying, and H. Sui, "Predicting protein secondary structure using a mixed-modal SVM method in a compound pyramid model," Knowledge-Based Systems, vol. 24, no. 2, pp. 304-313, Mar. 2011.
M. H. Zangooei and S. Jalili, "PSSP with dynamic weighted kernel fusion based on SVM-PHGS," Knowledge-Based Systems, vol. 27, pp. 424–442, Mar. 2012.
Z. Aydin, Y. Altunbasak, and M. Borodovsky, "Protein secondary structure prediction for a single-sequence using hidden semi-Markov models," BMC Bioinformatics, vol. 7, no. 1, Mar. 2006, Art. no. 178.
J. Martin, J.-F. Gibrat, and F. Rodolphe, "Analysis of an optimal hidden Markov model for secondary structure prediction," BMC Structural Biology, vol. 6, no. 1, Dec. 2006, Art. no. 25.
W. Yang, K. Wang, and W. Zuo, "Prediction of protein secondary structure using large margin nearest neighbour classification," International Journal of Bioinformatics Research and Applications, vol. 9, no. 2, pp. 207–219, Jan. 2013.
A. Drozdetskiy, C. Cole, J. Procter, and G. J. Barton, "JPred4: a protein secondary structure prediction server," Nucleic Acids Research, vol. 43, no. W1, pp. W389–W394, Jul. 2015.
D. W. A. Buchan, S. M. Ward, A. E. Lobley, T. C. O. Nugent, K. Bryson, and D. T. Jones, "Protein annotation and modelling servers at University College London," Nucleic Acids Research, vol. 38, no. suppl_2, pp. W563–W568, Jul. 2010.
Z. Li and Y. Yu, "Protein Secondary Structure Prediction Using Cascaded Convolutional and Recurrent Neural Networks." arXiv, Apr. 25, 2016.
A. Busia and N. Jaitly, "Next-Step Conditioned Deep Convolutional Neural Networks Improve Protein Secondary Structure Prediction." arXiv, Feb. 13, 2017.
R. Heffernan, Y. Yang, K. Paliwal, and Y. Zhou, "Capturing non-local interactions by long short-term memory bidirectional recurrent neural networks for improving prediction of protein secondary structure, backbone angles, contact numbers and solvent accessibility," Bioinformatics, vol. 33, no. 18, pp. 2842–2849, Sep. 2017.
Y. Guo, W. Li, B. Wang, H. Liu, and D. Zhou, "DeepACLSTM: deep asymmetric convolutional long short-term memory neural models for protein secondary structure prediction," BMC Bioinformatics, vol. 20, no. 1, Jun. 2019, Art. no. 341.
C. Fang, Y. Shang, and D. Xu, "MUFOLD-SS: New deep inception-inside-inception networks for protein secondary structure prediction," Proteins: Structure, Function, and Bioinformatics, vol. 86, no. 5, pp. 592–598, 2018.
M. R. Uddin, S. Mahbub, M. S. Rahman, and M. S. Bayzid, "SAINT: self-attention augmented inception-inside-inception network improves protein secondary structure prediction," Bioinformatics, vol. 36, no. 17, pp. 4599–4608, Nov. 2020.
J. Hanson, K. Paliwal, T. Litfin, Y. Yang, and Y. Zhou, "Improving prediction of protein secondary structure, backbone angles, solvent accessibility and contact numbers by using predicted contact maps and an ensemble of recurrent and residual convolutional neural networks," Bioinformatics, vol. 35, no. 14, pp. 2403–2410, Jul. 2019.
M. S. Klausen et al., "NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning," Proteins: Structure, Function, and Bioinformatics, vol. 87, no. 6, pp. 520–527, 2019.
Uzma, U. Manzoor, and Z. Halim, "Protein encoder: An autoencoder-based ensemble feature selection scheme to predict protein secondary structure," Expert Systems with Applications, vol. 213, Mar. 2023, Art. no. 119081.
A. Elnaggar et al., "Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose Modelling." bioRxiv, Jan. 18, 2023.
T. S. Mian, "Evaluation of Stock Closing Prices using Transformer Learning," Engineering, Technology & Applied Science Research, vol. 13, no. 5, pp. 11635–11642, Oct. 2023.
M. Steinegger and J. Soding, "Clustering huge protein sequence sets in linear time," Nature Communications, vol. 9, no. 1, Jun. 2018, Art. no. 2542.
B. E. Suzek, Y. Wang, H. Huang, P. B. McGarvey, and C. H. Wu, "UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches," Bioinformatics, vol. 31, no. 6, pp. 926–932, Mar. 2015.
A. Vaswani et al., "Attention is All you Need," in 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, Dec. 2017, vol. 30, pp. 1–15.
A. Zemla, C. Venclovas, K. Fidelis, and B. Rost, "A modified definition of Sov, a segment-based measure for protein secondary structure prediction assessment," Proteins: Structure, Function, and Bioinformatics, vol. 34, no. 2, pp. 220–223, 1999.
I. Drori et al., "High Quality Prediction of Protein Q8 Secondary Structure by Diverse Neural Network Architectures." arXiv, Nov. 17, 2018.
Downloads
How to Cite
License
Copyright (c) 2024 Ammar Kazm, Aida Ali, Haslina Hashim
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
