InstructPLM: Aligning protein language models to follow protein structural instructions

Large language models are known for their effectiveness in capturing complex patterns, including coevolutionary relationships and underlying protein languages. However, current methods often struggle to demonstrate the occurrence of genomic insertions, duplications, and insertions/deletions (indels), which account for approximately 14% of human pathogenic mutations. Given that structure determines function, mutant proteins with similar structures are more likely to be conserved through biological evolution. Inspired by this, the authors used cross-modality alignment and instruct fine-tuning techniques inspired by large language models to align generative protein language models with protein structure instructions. Specifically, they propose a method to generate variable-length and diverse proteins to explore and simulate the complex evolution of life, thereby expanding the options for protein engineering. The protein language model-based method InstructPLM proposed by the author has shown significant performance improvements in both computer simulations and experiments. On the natural protein backbone, it achieves a perplexity of 2.68 and a sequence recovery rate of 57.51, which are 39.2% and 25.1% higher than ProteinMPNN respectively. In addition, the authors verified the effectiveness of the model by redesigning PETase and L-MDH. For PETase, all 15 designed variable-length PETases exhibited depolymerization activity, 11 of which exceeded the wild-type activity level. For L-MDH, an enzyme that lacks an experimentally determined structure, InstructPLM was able to design a functional enzyme with the predicted structure of AF2. The code and model weights of InstructPLM are public at https://github.com/Eikor/InstructPLM.

Mutation, including point mutations caused by single nucleotide changes as well as deletions, duplications, and insertions of multiple bases, are the driving force of evolution, provide the basis for life to rapidly adapt to changing environmental conditions, and are the foundation of genetic diversity. reason. These genetic variations provide abundant protein resources for protein engineering. In protein engineering, a typical challenge is protein sequence design, also known as protein reverse folding, which requires finding an amino acid sequence that can fold into a specific protein backbone structure. Highly precise protein sequence design can lead to more efficient enzymes, improved protein-based therapeutics, and engineered proteins for industrial purposes such as biofuel production and environmental remediation. The recent emergence of deep learning-based methods, such as ProteinMPNN and ESM-IF, has greatly promoted the development of the field of protein sequence design. These methods formalize protein sequence design as a multimodal learning problem and solve the problem by converting the protein backbone structure into the corresponding sequence by training the encoder-decoder mode of the model. The rapid development of deep learning architectures, including graph neural networks and Transformers, combined with high-quality protein structure data from the Protein Data Bank (PDB) and AlphaFoldDB, has facilitated significant progress in the field. For example, myoglobin and tobacco etch virus (TEV) protease designed by ProteinMPNN show higher expression, stability and functionality; conjugates designed de novo by ProteinMPNN show high affinity and specificity for the target.

Multimodal learning, although simple and effective, faces significant obstacles due to the scarcity of high-quality cross-modal data. For example, in the field of visual-language understanding, training models from scratch requires high-quality image-text pair datasets, such as image description and visual question answering. In protein sequence design, the training of the above-mentioned ProteinMPNN relies on protein components in the Protein Data Bank (PDB) determined by X-ray crystallography or cryo-electron microscopy experiments. To solve the problem of limited data, a new strategy has emerged in the field of large-scale visual-language models, namely cross-modal alignment, which is said to have been integrated into GPT-4 (although the technical details of GPT-4 have not been publicly disclosed). This technique leverages existing pre-trained single-modal models to facilitate multi-modal understanding. Pretrained unimodal models learn domain-specific representations through extensive training on large unimodal datasets, but they often lack the ability for cross-modal understanding and reasoning. Cross-modal alignment bridges this gap by carefully aligning pre-trained visual models with pre-trained language models, enabling efficient cross-modal generation, understanding, and reasoning. Despite the success of cross-modal alignment techniques in visual-verbal understanding, its potential in protein sequence design remains unexplored due to inherent differences between protein science and visual-verbal tasks. Therefore, addressing this challenge requires the development of powerful single-modal models and alignment techniques specifically tailored for proteins.

Fortunately, protein language models (pLMs), such as ESM, ProtGPT, and ProGen, have become key innovations in protein sequence modeling in bioinformatics and computational biology. Inspired by the achievements of large-scale language models GPT and BERT, pLMs are pre-trained on large protein sequence datasets, resulting in a comprehensive understanding of the protein domain. This includes capturing the distribution of observed evolved sequences, generating new feasible sequences, and predicting protein fitness. Specifically, pLMs have demonstrated the ability to generate functional protein sequences based on specific conditions. For example, GPT-based pLMs, such as ProGen and ProtGPT, can generate proteins based on homologous samples or control tags that specify protein properties; ESM-based pLMs design the desired protein by applying or sampling from a pre-trained mask language model sequence. However, unlike general language models that demonstrate zero-shot generalization capabilities and understand user intentions on a wide range of tasks through methods such as instruction fine-tuning or reinforcement learning, pLMs are still incompetent in how to follow fine and complex biological instructions to generate protein sequences or even simulate the evolution of life. It is an open research field. As stated in Seven Technologies to Watch in 2024, regarding deep learning for protein design, “Sequence-based methods can exploit and adapt existing protein features to form new frameworks, but they rely on the customization of structural elements or features. The design is less effective.”

In this work, the authors demonstrate the successful application of cross-modal alignment and instruction fine-tuning techniques originally developed for large language models and now applied to the field of protein sequence design. This transferability emphasizes the broad applicability of these technologies in the life sciences. The model proposed by the author, InstructPLM, uses a lightweight cross-attention layer to align a fixed protein backbone encoder with a fixed protein language model decoder, aiming to teach the protein language model to design sequences according to protein structure instructions. This configuration can take advantage of the strong generalization ability of pLMs while motivating pLMs to design according to residue-level protein structural instructions. InstructPLM surpasses existing sequence design techniques in terms of perplexity and sequence recovery, while only increasing parameter training by 2%.

Model Overview

The overall model architecture of InstructPLM consists of three parts: a protein language decoder, a protein backbone encoder, and a protein structure to sequence adapter.

A diagram of a diagram

Description automatically generated

Fig1.

1. For the protein language decoder part, the author here uses the model with the most parameters in ProGen2, ProGen2-xlarge, with 6.4B parameters;

2. For the protein backbone encoder part, the author initializes its encoder from existing protein sequence design models (such as ProteinMPNN, ESM-IF);

3. The adapter from protein structure to sequence is a key component in InstructPLM, responsible for aligning the structure and sequence in the semantic space. It is also the only module in InstructPLM that requires training. This adapter contains a randomly initialized single-layer cross-attention module. This module uses multiple trainable embeddings as query vectors, and the output of the protein backbone encoder as key/values for cross-attention computation. This cross-attention module compresses protein backbone features into a fixed-length structural instruction. In addition, the author follows the method of Qwen-VL and adds one-dimensional absolute position encoding to the cross-attention to retain protein primary structure information during the compression process. The compressed sequence of structural features of the protein backbone is then fed into the protein language model as a soft hint.

In silico experimental validation of protein sequence design

A screenshot of a computer screen

Description automatically generated

Fig2.

A table with numbers and a number of short

Description automatically generated with medium confidence

Table1.

For computational evaluation, the authors evaluated the performance of InstructPLM using CATH 4.2, a widely adopted benchmark in protein sequence design. The author uses the official training set, validation set and test set, which contain 18,024, 608 and 1,120 proteins respectively. Table 1 and the left panel of Fig 2(A) show the detailed sequence design performance on the CATH 4.2 holdout test set.

InstructPLM demonstrates superior performance compared to models trained from scratch on the CATH 4.2 dataset. In terms of perplexity, InstructPLM significantly improves the best baseline ESM-IF by 32.3% (from 3.96 to 2.68). In terms of sequence recovery metrics, InstructPLM also showed significant enhancement, with the recovery rate increasing by 11.3% (from 51.66 to 57.51) compared with the best baseline PiFold. Of particular note, the comparison between InstructPLM and ProteinMPNN highlights the effectiveness of protein language models in characterizing evolving sequence distributions. This is obvious since they use the same main chain encoder, the main difference is their sequence decoder.

More interestingly, InstructPLM shows a consistent pattern of improvement in different subsets of CATH 4.2, especially achieving more significant enhancements when designing short proteins and single-chain proteins. This observation suggests that the model architecture, training technique, and prior knowledge of the pLM are particularly effective for these protein types. This also shows that InstructPLM is not only a powerful general protein designer, but also excels in specialized applications, especially when the protein sequence is short or consists of a single chain.

The traditional sequence recovery metric is not suitable for assessing the quality of variable-length sequences because it is assessed based on the position-by-position accuracy of a designed sequence against its native counterpart. To effectively evaluate the unique capabilities of InstructPLM, the authors propose the use of sequence homology scores as a new metric. The authors use DEDAL, a state-of-the-art homology detection method based on deep learning. The method shows significant improvements in alignment accuracy in the search for distal homologs, with performance two to three times higher than traditional methods. In the test group of CATH 4.2, the authors randomly designed 20 protein sequences for each natural protein structure, with the temperature set to 0.15. The sequence homology score determined by DEDAL is used as a metric to compare the similarity of natural and artificially designed sequences. As shown in the right panel of Figure 2(A), the homology score distribution of various methods is shown. The average homology score of InstructPLM is 63.97, which exceeds the 23.25/8.83/18.9 of ProteinMPNN/ESM-IF/PiFold respectively. . Figure 2(B) shows the homology score distribution of sequences of different lengths in detail, proving that InstructPLM maintains a consistent high performance level when designing sequences of different lengths.

Wet experimental verification of InstructPLM

A diagram of a structure screening

Description automatically generated

Fig3.

Although InstructPLM achieved significant improvements in in silico sequence design metrics such as perplexity, sequence recovery, and homology scores, the authors also tested InstructPLM's ability to design functional proteins. Specifically, the authors evaluate InstructPLM on PETase (PDB code: 7SH6) and L-MDH (Uniprot: A0A319AA41). It is worth noting that L-MDH does not have an experimentally determined structure, so the authors designed L-MDH based on its AF2 predicted structure. The design process is outlined in Figure 3(A). Using the backbone structure as a starting point, InstructPLM generated 10,000 potential sequences via autoregressive generation, using top-p sampling (p = 0.9) and temperature 0.8. The structures of the sequences were predicted using ESMFold, and TM-Score was calculated using DeepScore in the DeepAlign package to evaluate structural similarity. Finally, the authors selected the sequence with the highest TM-Score ranking for further experiments.

Designing PETase was a test of InstructPLM's ability to improve naturally occurring proteins. The author selects the top 15 sequences with the highest TM-Score for experiments. The lengths of these 15 sequences ranged from 261 to 269, reflecting InstructPLM's ability to generate variable length sequences. All 15 sequences designed by InstructPLM showed good expression levels when expressed in E. coli. Quantification of PET hydrolysis reaction products—bis(2-hydroxyethyl)terephthalate (BHET), mono(2-hydroxyethyl)terephthalate (MHET), and p-phenylene glycol—by high-performance liquid chromatography Dicarboxylic acid (TPA). All 15 sequences designed by InstructPLM have PET hydrolysis activity, and 11 of them have better activity than wild type (PDB code: 5XJH), as shown in Figure 3(B). Figure 3(E) shows an example of a sequence generated by InstructPLM that has both a catalytic triad and two disulfide bonds.

L-MDH (malate dehydrogenase, Uniprot: A0A319AA41) is a key enzyme in the citric acid cycle for which no experimentally determined structure is currently available. The authors relied on the structure predicted by AlphaFold2 as the basis for the design. Following the same protocol as PETase, the authors used InstructPLM to generate 10,000 sequences and selected the 15 sequences with the highest TM-Scores. Among the 15 designed L-MDH sequences, experimental verification showed that three exhibited detectable enzymatic activity. Although these active variants did not outperform the wild-type enzyme, given the lack of an experimentally confirmed structure of L-MDH, the fact that these sequences exhibited activity is a significant achievement in itself. The ability to generate functional enzymes based on fully computationally predicted structures highlights the potential of combining advanced protein structure prediction tools such as AlphaFold and structure design tools such as Chroma with InstructPLM. Notably, all three sequences are shorter than the wild type, further demonstrating the flexibility and adaptability of InstructPLM. The authors compared L-MDH with three enzymes with detectable enzymatic activity through sequence alignment and found that they all have the same active site HIS179.

Reference: Qiu, J., Xu, J., Hu, J., Cao, H., Hou, L., Gao, Z., ... & Chen, G. (2024). InstructPLM: Aligning Protein Language Models to Follow Protein Structure Instructions. bioRxiv, 2024-04.

Preprint

Preprint

InstructPLM: Aligning protein language models to follow protein structural instructions