Natural Language Processing (NLP) is a pre-eminent AI technology that enables machines to read, decipher, understand, and make sense of human languages. From text prediction and sentiment analysis to speech recognition, NLP is allowing machines to emulate human intelligence and abilities impressively.
For building NLP applications, language models are the key. However, building complex NLP language models from scratch is a tedious task. That is why AI and ML developers and researchers swear by pre-trained language models. These models utilize the transfer learning technique for training wherein a model is trained on one dataset to perform a task. Then the same model is repurposed to perform different NLP functions on a new dataset.
The pre-trained model solves a specific problem and requires fine-tuning, which saves a lot of time and computational resources to build a new language model. There are several pre-trained NLP models available that are categorized based on the purpose that they serve. Let's take a look at the top 15 pre-trained NLP models.
UPDATE: This article has been revised to include the latest findings in the field of large language models.
Pre-Trained NLP Language Models
1. GPT-4 (Generative Pre-trained Transformer 4)
GPT-4 is a large language model (LLM) developed by OpenAI. It is the fourth generation of the GPT language model series, and was released on March 14, 2023. GPT-4 is a multimodal model, meaning that it can take both text and images as input. This makes it more versatile than previous GPT models, which could only take text as input.
This model is now accessible to the public through ChatGPT Plus, while access to its commercial API is available through a waitlist. During its development, GPT-4 was trained to anticipate the next piece of content and underwent fine-tuning using feedback from both humans and AI systems. This was done to ensure its alignment with human values and compliance with desired policies.
Regarding improvements, GPT-4 has enhanced ChatGPT's capabilities. However, it's worth noting that it still faces some of the challenges observed in previous models.
Some of the key features of GPT-4 include:
- It is 10 times larger than GPT-3.5, with 175 billion parameters.
- It can handle longer sequences of text, up to 32,000 tokens (equivalent to 25,000 words).
- It is more creative and collaborative, able to generate, edit, and iterate with users on creative and technical writing tasks.
- It is still under development, but it has the potential to be a powerful tool for a variety of applications, including content creation, translation, writing assistance, education, customer service, and research.
2. BERT (Bidirectional Encoder Representations from Transformers)
BERT is a technique for NLP pre-training, developed by Google. It utilizes the Transformer, a novel neural network architecture that’s based on a self-attention mechanism for language understanding. It was developed to address the problem of sequence transduction or neural machine translation. That means, it suits best for any task that transforms an input sequence to an output sequence, such as speech recognition, text-to-speech transformation, etc.
In its vanilla form, the transformer includes two separate mechanisms: an encoder (which reads the text input) and a decoder (which produces a prediction for the task). The goal of the BERT mechanism is to generate a language model. Thus, only the encoder mechanism is necessary.
The BERT algorithm is proven to perform 11 NLP tasks efficiently. It’s trained on 2,500 million Wikipedia words and 800 million words of the BookCorpus dataset. Google Search is one of the most excellent examples of BERT’s efficiency. Other applications from Google, such as Google Docs, Gmail Smart Compose utilizes BERT for text prediction.
3. RoBERTa (Robustly Optimized BERT Pretraining Approach)
RoBERTa is an optimized method for the pre-training of a self-supervised NLP system. It builds the language model on BERT’s language masking strategy that enables the system to learn and predict intentionally hidden sections of text.
RoBERTa modifies the hyperparameters in BERT such as training with larger mini-batches, removing BERT’s next sentence pretraining objective, etc. Pre-trained models like RoBERTa is known to outperform BERT in all individual tasks on the General Language Understanding Evaluation (GLUE) benchmark and can be used for NLP training tasks such as question answering, dialogue systems, document classification, etc.
4. PaLM (Pathways Language Model)
Google Research introduced the Pathways Language Model, abbreviated as PaLM. It's a significant step in language technology, featuring an enormous 540 billion parameters. PaLM's training employed an efficient computing system called Pathways, making it possible to train it across many processors.
PaLM's training process was remarkable for its scalability. It was trained across a substantial 6144 TPU v4 chips, making it one of the most extensive TPU-based training configurations to date.
For training data, PaLM utilized a diverse mix of sources, including English and multilingual datasets. This encompassed web documents, books, Wikipedia content, conversations, and even code from GitHub.
What sets PaLM apart are its capabilities:
- It excels in language tasks like answering questions and completing sentences.
- In terms of reasoning, it can handle complex math problems and provide clear explanations.
- For coding tasks, PaLM is on par with specialized models but requires far less code to learn.
PaLM isn't just a research achievement; it has practical uses across various business domains. It can assist in building chatbots, providing answers, translating languages, organizing documents, generating ads, and aiding in programming tasks.
5. OpenAI’s GPT-3
GPT-3 is a transformer-based NLP model that performs translation, question-answering, poetry composing, cloze tasks, along with tasks that require on-the-fly reasoning such as unscrambling words. Moreover, with its recent advancements, the GPT-3 is used to write news articles and generate codes.
GPT-3 can manage statistical dependencies between different words. It is trained on over 175 billion parameters on 45 TB of text that’s sourced from all over the internet. With this, it is one of the biggest pre-trained NLP models available.
What differentiates GPT-3 from other language models is it does not require fine-tuning to perform downstream tasks. With its ‘text in, text out’ API, the developers are allowed to reprogram the model using instructions.
6. ALBERT
The increasing size of pre-trained language models helps in improving the performance of downstream tasks. However, as the model size increases, it leads to issues such as longer training times and GPU/TPU memory limitations. To address this problem, Google presented a lite version of BERT (Bidirectional Encoder Representations from Transformers). This model was introduced with two parameter-reduction techniques:
- Factorized Embedding Parameterization: Here, the size of the hidden layers are separated from the size of vocabulary embeddings.
- Cross-Layer Parameter Sharing: This prevents the number of parameters from growing with the depth of the network.
These parameter reduction techniques help in lowering memory consumption and increase the training speed of the model. Moreover, ALBERT introduces a self-supervised loss for sentence order prediction which is a BERT limitation with regard to inter-sentence coherence.
7. XLNet
Denoising autoencoding based language models such as BERT helps in achieving better performance than an autoregressive model for language modeling. That is why there is XLNet that introduces the auto-regressive pre-training method which offers the following benefits- it enables learning bidirectional context and helps overcome the limitations of BERT with its autoregressive formula. XLNet is known to outperform BERT on 20 tasks, which includes natural language inference, document ranking, sentiment analysis, question answering, etc.
8. OpenAI’s GPT2
Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on task-specific datasets. OpenAI’s GPT2 demonstrates that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of web pages called WebText. The model generates coherent paragraphs of text and achieves promising, competitive or state-of-the-art results on a wide variety of tasks.
9. StructBERT
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU) and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman, experts have extended BERT to a new model, StructBERT, by incorporating language structures into pre-training.
The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7. Like other pre-trained language models, StructBERT may assist businesses with a variety of NLP tasks, including question answering, sentiment analysis, document summarization, etc.
10. T5 (Text-to-Text Transfer Transformer)
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice.
The Google research team suggests a unified approach to transfer learning in NLP to set a new state of the art in the field. To this end, they propose treating each NLP problem as a “text-to-text” problem. Such a framework allows using the same model, objective, training procedure, and decoding process for different tasks, including summarization, sentiment analysis, question answering, and machine translation. The researchers call their model a Text-to-Text Transfer Transformer (T5) and train it on the large corpus of web-scraped data to get state-of-the-art results on several NLP tasks.
READ MORE: Top 5 NLP Applications Of Transfer Learning
11. Llama (Large Language Model Meta AI)
Meta's LLM, known as Large Language Model Meta AI or Llama, made its debut in 2023.
This is their advanced language model, and the largest version of Llama is quite substantial, containing a vast 70 billion parameters. Initially, access to Llama was restricted to approved researchers and developers. However, it has now been made open source, allowing a wider community to use and explore its capabilities.
What's particularly beneficial about Llama is its adaptability. It comes in various sizes, including smaller versions that demand less computing power. This flexibility makes it more accessible for practical use, testing, and experimentation.
Interestingly, Llama's introduction to the public happened unintentionally, not as part of a scheduled launch. This unforeseen occurrence led to the development of related models, such as Orca, which leverage the solid linguistic capabilities of Llama.
12. ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately)
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of computing to be effective. As an alternative, experts propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, their approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, experts train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not.
T5 enables the model to learn from all input tokens instead of the small masked-out subset. It is not adversarial, despite the similarity to GAN, as the generator producing tokens for replacement is trained with maximum likelihood. Because of its computational efficiency, the ELECTRA.
13. DeBERTa (Decoding-enhanced BERT with disentangled attention)
The authors from Microsoft Research propose DeBERTa, with two main improvements over BERT, namely disentangled attention and an enhanced mask decoder. DeBERTa has two vectors representing a token/word by encoding content and relative position respectively. The self-attention mechanism in DeBERTa processes self-attention of content-to-content, content-to-position, and also position-to-content, while the self-attention in BERT is equivalent to only have the first two components.
The authors hypothesize that position-to-content self-attention is also needed to comprehensively model relative positions in a sequence of tokens. Furthermore, DeBERTa is equipped with an enhanced mask decoder, where the absolute position of the token/word is also given to the decoder along with the relative information. A single scaled-up variant of DeBERTa surpasses the human baseline on the SuperGLUE benchmark for the first time. The ensemble DeBERTa is the top-performing method on SuperGLUE at the time of this publication.
14. ELMO (Embeddings from Language Models)
ELMo, short for "Embeddings from Language Models," is used to create word embeddings, which are numerical representations of words, but what sets ELMo apart is its keen ability to capture the context and significance of words within sentences.
Unlike traditional word embeddings, like Word2Vec or GloVe, which assign fixed vectors to words regardless of context, ELMo takes a more dynamic approach. It grasps the context of a word by considering the words that precede and follow it in a sentence, thus delivering a more nuanced understanding of word meanings.
The architecture of ELMo is deep and bidirectional. This means it employs multiple layers of recurrent neural networks (RNNs) to analyze the input sentence from both directions – forward and backward. This bidirectional approach ensures that ELMo comprehends the complete context surrounding each word, which is crucial for a more accurate representation.
Moreover, they can be fine-tuned for specific NLP tasks, such as sentiment analysis, named entity recognition, or machine translation, to achieve excellent results.
15. UniLM (Unified Language Model)
UniLM, or the Unified Language Model, is an advanced language model developed by Microsoft Research. What sets it apart is its ability to handle a variety of language tasks without needing specific fine-tuning for each task. This unified approach simplifies the use of NLP technology across various business applications.
Key to UniLM's effectiveness is its bidirectional transformer architecture, which allows it to understand the context of words in sentences from both directions. This comprehensive understanding is essential for tasks like text generation, translation, text classification, and summarization. It can streamline complex processes such as document categorization and text analysis, making them more efficient and accurate.
Building an AI Application with Pre-Trained NLP Models
The importance and advantages of pre-trained language models are quite clear. Thankfully, developers have access to these models that helps them to achieve precise output, save resources, and time of AI application development.
But, which NLP language model works best for your AI project? Well, the answer to that depends upon the scale of the project, type of dataset, training methodologies, and several other factors. To understand which NLP language model will help your project to achieve maximum accuracy and reduce its time to market, you can connect with our AI experts.
For that, you can set up a free consultation session with them wherein they will be guiding you with the right approach to the development of your AI-based application.
