BERT vs GPT: A Comparison of Transformer Models

BERT and GPT are two of the most influential Transformer models in the field of natural language processing. Let’s explore their differences and their specific applications.

Architecture and How It Works

BERT (Bidirectional Encoder Representations from Transformers)

class BERT(nn.Module):
    def __init__(self, vocab_size, hidden_size, num_layers, num_heads):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.position_encoding = PositionalEncoding(hidden_size)
        self.encoder_layers = nn.ModuleList([
            TransformerEncoderLayer(hidden_size, num_heads)
            for _ in range(num_layers)
        ])
        
    def forward(self, input_ids, attention_mask=None):
        # Embedding and position encoding
        x = self.embedding(input_ids)
        x = self.position_encoding(x)
        
        # Pass through the encoder layers
        for layer in self.encoder_layers:
            x = layer(x, attention_mask)
            
        return x

GPT (Generative Pre-trained Transformer)

class GPT(nn.Module):
    def __init__(self, vocab_size, hidden_size, num_layers, num_heads):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.position_encoding = PositionalEncoding(hidden_size)
        self.decoder_layers = nn.ModuleList([
            TransformerDecoderLayer(hidden_size, num_heads)
            for _ in range(num_layers)
        ])
        
    def forward(self, input_ids, attention_mask=None):
        # Embedding and position encoding
        x = self.embedding(input_ids)
        x = self.position_encoding(x)
        
        # Pass through the decoder layers
        for layer in self.decoder_layers:
            x = layer(x, attention_mask)
            
        return x

Key Differences

1. Attention Direction

• BERT: Bidirectional (sees the full context)
• GPT: Unidirectional (sees only the past context)

2. Architecture

• BERT: Encoder-only
• GPT: Decoder-only

3. Applications

• BERT:

  • Text classification
  • Named entity recognition
  • Question answering
  • Sentiment analysis

• GPT:

  • Text generation
  • Text completion
  • Dialogue
  • Content creation

Usage Examples

BERT for Classification

from transformers import BertTokenizer, BertForSequenceClassification

# Load the model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Prediction
text = "J'adore ce produit !"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs)
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

GPT for Generation

from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load the model and tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Text generation
prompt = "L'intelligence artificielle est"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, num_return_sequences=1)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

Performance and Limitations

BERT

• Strengths:

  • Better understanding of context
  • Excellent performance on classification tasks
  • Well suited to comprehension tasks

• Limitations:

  • Cannot generate text
  • Requires task-specific fine-tuning

GPT

• Strengths:

  • Fluent text generation
  • Creativity in output
  • Well suited to generation tasks

• Limitations:

  • Bias in generated text
  • No bidirectional understanding
  • Can produce incorrect information

Conclusion

BERT and GPT represent two different but complementary approaches to natural language processing. The choice between them depends on the specific needs of your application.

Additional Resources

• BERT Paper
• GPT Paper
• Hugging Face Transformers