Natural Language Processing with Transformers and Hugging Face

How Transformers Changed NLP

To appreciate why transformers matter, you need to understand what came before them. Recurrent Neural Networks (RNNs) and their improved variants, LSTMs and GRUs, were the state of the art for sequential text processing throughout most of the 2010s. These architectures processed tokens one at a time, maintaining a hidden state that theoretically captured everything seen so far. In practice, they suffered from a fundamental limitation: the further back in the sequence a relevant piece of information was, the less influence it had on the current output. This "vanishing gradient" problem meant that understanding long-range dependencies was genuinely difficult.

Transformers replaced sequential processing entirely. Instead of reading text left-to-right and maintaining a running memory, transformers process all tokens simultaneously. Every word in a sentence attends to every other word in a single forward pass. This architectural choice had three enormous consequences. First, transformers can capture long-range dependencies without degradation, a word at position 500 relates to a word at position 1 just as easily as position 499. Second, because all positions are computed in parallel, transformers train dramatically faster on modern GPU hardware. Third, the self-attention mechanism is interpretable: you can literally inspect which tokens attend to which, giving you insight into what the model learned.

The "Attention Is All You Need" paper proved that you don't need recurrence or convolution to build powerful sequence models. Attention alone, applied repeatedly across many layers, is sufficient to learn language at superhuman levels on benchmarks. Every major NLP model since 2018 is built on this insight. Understanding this history tells you why, when someone asks "should I use an LSTM or a transformer?", the answer is almost always transformer.

The Transformer Revolution

Here's the thing about transformers: they cracked the code on context. Before transformers, neural networks processed text sequentially, one word at a time. That meant "bank" in "a river bank" and "bank" in "visit the bank" required the model to remember everything that came before. What if the previous sentence was 500 words ago? Good luck.

Transformers flipped the script with self-attention. Instead of processing words one-by-one, self-attention lets every word look at every other word simultaneously. Each word computes relationships with all others, understanding context from both directions. That's why BERT reads the whole sentence before deciding what a word means.

Attention Mechanism Intuition

The attention mechanism sounds abstract until you ground it in something concrete, so let's try a different angle. Imagine you're reading a sentence and trying to understand the pronoun "it" in "The trophy didn't fit in the suitcase because it was too big." Without any other context, "it" is ambiguous, does it refer to the trophy or the suitcase? A human reader resolves this instantly by reasoning about size: a trophy that "doesn't fit" in something implies the trophy is the bigger object. The attention mechanism does the same thing computationally.

When the transformer processes the word "it," the attention mechanism computes a score between "it" and every other word in the sentence. Words that are semantically relevant to resolving "it" receive high attention scores; irrelevant words receive low scores. The mechanism learns, through training on millions of examples, that pronouns typically attend strongly to candidate referents, and that physical constraints like "fit" modulate which referent is most plausible. This is why self-attention is described as "dynamic", unlike static word embeddings, the representation of "it" changes depending on the full sentence context.

The multi-head version extends this: rather than computing one set of attention relationships, you compute several in parallel. One head might specialize in tracking syntactic dependencies (subjects attending to verbs), another in coreference resolution (pronouns attending to nouns), another in semantic similarity (synonyms and related words attending to each other). The final representation of each word is a weighted combination of these different relational views. This is why transformer representations are so much richer than anything RNNs produced, each token's embedding captures multiple simultaneous relational contexts, not just a single compressed sequential summary.

The Self-Attention Mechanism

Let's unpack this. Self-attention answers a deceptively simple question: What other words matter for understanding this word?

Imagine the sentence: "The cat sat on the mat because it was warm."

When processing "it," the model needs to know that "it" refers to "mat," not "cat." Self-attention computes three things for each word:

1. Query (Q): "What am I looking for?"
2. Key (K): "What information do I contain?"
3. Value (V): "What do I contribute?"

The model multiplies Q against all Ks to find relevant words, then extracts their Values. The word "it" queries, and the word "mat" has a strong key match, boom, "it" learned its antecedent.

Mathematically:

Attention(Q, K, V) = softmax(Q × K^T / √d_k) × V

The division by √d_k scales things so the softmax doesn't collapse into one-hot distributions. Without this scaling factor, the dot products can grow very large in high-dimensional spaces, pushing the softmax into regions where gradients vanish, the division keeps training stable.

Multi-Head Attention

One attention head isn't enough. What if one head tracks grammar, another tracks sentiment, another tracks entities? That's multi-head attention. Instead of one set of Q, K, V matrices, we train multiple sets in parallel. Each head learns different relationships. Then we concatenate all heads and project the result.

# Conceptually:
attention_heads = [attention(Q_i, K_i, V_i) for i in range(num_heads)]
output = W_o @ concatenate(attention_heads)

Real transformers use 8, 12, or 16 heads. It's like having multiple perspectives on the same problem. The projection matrix W_o then combines these diverse perspectives into a single unified representation that downstream layers can work with.

Positional Encoding

Here's a problem: if words are just tokens, and self-attention sees them all at once, how does the model know what comes first? "The cat sat" vs. "sat cat the" would look identical.

Transformers solve this with positional encoding, learnable vectors added to word embeddings. Position 0 gets one encoding, position 1 gets another. The model learns that position matters.

Early transformers used sinusoidal encodings (fixed math formulas). Modern approaches learn encodings. Either way, the model knows word order.

BERT vs GPT: Encoder vs Decoder

Now things get spicy. BERT and GPT are both transformer-based, but they're architecturally opposite. Understanding this split determines what you use when.

BERT: The Bidirectional Encoder

BERT reads text left-to-right and right-to-left simultaneously. During training, BERT randomly masks 15% of tokens and predicts them from context. This forces the model to understand bidirectional relationships.

BERT excels at:

• Text classification
• Named entity recognition
• Sentiment analysis
• Question answering
• Any task needing full-context understanding

The catch? BERT can't generate text. It's an encoder. You feed it text and get back hidden representations.

GPT: The Autoregressive Decoder

GPT reads left-to-right only. It predicts the next token based on previous tokens. During training, this "causal masking" ensures the model can't cheat by looking ahead.

GPT excels at:

• Text generation
• Chat/conversational tasks
• Completion-based applications
• Language modeling
• Anything involving generating new text

GPT is autoregressive, it generates one token at a time, feeding predictions back in.

Which Do I Use?

• Sentiment classification? BERT.
• Chatbot? GPT.
• Named entity recognition? BERT.
• Story generation? GPT.
• Semantic similarity? BERT.
• Code completion? GPT.

Simple rule: if you're understanding existing text, use an encoder (BERT). If you're creating new text, use a decoder (GPT).

The Hugging Face Ecosystem

The Hugging Face ecosystem is what transforms transformer research from academic papers into production-ready code. Founded in 2016 and initially building a chatbot app, Hugging Face pivoted to becoming the infrastructure layer for the entire ML community, and that pivot paid off enormously. Today, the Hugging Face Hub hosts over 500,000 models, 50,000 datasets, and 50,000 demo applications. It is, without exaggeration, the GitHub of machine learning.

The transformers library is the centerpiece. It provides unified APIs for loading, running, and fine-tuning thousands of pre-trained models across dozens of architectures. Before this library existed, using BERT meant navigating Google's original TensorFlow implementation, which was notoriously difficult to adapt. Using GPT-2 meant OpenAI's PyTorch code, structured differently. Every new architecture meant a new codebase to understand. Hugging Face standardized the interface: AutoTokenizer, AutoModel, and the pipeline() abstraction work the same way whether you're loading a tiny DistilBERT or a massive Falcon-180B. This consistency dramatically reduces the cognitive overhead of experimenting with different architectures. You can swap models by changing a single string, not by rewriting your preprocessing pipeline.

The ecosystem also includes the datasets library for efficient data loading at scale, evaluate for standardized metrics, accelerate for distributed training, and PEFT (Parameter-Efficient Fine-Tuning) for LoRA and other adapter-based techniques. These libraries are designed to compose, you can combine them freely, and they integrate with both PyTorch and JAX/Flax. If you're doing any serious NLP work in 2024 and beyond, Hugging Face is your starting point, not an optional add-on.

Okay, enough theory. Let's build something. Hugging Face is the go-to library because it:

1. Abstracts complexity: No need to implement attention from scratch
2. Provides pretrained models: BERT, GPT-2, RoBERTa, DistilBERT, and hundreds more
3. Handles tokenization: Different models need different tokenizers
4. Simplifies training: The Trainer class handles boilerplate
5. Hub integration: Share and discover models instantly

Zero-Code NLP with pipeline()

The pipeline() function is your fastest path from text to predictions. Under the hood it handles model selection, tokenizer initialization, device placement, and output post-processing, all the boilerplate you'd otherwise write by hand. It's an excellent starting point for prototyping, and for many production use cases, it's genuinely all you need. Want to do NLP without thinking about architecture?

from transformers import pipeline
 
# Sentiment analysis
classifier = pipeline("sentiment-analysis")
result = classifier("I absolutely love this product!")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.9991}]
 
# Named entity recognition
ner = pipeline("ner")
entities = ner("Apple Inc. was founded by Steve Jobs in Cupertino.")
print(entities)
# Output: [{'entity': 'B-ORG', 'score': 0.999, 'index': 1, 'word': 'Apple'},
#          {'entity': 'B-PER', 'score': 0.997, 'index': 5, 'word': 'Steve'},
#          {'entity': 'I-PER', 'score': 0.996, 'index': 6, 'word': 'Jobs'}, ...]
 
# Zero-shot classification
zero_shot = pipeline("zero-shot-classification")
result = zero_shot("This course teaches deep learning",
                    candidate_labels=["education", "sports", "tech"])
print(result)
# Output: {'sequence': 'This course teaches deep learning',
#          'labels': ['tech', 'education', 'sports'],
#          'scores': [0.95, 0.03, 0.02]}

No model selection, no tokenization code, no training loop. The pipeline picks an appropriate pretrained model, handles tokenization, and returns predictions. The zero-shot classification example is particularly noteworthy, you're classifying into categories the model has never explicitly been trained on, using only its general language understanding. This is democratization in action.

Tokenizers: Breaking Text into Pieces

Not all tokenizers are equal. Different models expect different tokenization strategies:

Byte-Pair Encoding (BPE): Merges frequent byte pairs. Used by GPT-2, GPT-3. WordPiece: Like BPE but marks subword pieces with ##. Used by BERT. SentencePiece: Language-agnostic, handles multilingual models well.

Why does this matter? Tokenizers affect:

• Vocabulary size
• How words break into subwords
• Special tokens ([CLS], [SEP], [PAD], [UNK])
• Encoding efficiency

Getting the tokenizer wrong is one of the most common sources of subtle bugs in NLP pipelines. If you load a model trained with WordPiece tokenization but accidentally apply a BPE tokenizer, you'll get different token IDs than the model expects, and performance will silently degrade. Always match your tokenizer to your model, AutoTokenizer does this for you automatically, which is another reason to use it.

from transformers import AutoTokenizer
 
# BERT tokenizer
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = bert_tokenizer.tokenize("Hello, world!")
print(tokens)
# Output: ['hello', ',', 'world', '!']
 
# Encode to token IDs
ids = bert_tokenizer.encode("Hello, world!")
print(ids)
# Output: [101, 7592, 1010, 2088, 999, 102]
 
# Special tokens
print(bert_tokenizer.special_tokens_map)
# Output: {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]',
#          'cls_token': '[CLS]', 'mask_token': '[MASK]'}

The [CLS] token is special, it's prepended to every sequence and contains aggregated sentence-level information. [SEP] separates two sentences. [PAD] fills shorter sequences to batch size. The token IDs 101 and 102 at the start and end of the encoded output are [CLS] and [SEP] respectively, BERT adds them automatically. Understanding these matters when debugging mismatches between expected and actual model inputs.

AutoModel and AutoTokenizer: The Swiss Army Knife

The Auto classes represent one of Hugging Face's best design decisions. Rather than requiring you to know the exact class name for each architecture, BertForSequenceClassification vs RobertaForSequenceClassification vs DistilBertForSequenceClassification, the Auto classes inspect the model card, detect the architecture, and instantiate the right class automatically. This means you can experiment with different base models by changing a single string, which is exactly what you want when evaluating whether RoBERTa outperforms BERT on your specific dataset.

from transformers import AutoModel, AutoTokenizer
 
# Load ANY pretrained model by name
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
 
# Works for GPT-2 too
gpt_tokenizer = AutoTokenizer.from_pretrained("gpt2")
gpt_model = AutoModel.from_pretrained("gpt2")
 
# The model's last layer is learned, the rest are pretrained
# No redownloading, no name changes

AutoModel and AutoTokenizer inspect the model name, figure out the architecture, and load it. This abstraction is beautiful because you can swap models without changing code. The first time you call from_pretrained(), weights download from the Hub and cache locally. Subsequent calls use the cache, no network required.

Fine-Tuning BERT for Text Classification

Now the real work. Fine-tuning is what makes transformer models practically useful. You're not training from scratch, that would require massive compute budgets. Instead, you're taking a model that already understands language at a deep level and teaching it the specific vocabulary and judgment of your task. For sentiment analysis, this means the model already knows what words like "incredible" and "disappointing" mean, you're just teaching it to map that understanding to your particular label schema. Let's fine-tune BERT for sentiment classification on a custom dataset.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset, Dataset
import numpy as np
 
# 1. Load tokenizer and model
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
 
# 2. Prepare dataset
raw_data = {
    'text': [
        'This movie is amazing!',
        'I hated it, waste of time.',
        'Pretty good, would watch again.',
        'Terrible acting, boring plot.'
    ],
    'label': [1, 0, 1, 0]  # 1=positive, 0=negative
}
dataset = Dataset.from_dict(raw_data)
 
# 3. Tokenize
def preprocess(examples):
    return tokenizer(examples['text'], truncation=True, padding='max_length', max_length=128)
 
tokenized = dataset.map(preprocess, batched=True)
 
# 4. Split train/val
train_val = tokenized.train_test_split(test_size=0.2)
 
# 5. Define training arguments
args = TrainingArguments(
    output_dir='./bert_sentiment',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    learning_rate=2e-5,
    evaluation_strategy='epoch',
    save_strategy='epoch'
)
 
# 6. Create trainer and train
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_val['train'],
    eval_dataset=train_val['test']
)
 
trainer.train()
 
# 7. Predict on new text
def predict_sentiment(text):
    inputs = tokenizer(text, return_tensors='pt', truncation=True, padding='max_length', max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    logits = outputs.logits
    predicted_class = torch.argmax(logits, dim=-1).item()
    return "Positive" if predicted_class == 1 else "Negative"
 
print(predict_sentiment("This is the best day ever!"))
# Output: Positive

Notice the pattern: AutoModelForSequenceClassification automatically adds a classification head on top of BERT. The pretrained weights encode BERT's language understanding; the randomly initialized classification head is what you're training. The Trainer handles the optimization loop, and learning_rate=2e-5 is a deliberate choice, low enough not to catastrophically forget BERT's pretrained knowledge, high enough to meaningfully update the new head.

The Trainer class handles:

• Forward/backward passes
• Gradient accumulation
• Learning rate scheduling
• Evaluation loops
• Model checkpointing

You don't write training code. You just define arguments and let Trainer run.

Fine-Tuning Best Practices

Fine-tuning transformers is not like training a neural network from scratch, and the differences matter. Get these details right and you'll converge faster to better results; ignore them and you'll wonder why your model learns nothing or forgets everything it knew.

The learning rate is your most critical hyperparameter. BERT and RoBERTa fine-tune well at 2e-5 to 5e-5. Higher rates cause catastrophic forgetting, the model overwrites its pretrained weights instead of adapting them. Lower rates mean painfully slow convergence. If your model isn't improving after one epoch, try bumping the learning rate slightly. If it's losing all performance on general language tasks, it's too high.

Batch size and gradient accumulation interact. Transformers often perform better with larger effective batch sizes, but GPU memory limits your per-device batch size. Use gradient_accumulation_steps in TrainingArguments to simulate larger batches: with 4 accumulation steps and a per-device batch of 8, your effective batch size is 32. For most classification tasks, effective batch sizes of 16–32 work well. For tasks with very long sequences, you may need to drop to 8 or 4 and increase accumulation.

Three to five epochs is usually sufficient for fine-tuning. Unlike training from scratch, transformer fine-tuning converges quickly because the model starts from a highly capable state. Running too many epochs causes overfitting, especially on small datasets. Use evaluation_strategy='epoch' and monitor your validation loss, stop when it stops improving. The load_best_model_at_end=True flag in TrainingArguments handles this automatically.

For small datasets (fewer than 1,000 examples), consider using DistilBERT instead of full BERT. DistilBERT is 40% smaller and 60% faster, while retaining 97% of BERT's performance on most tasks. The smaller model is less likely to overfit on limited data, and it's faster to iterate with during development. Reserve full BERT or RoBERTa for larger datasets where you need the extra capacity.

When working with domain-specific text, medical records, legal documents, financial reports, consider starting from a domain-adapted base model rather than vanilla BERT. Models like BioBERT (biomedical text), LegalBERT (legal text), and FinBERT (financial text) have continued pretraining on in-domain corpora, giving you a much better starting point for specialized applications. The Hugging Face Hub makes finding these variants straightforward, search for your domain alongside "BERT" and you'll typically find several options.

The Datasets Library: Beyond pandas

The datasets library is built for machine learning. Unlike pandas DataFrames, it streams data, handles caching, and integrates seamlessly with transformers. The key difference from pandas isn't just performance, it's the lazy evaluation model. When you call .map() on a Hugging Face dataset, you're defining a transformation to be applied, not actually applying it yet. The computation happens when you iterate. This means you can compose many transformations without materializing intermediate results in memory.

from datasets import load_dataset
 
# Load from Hub
dataset = load_dataset('imdb')
print(dataset)
# Output: DatasetDict({
#     train: Dataset({features: ['text', 'label'], num_rows: 25000})
#     test: Dataset({features: ['text', 'label'], num_rows: 25000})
# })
 
# Map functions across dataset (lazy evaluation)
def add_length(example):
    example['length'] = len(example['text'].split())
    return example
 
dataset = dataset.map(add_length)
 
# Filter
small_reviews = dataset['train'].filter(lambda x: x['length'] < 100)
 
# Batch operations
def batch_tokenize(batch):
    return tokenizer(batch['text'], truncation=True, padding='max_length', max_length=128)
 
tokenized = dataset.map(batch_tokenize, batched=True)

Datasets are lazy, operations aren't computed until you iterate. This scales to billion-token corpora without memory explosion. The Arrow-based columnar storage format also means random access is fast and serialization is efficient, tokenized datasets cache to disk so re-running your script skips the tokenization step entirely.

Pushing to Hugging Face Hub

After fine-tuning, sharing your model is a first-class workflow in the Hugging Face ecosystem. The Hub acts as version control for model weights, each push creates a new commit, and you can inspect the full history, roll back to previous checkpoints, and compare model cards. More practically, pushing to the Hub means your model becomes instantly accessible to anyone in the world with a Python environment, without requiring them to run your training code.

# Login first (one-time)
from huggingface_hub import login
login()  # Opens browser to get token
 
# Push to Hub
model.push_to_hub("my-sentiment-bert")
tokenizer.push_to_hub("my-sentiment-bert")
 
# Now anyone can load it
from transformers import AutoModel
loaded_model = AutoModel.from_pretrained("your-username/my-sentiment-bert")

Your model is live. Others can use it with:

pipeline("sentiment-analysis", model="your-username/my-sentiment-bert")

When you push, write a model card, a README in the repository that explains what task the model is trained for, what dataset was used, what evaluation metrics it achieves, and any known limitations. The Hugging Face ModelCard class can help generate this programmatically. Good model cards make your work reproducible and help others decide whether your model fits their use case.

Attention Visualization: See What the Model Learned

Attention visualization is more than a curiosity, it's a practical debugging tool. When your model makes an error, visualizing which tokens it attended to can reveal whether the problem is in the input representation, the attention patterns, or the classification head. It's also the fastest way to build intuition about what transformers actually learn, which makes you a better practitioner. Here's where we peek under the hood, let's visualize which tokens attend to which:

import matplotlib.pyplot as plt
import numpy as np
from transformers import AutoTokenizer, AutoModel
 
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
 
# Forward pass with attention output enabled
text = "The cat sat on the mat"
inputs = tokenizer.encode(text, return_tensors='pt')
outputs = model(inputs, output_attentions=True)
 
# Extract attention weights from last layer
attention = outputs.attentions[-1]  # (batch, heads, seq_len, seq_len)
attention = attention[0, 0, :, :].detach().numpy()  # First batch, first head
 
# Decode tokens
tokens = tokenizer.convert_ids_to_tokens(inputs[0])
 
# Plot heatmap
plt.figure(figsize=(10, 8))
plt.imshow(attention, cmap='viridis')
plt.colorbar()
plt.xticks(range(len(tokens)), tokens, rotation=45)
plt.yticks(range(len(tokens)), tokens)
plt.title("BERT Self-Attention (Layer 0, Head 0)")
plt.tight_layout()
plt.savefig('attention_viz.png')
plt.show()
 
# What does "sat" attend to?
sat_idx = tokens.index('sat')
sat_attention = attention[sat_idx]
print(f"\n'{tokens[sat_idx]}' attends to:")
for i, score in enumerate(sat_attention):
    print(f"  {tokens[i]}: {score:.3f}")
# Output: '[CLS]': 0.001, 'the': 0.150, 'cat': 0.200, 'sat': 0.300, 'on': 0.250, ...

This visualization is powerful. You can debug what the model learned, spot attention collapse (when all attention goes to [CLS]), and understand failure modes. One common pathology is when the model attends almost exclusively to [SEP], this usually indicates the task wasn't learned meaningfully. Another is when early layers show broad, diffuse attention while later layers show sharp, task-specific patterns; this is healthy behavior that confirms the hierarchical nature of transformer representations.

Key Takeaways

1. Transformers revolutionized NLP by replacing sequential processing with self-attention, letting words understand context bidirectionally.

2. BERT (encoder) understands text; GPT (decoder) generates text. Pick based on your task.

3. Hugging Face abstracts away complexity. The pipeline() function gives you state-of-the-art NLP in three lines.

4. Fine-tuning is fast. With Trainer and AutoModel, you can adapt pretrained models to your task in minutes.

5. Tokenizers matter. Different models use different tokenization strategies. Understanding this prevents subtle bugs.

6. The Hub is collaborative. Push your models and benefit from community discoveries.

7. Attention visualization reveals what models learn. Use it to debug and build intuition.

Transformers aren't magic, they're elegant mathematics. But with Hugging Face, you don't need a PhD to use them. You need curiosity and a Python environment. Start with pipelines, graduate to fine-tuning, then push to the Hub. The barrier to entry has never been lower.

Where to Go From Here

We've covered a lot of ground: the transformer architecture from first principles, how BERT and GPT differ and why it matters for your task selection, the Hugging Face ecosystem as a unified interface to hundreds of models, and the practical mechanics of fine-tuning, dataset handling, and model sharing. But we've only scratched the surface of what's possible.

The next natural extensions are parameter-efficient fine-tuning techniques like LoRA (Low-Rank Adaptation) and adapters, which let you fine-tune models with a fraction of the trainable parameters, critical when you're working with larger architectures like Llama or Mistral that won't fit in GPU memory under full fine-tuning. The PEFT library from Hugging Face makes these techniques as accessible as the standard Trainer workflow. If you're building production applications, vLLM and TGI (Text Generation Inference) are worth exploring for efficient inference serving.

The field is moving fast, but the core intuitions don't change. Attention is the mechanism by which transformers understand relationships between tokens. The encoder-decoder split determines whether a model comprehends or generates. Pretraining then fine-tuning is the dominant paradigm because it separates the expensive work (learning language) from the cheap work (learning your task). Master these principles and you'll be able to evaluate new architectures as they emerge, understand why they improve on previous work, and make principled decisions about which tool to reach for on each new problem.

The gap between "I understand transformers conceptually" and "I can deploy a fine-tuned transformer to production" used to be enormous. Hugging Face has collapsed that gap to a weekend project for anyone who knows Python and has a few hundred labeled examples. You now have everything you need to be on the right side of that divide.