Notation and Vocabulary

Before diving into the math, let’s establish a common vocabulary. In machine learning and neural networks, you’ll frequently encounter:

• Scalars: Single numbers (e.g., $x = 5$)
• Vectors: Ordered lists of numbers, denoted as $\mathbf{v} = [v_1, v_2, \dots, v_n]$
• Matrices: 2D arrays of numbers, denoted as $\mathbf{A}$
• Functions: Mappings from inputs to outputs, $f: \mathbb{R}^n \to \mathbb{R}^m$
• Parameters: Values we learn during training (weights, biases)

This notation becomes our shared language for describing computations in neural networks.

Linear Algebra: Vectors and Matrices

Linear algebra is the workhorse of machine learning. Every computation in a neural network can be expressed as operations on vectors and matrices.

Vectors represent data points. For example, an image with 784 pixels can be represented as a vector $\mathbf{x} \in \mathbb{R}^{784}$.

Matrices represent transformations. A weight matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$ transforms an $n$-dimensional input into an $m$-dimensional output.

Key operations:

1. Vector addition: $\mathbf{u} + \mathbf{v} = [u_1+v_1, u_2+v_2, \dots]$
2. Scalar multiplication: $c\mathbf{v} = [cv_1, cv_2, \dots]$
3. Dot product: $\mathbf{u} \cdot \mathbf{v} = \sum_i u_i v_i$
4. Matrix multiplication: $(\mathbf{A}\mathbf{B})_{ij} = \sum_k A_{ik} B_{kj}$

The dot product is particularly important—it measures similarity between vectors and is the fundamental operation in neural networks.

Why it matters: Neural networks transform data through sequences of linear transformations (matrix multiplications). Understanding these operations helps us understand what the network is learning.

Basic Probability: Random Variables and Distributions

Neural networks operate in a world of uncertainty. Probability theory gives us tools to model and reason about this uncertainty.

Random Variables: Variables whose values depend on random outcomes. For example, $X$ could represent the pixel values in an image, which vary randomly based on the image content.

Distributions: Functions that describe the likelihood of different outcomes:

• Bernoulli distribution: $P(X=1) = p, P(X=0) = 1-p$ (binary outcomes)
• Gaussian (Normal) distribution: $f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ (bell curve)

Key concepts:

1. Expected value: $\mathbb{E}[X] = \sum_x x \cdot P(X=x)$ (the average value)
2. Variance: $\text{Var}(X) = \mathbb{E}[(X-\mathbb{E}[X])^2]$ (spread of the distribution)
3. Independence: Two events are independent if $P(A \cap B) = P(A)P(B)$

Why it matters: Many neural network outputs are probabilistic (e.g., classification probabilities). We also use probability to measure uncertainty, regularize models, and generate diverse outputs.

Derivatives and Gradients

To train neural networks, we need to optimize functions. Derivatives tell us how functions change.

Derivative: The rate of change of a function at a point: $f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$

For a function $f(x) = x^2$, the derivative is $f'(x) = 2x$. This tells us that at $x=3$, the function is increasing at a rate of 6.

Gradient: The generalization of derivatives to multivariable functions. For $f: \mathbb{R}^n \to \mathbb{R}$, the gradient is:

$\nabla f(\mathbf{x}) = \left[\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \dots, \frac{\partial f}{\partial x_n}\right]$

The gradient points in the direction of steepest ascent—this is crucial for optimization.

Example: For $f(x, y) = x^2 + y^2$, $\nabla f = [2x, 2y]$. At point $(1, 2)$, the gradient is $[2, 4]$, pointing in the direction of fastest increase.

Chain Rule: When functions are composed, we can compute derivatives using:

$\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)$

This rule is the mathematical backbone of backpropagation, which we’ll cover in Phase 3.

Why it matters: Neural networks learn by adjusting parameters to minimize a loss function. The gradient tells us which direction to adjust each parameter to reduce the loss.

Common Loss Functions

Mean Squared Error (MSE): For regression problems

$\mathcal{L}_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2$

Where $y_i$ is the true value and $\hat{y}_i$ is the predicted value. The difference is squared to penalize large errors more heavily.

Cross-Entropy Loss: For classification problems

$\mathcal{L}_{\text{CE}} = -\sum_{i=1}^C y_i \log(\hat{y}_i)$

Where $C$ is the number of classes, $y_i$ is the true label (one-hot encoded), and $\hat{y}_i$ is the predicted probability. This loss heavily penalizes confident wrong predictions.

Example: If we’re classifying images as “cat”, “dog”, or “bird”, and the true label is “cat” ($y = [1, 0, 0]$), a prediction of $\hat{y} = [0.9, 0.05, 0.05]$ would have low loss, while $\hat{y} = [0.1, 0.8, 0.1]$ would have high loss.

Why it matters: The loss function is our objective. Everything in training is about minimizing this function. A good loss function accurately reflects what we care about in the real world.

Intuition

Imagine you’re on a mountain in thick fog. You want to reach the lowest point (the valley), but you can’t see the terrain. What do you do?

1. Feel the ground around you to determine which direction slopes downward
2. Take a step in that direction
3. Repeat until you reach the bottom

This is gradient descent: the loss landscape is our mountain, the gradient points downhill, and each step is a parameter update.

The Algorithm

For a parameter vector $\boldsymbol{\theta}$ (all weights and biases in the network):

1. Compute gradient:

$\nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta})$

2. Update parameters:

$\boldsymbol{\theta}_{\text{new}} = \boldsymbol{\theta} - \eta \cdot \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta})$

3. Repeat until convergence

The learning rate $\eta$ (eta) controls step size:

• Too small: Learning is very slow
• Too large: We might overshoot or diverge

Example: One-Dimensional Case

Let $L(\theta) = (\theta - 3)^2$. We want to find $\theta$ that minimizes this.

The gradient is: $\frac{dL}{d\theta} = 2(\theta - 3)$

Starting at $\theta = 0$ with learning rate $\eta = 0.1$:

• Step 1: Gradient $= 2(0 - 3) = -6$, update: $\theta_{\text{new}} = 0 - 0.1(-6) = 0.6$
• Step 2: Gradient $= 2(0.6 - 3) = -4.8$, update: $\theta_{\text{new}} = 0.6 - 0.1(-4.8) = 1.08$
• Step 3: Gradient $= 2(1.08 - 3) = -3.84$, update: $\theta_{\text{new}} = 1.08 - 0.1(-3.84) = 1.468$

We’re moving toward $\theta = 3$, the true minimum!

Stochastic Gradient Descent (SGD)

In practice, we use stochastic gradient descent: instead of computing the gradient over all training data, we use a small batch (mini-batch).

$\nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}) \approx \frac{1}{|B|} \sum_{i \in B} \nabla_{\boldsymbol{\theta}} \mathcal{L}_i(\boldsymbol{\theta})$

Where $B$ is a random batch of training examples.

Why SGD?

• Computationally efficient (don’t need all data for each update)
• Noisy updates help escape local minima
• Works better with large datasets

Advanced Optimizers

Modern training rarely uses vanilla SGD. Popular variants include:

• Momentum: Accumulates past gradients to accelerate through flat regions
• Adam: Combines momentum with adaptive learning rates
• RMSprop: Adapts learning rates per parameter

These build on the same gradient descent principle but use the gradient more intelligently.

Gradients Guide Learning

Each gradient component $\frac{\partial \mathcal{L}}{\partial \theta_i}$ tells us:

• How much the loss would change if we increased $\theta_i$ by a tiny amount
• Which direction to move $\theta_i$ to reduce loss

A large positive gradient means “decrease this parameter to reduce loss,” while a large negative gradient means “increase this parameter to reduce loss.”

Vanishing and Exploding Gradients

Two common problems in deep networks:

Vanishing gradients: In very deep networks, gradients can become extremely small, making early layers learn very slowly. This was a major challenge in early neural networks.

Exploding gradients: Gradients can become extremely large, causing parameters to change drastically and training to become unstable.

Solutions include:

• Careful initialization (e.g., Xavier, He initialization)
• Activation functions with well-behaved gradients (we’ll see these in Phase 3)
• Gradient clipping
• Residual connections (skip connections)

Gradient Flow Through the Network

In a deep network, gradients flow backward from the output to the input:

1
Input → [Layer 1] → [Layer 2] → ... → [Output]
2
              ↑           ↑           ↑
3
           Gradient   Gradient   Gradient (starts here)

If gradients vanish early, the early layers (which often learn fundamental features) don’t get useful training signals. If gradients explode, training becomes unstable.

Why it matters: Understanding gradient flow helps us design better architectures. Modern breakthroughs like residual networks were motivated specifically to improve gradient flow.

Structure

A perceptron takes $n$ inputs and produces one output:

1
Inputs: x₁, x₂, ..., xₙ
2
       ↓   ↓       ↓
3
Weights: w₁, w₂, ..., wₙ
4
       ↓   ↓       ↓
5
       Sum: z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
6
                              ↓
7
                         Activation: a = σ(z)
8
                              ↓
9
                           Output: y = a

Mathematically:

$z = \sum_{i=1}^n w_i x_i + b = \mathbf{w} \cdot \mathbf{x} + b$

$y = \sigma(z)$

Where:

• $\mathbf{w}$ is the weight vector
• $b$ is the bias (allows shifting the activation function)
• $\sigma$ is an activation function
• $z$ is called the “pre-activation”

Geometric Intuition

The computation $z = \mathbf{w} \cdot \mathbf{x} + b$ is a hyperplane in $\mathbb{R}^n$. The perceptron decides which side of this hyperplane the input falls on.

For 2D inputs $(x_1, x_2)$:

• The decision boundary is the line $w_1 x_1 + w_2 x_2 + b = 0$
• Points above the line have $z > 0$, below have $z < 0$

Limitations of Single Perceptrons

A single perceptron can only learn linearly separable functions. It cannot solve the XOR problem, which requires a non-linear decision boundary.

Solution: Stack multiple perceptrons to create a multi-layer perceptron (MLP).

Architecture

1
Input Layer    Hidden Layer 1    Hidden Layer 2    Output Layer
2
  [x₁] ──────►    [h₁₁] ──────►    [h₂₁] ──────►    [y₁]
3
  [x₂] ──────►    [h₁₂] ──────►    [h₂₂] ──────►    [y₂]
4
  [x₃] ──────►    [h₁₃] ──────►    [h₂₃] ──────►    [y₃]
5
                 ...                 ...

Each layer transforms the representation: $\mathbf{h}^{(l)} = \sigma(\mathbf{W}^{(l)} \mathbf{h}^{(l-1)} + \mathbf{b}^{(l)})$

Where $\mathbf{h}^{(0)} = \mathbf{x}$ (input) and $\mathbf{h}^{(L)} = \mathbf{y}$ (output).

Why multiple layers?: Each layer learns increasingly abstract features. In image recognition, early layers might detect edges, middle layers detect shapes, and later layers detect objects.

Universal Approximation Theorem

A remarkable theoretical result: A feed-forward network with a single hidden layer and non-linear activation can approximate any continuous function to arbitrary accuracy (given enough neurons).

This means neural networks are incredibly flexible function approximators—they can learn virtually any mapping from inputs to outputs given sufficient capacity.

Sigmoid

$\sigma(z) = \frac{1}{1 + e^{-z}}$

Properties:

• Output in $(0, 1)$ (interpretable as probability)
• Smooth gradient: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$
• Historically popular but suffers from vanishing gradients

Use case: Binary classification output

ReLU (Rectified Linear Unit)

$\text{ReLU}(z) = \max(0, z)$

Properties:

• Simple: $f(z) = 0$ if $z < 0$, else $f(z) = z$
• Sparse activation (many neurons output zero)
• Avoids vanishing gradient (gradient is 1 for positive inputs)
• Computationally efficient

Use case: Hidden layers in most modern networks

Why ReLU Won

ReLU’s simplicity and lack of vanishing gradient problems made it the default choice for hidden layers. Variants like Leaky ReLU and ELU address the “dying ReLU” problem (neurons that never activate).

The Chain Rule in Action

For a simple network with one hidden layer:

$\mathcal{L} = \mathcal{L}(y, \hat{y}) = \mathcal{L}(y, \sigma(\mathbf{W}^{(2)} \sigma(\mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}) + \mathbf{b}^{(2)}))$

To find $\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(1)}}$, we chain through the computation:

$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(1)}} = \underbrace{\frac{\partial \mathcal{L}}{\partial \hat{y}}}_{\text{Output layer}} \cdot \underbrace{\frac{\partial \hat{y}}{\partial \mathbf{z}^{(2)}}}_{\text{Output activation}} \cdot \underbrace{\frac{\partial \mathbf{z}^{(2)}}{\partial \mathbf{h}^{(1)}}}_{\text{Hidden to output}} \cdot \underbrace{\frac{\partial \mathbf{h}^{(1)}}{\partial \mathbf{z}^{(1)}}}_{\text{Hidden activation}} \cdot \underbrace{\frac{\partial \mathbf{z}^{(1)}}{\partial \mathbf{W}^{(1)}}}_{\text{Input to hidden}}$

Simplified Backprop Algorithm

1. Forward pass: Store all intermediate values ($\mathbf{z}^{(l)}$, $\mathbf{h}^{(l)}$)
2. Output gradient: Compute $\delta^{(L)} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(L)}}$
3. Backward pass: For each layer $l = L-1$ to $1$: $\delta^{(l)} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l+1)}} \cdot \frac{\partial \mathbf{z}^{(l+1)}}{\partial \mathbf{h}^{(l)}} \cdot \frac{\partial \mathbf{h}^{(l)}}{\partial \mathbf{z}^{(l)}}$
4. Parameter gradients: $\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}} = \delta^{(l)} (\mathbf{h}^{(l-1)})^T$ $\frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}} = \delta^{(l)}$

Computational Efficiency

The key insight of backpropagation is that we compute gradients in $O(N)$ time where $N$ is the number of parameters, rather than $O(N^2)$ for naive numerical differentiation. This makes training large networks feasible.

Example: Computing a Gradient

For a single neuron with input $x$, weight $w$, bias $b$, sigmoid activation, and MSE loss:

1. Forward: $z = wx + b$, $a = \sigma(z)$, $\mathcal{L} = \frac{1}{2}(a - y)^2$
2. Backward: $\frac{\partial \mathcal{L}}{\partial a} = a - y$ $\frac{\partial a}{\partial z} = a(1 - a)$ $\frac{\partial z}{\partial w} = x$ $\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial w} = (a - y) \cdot a(1 - a) \cdot x$

This tells us exactly how to adjust $w$ to reduce the loss.

From One-Hot to Dense Vectors

One-hot encoding: Represent each word as a sparse vector where only one position is 1.

1
"cat"    = [1, 0, 0, 0, ...]
2
"dog"    = [0, 1, 0, 0, ...]
3
"bird"   = [0, 0, 1, 0, ...]

Problems: High dimensionality, no notion of similarity, no relationship between words.

Word embeddings: Learn dense, low-dimensional vectors where semantic similarity corresponds to geometric similarity.

1
"cat"    ≈ [0.2, -0.5, 0.8, 0.1, ...]
2
"dog"    ≈ [0.3, -0.4, 0.7, 0.2, ...]
3
"bird"   ≈ [0.9, 0.3, -0.2, 0.5, ...]

Now “cat” and “dog” have similar vectors (they’re semantically related), while “bird” is different.

Why Embeddings Work

Words that appear in similar contexts should have similar meanings. By learning to predict words from context (or vice versa), we automatically capture semantic relationships.

Famous property: Vector arithmetic captures relationships:

1
king - man + woman ≈ queen

The direction from “man” to “woman” is similar to the direction from “king” to “queen”!

Training Embeddings

Word2Vec: Two approaches:

1. Skip-gram: Predict context words from target word
2. CBOW: Predict target word from context

GloVe: Factorize word co-occurrence matrix

Modern approach: Learn embeddings as part of the model training rather than pre-training separately.

Beyond Words

Embeddings extend beyond words:

• Character embeddings: For subword information
• Positional embeddings: For sequence order
• Sentence embeddings: For entire documents
• Multimodal embeddings: For images, audio, etc.

The Problem with Fixed Representations

Before attention, sequence models (like RNNs) compressed an entire input sequence into a single fixed-size vector:

1
Input: "The cat sat on the mat"
2
↓ (encode entire sentence)
3
Fixed vector: [0.1, -0.3, 0.7, ...]
4
↓ (decode)
5
Output: "El gato se sentó en la alfombra"

Problem: All information must be compressed into one vector, which becomes a bottleneck for long sequences.

The Attention Solution

Instead of using a fixed vector, let each output step look at different parts of the input, weighted by relevance.

1
When generating "El"    → Focus on "The"
2
When generating "gato" → Focus on "cat"
3
When generating "se"    → Focus on "sat"

Mathematically, for output position $i$, we compute a weighted sum of input representations:

$\mathbf{c}_i = \sum_{j} \alpha_{ij} \mathbf{h}_j$

Where $\alpha_{ij}$ is the attention weight from output position $i$ to input position $j$.

Computing Attention Weights

How do we compute $\alpha_{ij}$? We use a compatibility function that measures how well input position $j$ relates to output position $i$.

Dot-product attention:

$e_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j$

$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$

Where:

• $\mathbf{q}_i$ is the query vector (what we’re looking for)
• $\mathbf{k}_j$ is the key vector (what the input offers)
• $\mathbf{h}_j$ is the value vector (the actual content)

The softmax ensures weights sum to 1 (probability distribution).

Intuition: Database Analogy

Think of it like a database query:

• Query (Q): What you’re searching for
• Key (K): How items are indexed
• Value (V): The actual content

1
Database:
2
  Key: "name", Value: "Alice"
3
  Key: "age", Value: 30
4
  Key: "city", Value: "NYC"
5

6
Query: "name"
7
→ Match: Key "name" matches Query "name"
8
→ Return: Value "Alice"

In Neural Networks

Each input position has its own K and V. Each output position has its own Q. We compute similarity between Q and all K’s, then weight V’s by these similarities.

Q, K, V in Code

1
def scaled_dot_product_attention(Q, K, V):
2
    """
3
    Args:
4
        Q: (batch_size, seq_len, d_k)
5
        K: (batch_size, seq_len, d_k)
6
        V: (batch_size, seq_len, d_v)
7
    Returns:
8
        output: (batch_size, seq_len, d_v)
9
        attention_weights: (batch_size, seq_len, seq_len)
10
    """
11
    # Compute similarity scores
12
    scores = Q @ K.transpose(-2, -1)  # (batch_size, seq_len, seq_len)
13

14
    # Scale to prevent extremely large softmax values
15
    scores = scores / np.sqrt(K.shape[-1])
16

17
    # Apply softmax to get attention weights
18
    attention_weights = softmax(scores, axis=-1)
19

20
    # Weight values by attention weights
21
    output = attention_weights @ V
22

23
    return output, attention_weights

Key steps:

1. Compute Q·K to get similarity scores
2. Scale by $\sqrt{d_k}$ (stabilizes gradients)
3. Apply softmax to get probability distribution
4. Weight sum of V’s

Multi-Head Attention

Instead of one set of Q, K, V, we use multiple “heads” that learn different relationships:

1
class MultiHeadAttention:
2
    def __init__(self, d_model, num_heads):
3
        self.d_model = d_model
4
        self.num_heads = num_heads
5
        self.d_k = d_model // num_heads
6

7
        # Learnable projection matrices
8
        self.W_Q = initialize_weights((d_model, d_model))
9
        self.W_K = initialize_weights((d_model, d_model))
10
        self.W_V = initialize_weights((d_model, d_model))
11
        self.W_O = initialize_weights((d_model, d_model))
12

13
    def forward(self, x):
14
        batch_size, seq_len, _ = x.shape
15

16
        # Project to Q, K, V for each head
17
        Q = x @ self.W_Q  # (batch, seq_len, d_model)
18
        K = x @ self.W_K
19
        V = x @ self.W_V
20

21
        # Split into heads and reshape
22
        Q = Q.reshape(batch_size, seq_len, self.num_heads, self.d_k)
23
        K = K.reshape(batch_size, seq_len, self.num_heads, self.d_k)
24
        V = V.reshape(batch_size, seq_len, self.num_heads, self.d_k)
25

26
        # Transpose to (batch, num_heads, seq_len, d_k)
27
        Q = Q.transpose(1, 2)
28
        K = K.transpose(1, 2)
29
        V = V.transpose(1, 2)
30

31
        # Compute attention for each head
32
        scores = Q @ K.transpose(-2, -1) / np.sqrt(self.d_k)
33
        attention_weights = softmax(scores, axis=-1)
34
        attended = attention_weights @ V
35

36
        # Concatenate heads and project
37
        attended = attended.transpose(1, 2).reshape(batch_size, seq_len, self.d_model)
38
        output = attended @ self.W_O
39

40
        return output, attention_weights

Each head can focus on different types of relationships:

• Head 1 might focus on syntactic structure
• Head 2 might focus on semantic similarity
• Head 3 might focus on positional relationships

Key Innovations

1. Self-attention: Instead of attention between encoder and decoder, each position attends to all other positions in the same sequence.

1
For each word "cat", compute attention weights with all words:
2
"cat" attends to: "The"(0.1), "cat"(0.6), "sat"(0.2), "on"(0.05), "the"(0.03), "mat"(0.02)

2. Positional encoding: Since attention doesn’t inherently capture order, we add position information:

$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})$

$PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})$

These sinusoidal encodings allow the model to learn relative positions.

3. Parallelization: Unlike RNNs which process sequentially, transformers process all positions simultaneously, enabling massive parallelization on GPUs.

Transformer Architecture

1
Input Embedding + Positional Encoding
2
           ↓
3
    ┌─────────────────┐
4
    │  Encoder Stack  │
5
    │  (N times)      │
6
    │                 │
7
    │  ┌───────────┐  │
8
    │  │ Multi-Head│  │
9
    │  │ Attention │  │
10
    │  └───────────┘  │
11
    │        ↓         │
12
    │  ┌───────────┐  │
13
    │  │ Feed-     │  │
14
    │  │ Forward   │  │
15
    │  └───────────┘  │
16
    └─────────────────┘
17
           ↓
18
    ┌─────────────────┐
19
    │  Decoder Stack  │
20
    │  (N times)      │
21
    │                 │
22
    │  ┌───────────┐  │
23
    │  │ Masked    │  │
24
    │  │ Multi-Head│  │
25
    │  │ Attention │  │
26
    │  └───────────┘  │
27
    │        ↓         │
28
    │  ┌───────────┐  │
29
    │  │ Cross     │  │
30
    │  │ Attention│  │
31
    │  └───────────┘  │
32
    │        ↓         │
33
    │  ┌───────────┐  │
34
    │  │ Feed-     │  │
35
    │  │ Forward   │  │
36
    │  └───────────┘  │
37
    └─────────────────┘
38
           ↓
39
        Output

Encoder: Processes input sequence and produces contextual representations

Decoder: Generates output sequence, attending to both encoder output and previously generated tokens

Masked attention: Prevents decoder from “seeing the future” during training (autoregressive)

Why Transformers Work So Well

1. Long-range dependencies: Attention connects any two positions directly, no matter how far apart
2. Parallelization: Process entire sequences at once, enabling training on massive datasets
3. Interpretability: Attention weights show what the model focuses on
4. Scalability: Performance continues to improve with more data and compute

From Transformers to LLMs

Modern large language models are essentially:

• Decoder-only transformers (for autoregressive text generation)
• Trained on massive text datasets
• Scaled up with more parameters, more data, and more compute

The architecture we’ve discussed is essentially the same as GPT, BERT, and LLaMA—just scaled up.