Neural Networks & Deep Learning – Complete Notes

import tensorflow as tf
from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),      # input: 784 pixels
    layers.Dense(128, activation='relu'),       # hidden layer 1
    layers.Dense(64,  activation='relu'),       # hidden layer 2
    layers.Dense(10,  activation='softmax')     # output: 10 classes
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# model.fit(x_train, y_train, epochs=5)

• Automatic feature learning from raw data.
• Universal approximators — can approximate any continuous function with enough units.
• Scales to millions of parameters and examples with GPUs/TPUs.
• Transferable — pretrained models reuse learning across tasks.
• State-of-the-art in vision, speech, NLP, medicine, and games.
• Flexible architectures — same building blocks combine many ways.

• Data-hungry — needs large labelled datasets.
• Compute-hungry — GPUs and long training times.
• Black box — hard to interpret decisions.
• Overfitting risk on small data.
• Hyperparameter-sensitive — many knobs to tune.
• Energy cost — training large models consumes significant power.

• Layered architecture: Input → hidden layers → output.
• Weights & biases: Learnable parameters.
• Activation functions: Non-linearity (ReLU, sigmoid, tanh, softmax).
• Loss function: Guides learning (MSE, cross-entropy).
• Optimization: Gradient descent variants.
• Backpropagation: Efficient gradient computation.

     INPUT       HIDDEN 1     HIDDEN 2      OUTPUT
      x1 ─────▶  h1 ──┐  ┌── h1' ──┐
      x2 ─────▶  h2 ──┼──▶ h2' ──┼──▶  y1
      x3 ─────▶  h3 ──┼──▶ h3' ──┼──▶  y2
                     │          │
                    Bias       Bias

    Every line = a weight. Each hidden neuron:
      z = Σ (wᵢ · xᵢ) + b
      a = activation(z)

1. Explain with a diagram how a neuron computes its output.
2. List three tasks where neural networks outperform classical ML.
3. Write TensorFlow/Keras code for a 2-hidden-layer network on MNIST.
4. Why is a non-linear activation necessary?
5. Compare CNNs, RNNs and Transformers in one table.

# Pure Python perceptron
import numpy as np
def perceptron(x, w, b):
    z = np.dot(w, x) + b
    return 1 if z >= 0 else 0

# PyTorch single neuron
import torch, torch.nn as nn
neuron = nn.Linear(in_features=3, out_features=1)
x = torch.tensor([[1.0, 2.0, 3.0]])
y = torch.relu(neuron(x))
print(y)

A. Common activation functions:

B. Perceptron types: Single-layer perceptron (linear), Multi-Layer Perceptron (MLP, non-linear), and variants with different activations. The "linear perceptron" without activation can only classify linearly separable data.

• Simplicity: A perceptron is trivial to implement and understand.
• ReLU family: Fast to compute, sparse, works well in deep networks.
• Softmax: Gives calibrated probabilities for classification.
• Non-linearity enables universal approximation.
• Activations are differentiable for backpropagation.
• Choice affects learning dynamics — right activation speeds up convergence.

• Sigmoid / tanh: Saturate → vanishing gradients.
• ReLU: "Dying ReLU" — neurons stuck at zero.
• Softmax: Not ideal for multi-label (use sigmoid per class).
• Single perceptron cannot learn XOR — needs hidden layers.
• Choice is empirical — no universal rule for which activation to use.
• Over-smooth activations (GELU, Swish) can slow training on tiny datasets.

• Non-linearity: The key reason deep networks work.
• Differentiability: Needed for gradient-based training.
• Vanishing gradient: Sigmoid/tanh problem, mitigated by ReLU family.
• Zero-centred: Tanh better than sigmoid for optimization.
• Output layer: Softmax for multi-class, sigmoid for binary, linear for regression.
• Hidden layer default: ReLU or GELU.

    Sigmoid          Tanh              ReLU
    ───────          ────              ────
     1 ┤─╮            1 ┤  ╭─          ┤      /
       │  ╰─            │ /             │    /
     0 ┤───╮          0 ┤╯             │  /
       │    ╰─          │               │ /
    -1 ┤                 -1 ┤             └───────
        z                 z                z

    Dying ReLU problem:

      ReLU(z) = max(0, z)
      If z < 0 for all inputs → gradient = 0
      Neuron "dies" and never recovers.

    Fix: Leaky ReLU, ELU, GELU, Swish.

1. Implement a perceptron from scratch in Python (without libraries).
2. Plot sigmoid, tanh, ReLU and Leaky ReLU for z ∈ [−5, 5].
3. Show with a small example that XOR is not linearly separable.
4. Explain the dying ReLU problem and how Leaky ReLU fixes it.
5. Which activation would you use at the output of a 10-class classifier?

# Forward pass
z1 = X @ W1 + b1
a1 = relu(z1)
z2 = a1 @ W2 + b2
y_hat = softmax(z2)

# Loss
loss = -mean( y_onehot * log(y_hat) )

# Backward pass (chain rule — PyTorch does this for you)
loss.backward()

# Update
optimizer.step()
optimizer.zero_grad()

A. Network types:

B. Backpropagation variants: Full-batch, stochastic (SGD), mini-batch, and variants with momentum, Nesterov, RMSProp, Adam, AdamW.

C. Loss functions: MSE (regression), Binary cross-entropy (binary classification), Categorical cross-entropy (multi-class), Focal loss (class imbalance).

• Efficient: Backprop computes all gradients in roughly one forward + one backward pass.
• General: Works with any differentiable architecture.
• Automatic in modern frameworks — PyTorch/TensorFlow compute gradients via autograd.
• Scalable: GPUs make training large networks practical.
• Foundation for CNNs, RNNs, Transformers — same algorithm.

• Vanishing/exploding gradients in very deep networks.
• Requires differentiability — no discrete ops directly.
• Memory for stored activations during training.
• Local minima / saddle points can slow SGD.
• Hyperparameter sensitivity — LR, batch size, weight init matter.
• Numerical instability — need stable cross-entropy, ReLU, BN.

• Chain rule: Gradients flow backwards through layers.
• Weight initialization: Xavier, He — crucial for depth.
• Mini-batch training: Balance between SGD and full-batch.
• Epoch: One full pass over the training set.
• Learning rate: Too high → diverge; too low → slow.
• Autograd: Framework computes gradients via computational graph.

    Forward:  X → [W1] → z1 → ReLU → [W2] → z2 → softmax → ŷ
                                                              │
                                                              ▼
                                                            Loss
                                                              │
                                                              ▼
    Backward:            ◀──  ∂L/∂W2  ◀──  ∂L/∂a1  ◀──  ∂L/∂z1
                              (chain rule)
                                │
                                ▼
                         W2 ← W2 − η·∂L/∂W2
                         W1 ← W1 − η·∂L/∂W1

    Each training step:

      1. Forward pass  → predictions
      2. Compute loss
      3. Backward pass → gradients
      4. Optimizer step (update weights)
      5. zero_grad() → reset for next step

1. Write a 3-layer MLP in PyTorch for the Iris dataset.
2. Compute gradients manually for a 2-layer network on paper.
3. Explain why weight initialization matters for deep networks.
4. Compare SGD, mini-batch, and full-batch training.
5. Show how to log loss per epoch in Keras.

# PyTorch
import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# or: optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
# or: optim.AdamW(model.parameters(), lr=1e-3)

for epoch in range(10):
    optimizer.zero_grad()
    out  = model(x)
    loss = criterion(out, y)
    loss.backward()
    optimizer.step()

# Keras
model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

# Learning rate schedule
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)

A. Gradient descent variants:

B. Optimizers:

C. LR schedules: Step decay, exponential, cosine annealing, warmup + cosine, one-cycle, ReduceLROnPlateau.

• Simple and general: Works with any differentiable model.
• Mini-batch: Efficient use of GPUs.
• Adam family: Fast convergence, minimal tuning.
• Schedules: Improve final accuracy.
• Adaptive methods handle sparse features and different scales.
• Momentum escapes shallow local minima.

• Learning rate sensitive: Bad η ruins training.
• Saddle points can trap SGD without momentum.
• Adam generalization sometimes slightly worse than fine-tuned SGD.
• Memory for optimizer state (Adam 2× parameters).
• Hyperparameters: β1, β2, ε, weight decay require tuning.
• Gradient noise from small batches can destabilize training.

• Iterative: Many small steps, not one big jump.
• Stochastic: Gradient from mini-batch approximates true gradient.
• Non-convex: Local minima, saddle points, plateaus.
• Adaptive methods: Per-parameter learning rates.
• Weight decay: L2 regularization equivalent for SGD.
• Gradient clipping: Prevents exploding gradients in RNNs.

    Loss surface (2D slice):

              ▲  Loss
              │    ╭──╮     local min
              │   ╱    ╲
              │  ╱      ╲
              │ ╱        ╲     saddle
              │╱          ╲      ────────
              │            ╲___╱
              │              ╲     global min
              └────────────────────────────────▶ θ

    Learning rate effect:

      Too high:   ╱╲╱╲╱╲   bounces, diverges
      Too low:    ────────  slow crawl
      Just right: ───────╲  smooth descent

    Momentum:

      v ← β·v − η·∇L
      θ ← θ + v
      Accumulates direction → faster, smoother

1. Train the same network with SGD and Adam; compare loss curves.
2. Experiment with LR 1e-1, 1e-3, 1e-5 and record final accuracy.
3. Add a cosine LR schedule and show its effect.
4. Explain momentum with a physical analogy.
5. Show gradient clipping in PyTorch with clip_grad_norm_.

# Keras
from tensorflow.keras import layers
model = tf.keras.Sequential([
    layers.Dense(256),
    layers.BatchNormalization(),
    layers.Activation('relu'),
    layers.Dropout(0.3),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

# PyTorch
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1  = nn.Linear(784, 256)
        self.bn1  = nn.BatchNorm1d(256)
        self.drop = nn.Dropout(0.3)
        self.fc2  = nn.Linear(256, 10)

    def forward(self, x):
        x = self.drop(F.relu(self.bn1(self.fc1(x))))
        return self.fc2(x)

A. Regularization techniques:

B. Normalization layers: Batch Norm, Layer Norm, Group Norm, Instance Norm, RMS Norm.

C. Where to place:

• Convolutional nets: Conv → BN → ReLU → Dropout.
• Transformers: Linear → LayerNorm (pre-norm).
• RNNs: Layer Norm is common; Batch Norm less so.

• Reduces overfitting: Better test accuracy.
• Faster training: BN allows higher learning rates.
• Stable training: Smoother loss curves.
• Dropout: Effectively ensembles many subnetworks.
• Augmentation expands data cheaply.
• Early stopping prevents wasteful training.

• Dropout slows convergence and hurts small datasets less than expected.
• BN: Unstable with small batch sizes.
• BN has different behaviour in train vs eval (must switch mode).
• L2 λ must be tuned — too high underfits.
• Layer Norm needed when batch statistics unreliable.
• Augmentation must be label-preserving; bad augmentations hurt.

• Train vs eval mode: Dropout and BN behave differently.
• Regularization strength: Higher dropout, higher λ → more regularization.
• BN has parameters: Learnable γ, β after normalisation.
• Running statistics: BN stores mean/var for inference.
• Complementary: Combine BN + Dropout + Augmentation + Early stopping.
• Task-dependent: Vision → BN + augmentation; NLP → LayerNorm + weight decay.

    Dropout (p = 0.5):

      Train:                Eval:
      a1 ──✗ (dropped)      a1' = a1 * (1 − p)
      a2 ──✓                a2' = a2 * (1 − p)
      a3 ──✗                a3' = a3 * (1 − p)
      a4 ──✓
      (each node dropped independently)

    BatchNorm:

      For each mini-batch:
        μ  = mean(x)
        σ² = var(x)
        x̂  = (x − μ) / √(σ² + ε)
        y  = γ·x̂ + β     (γ, β learnable)

    Training:
      uses batch statistics

    Inference:
      uses running mean/var stored during training

1. Compare training with and without Dropout on CIFAR-10.
2. Add BatchNorm and observe how the learning rate can be increased.
3. Show the difference in behaviour of Dropout in train vs eval mode.
4. Explain why Layer Norm is preferred in Transformers over Batch Norm.
5. List three label-preserving augmentations for images.

import torch, torch.nn as nn, torch.nn.functional as F

class SmallCNN(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool  = nn.MaxPool2d(2)
        self.fc1   = nn.Linear(64 * 8 * 8, 128)
        self.fc2   = nn.Linear(128, n_classes)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))   # 32×32 → 16×16
        x = self.pool(F.relu(self.conv2(x)))   # 16×16 → 8×8
        x = x.flatten(1)
        x = F.relu(self.fc1(x))
        return self.fc2(x)

# model = SmallCNN()
# print(model)

A. Core layers:

B. Convolution variants: 1×1, 3×3, 5×5, 7×7; stride; dilated (atrous); transposed (deconvolution); depthwise separable.

C. Padding modes: 'valid' (no padding), 'same' (output = input size).

• Weight sharing reduces parameters dramatically.
• Translation invariance — same feature detected anywhere.
• Hierarchical features — edges → textures → parts → objects.
• Local receptive fields match image statistics.
• Well-understood and heavily optimised.
• Transfer learning — pretrained CNNs work on many tasks.

• Fixed input size for fully connected heads.
• Not rotation invariant by default (needs augmentation).
• Compute-heavy for high-resolution inputs.
• Not ideal for sequences of variable length (RNNs/Transformers better).
• Requires large data unless using pretrained models.
• Interpretation — visualising filters helps but is limited.

• Local connectivity: Each neuron sees only a small region.
• Parameter sharing: Same filter across all positions.
• Receptive field grows with depth.
• Pooling reduces resolution, increases field.
• Padding and stride control output dimensions.
• Channel dimension separates filters (features).

    Input (5×5)              Filter (3×3)         Output (3×3)
    1 0 1 0 0                1 0 1               5 4 4
    0 1 1 1 0      *         0 1 0        =      4 4 5
    1 0 1 0 1                1 0 1               4 5 5
    0 1 1 1 0
    1 0 1 0 0

    Filter slides over input; at each position:
      output[i,j] = Σ input[i+m, j+n] * filter[m,n] + bias

    Pooling (2×2 max):

    1 3 | 2 4
    5 6 | 7 8       →   6  8
    ----+----
    1 2 | 3 4
    5 7 | 8 9       →   7  9

    Typical CNN block:

      Input → Conv → BN → ReLU → Pool → ... → Flatten → FC → Softmax

1. Compute the output size of a conv layer for input 32×32, kernel 3×3, stride 1, padding 1.
2. Implement a small CNN in PyTorch and train on CIFAR-10.
3. Compare max-pooling and average-pooling on the same network.
4. Explain how 1×1 convolutions reduce channels.
5. Visualise the first-layer filters of a trained CNN.

import torchvision.models as models
import torch.nn as nn

# Load a pretrained ResNet-50
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)

# Replace the classifier head for 10 classes
model.fc = nn.Linear(model.fc.in_features, 10)

# Now train just the head (transfer learning)

• Reuse: Pretrained models work on new tasks with little data.
• Known good defaults: Saves weeks of architecture search.
• Efficiency: Modern architectures are faster and more accurate.
• Deployment: MobileNet/EfficientNet work on phones.
• Interpretable evolution: Each generation solves a specific problem.

• Large models need serious compute (VGG has 138M parameters).
• Diminishing returns: Deeper isn't always better.
• Pretrained bias: ImageNet features may not transfer to very different domains (medical, satellite).
• Licensing: Some pretrained weights restrict commercial use.
• Overfitting if the head is not properly regularized.

• Depth matters up to a point (degradation solved by residual).
• 3×3 convolutions are the sweet spot (VGG principle).
• 1×1 convolutions are channel mixers (Inception, ResNet bottlenecks).
• Skip connections enable gradient flow in deep nets.
• Global average pooling replaces huge FC layers.
• Compound scaling balances depth, width, resolution.

    Residual (skip) block:

           x
           │
      ┌────┴────┐
      │  Conv   │
      │  BN     │
      │  ReLU   │
      │  Conv   │
      │  BN     │
      └────┬────┘
           │
           ▼
        x + F(x)  → ReLU  → output
           ▲
           │  (identity shortcut)

    Depth vs error (conceptually):

      Plain net  ──  deeper after ~20 layers, error rises
      ResNet     ──  deeper helps → 50, 101, 152, ...

    Parameter comparison:

      VGG-16       138M parameters
      ResNet-50     25M parameters
      MobileNet      4M parameters

1. Draw ResNet's residual block and explain why it helps.
2. Compare VGG-16 and ResNet-50 in parameters and accuracy.
3. Load a pretrained EfficientNet and fine-tune on a small dataset.
4. Explain the role of 1×1 convolutions in Inception and ResNet bottlenecks.
5. Describe depthwise separable convolutions from MobileNet.

import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, vocab_size, embed=64, hidden=128, n_classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed)
        self.rnn   = nn.RNN(embed, hidden, batch_first=True)
        self.fc    = nn.Linear(hidden, n_classes)

    def forward(self, x):
        # x: (batch, seq_len) → embeddings
        e = self.embed(x)
        out, h_n = self.rnn(e)   # h_n = last hidden state
        return self.fc(h_n.squeeze(0))

A. Sequence tasks:

B. RNN variants: Vanilla RNN, Bidirectional RNN, Deep (stacked) RNN, LSTM, GRU, and sequence-to-sequence with attention.

C. Problems: Vanishing gradients (long-term dependencies lost), exploding gradients, sequential computation (slow), no direct long-range skip.

• Handles variable-length sequences.
• Parameter sharing across time reduces parameters.
• Temporal memory — hidden state carries context.
• Flexible architecture — many-to-one, many-to-many, etc.
• Foundation for LSTM, GRU, attention, transformers.
• Works well on small sequences and simple tasks.

• Vanishing gradients: Forgets long-term dependencies.
• Exploding gradients: Unstable training; needs clipping.
• Sequential: Cannot parallelise across time steps.
• Hard to train on very long sequences.
• Slower than Transformers on modern hardware.
• Less expressive than attention for long-range context.

• Hidden state: Compressed memory of past.
• Time-unrolled: Can be drawn as a deep chain.
• Backprop through time (BPTT): Gradient flows across steps.
• Truncated BPTT: Limit gradient to a window.
• Bidirectional: Process sequence forwards and backwards.
• Encoder-decoder: One RNN encodes, another decodes.

    Folded:                 Unrolled:

      ┌──┐             ┌──┐    ┌──┐    ┌──┐    ┌──┐
      │h │──▶ h        │h₀│──▶ │h₁│──▶ │h₂│──▶ │h₃│
      └──┘             └──┘    └──┘    └──┘    └──┘
       ▲                 ▲       ▲       ▲       ▲
       │                 │       │       │       │
       x                 x₀      x₁      x₂      x₃

    Same weights Wx, Wh, b at every step.

    Vanishing gradient:

      Gradient of loss wrt h₀ involves product of
      Wh at each step → repeated multiplication
      shrinks or explodes.

1. Implement a many-to-one RNN for sentiment classification.
2. Explain with a diagram how BPTT works.
3. Show empirically that vanilla RNNs forget long-term dependencies.
4. Compare RNN, LSTM and GRU in one table.
5. Explain why gradient clipping is used with RNNs.

import torch.nn as nn

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed=128, hidden=256, n_classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed)
        self.lstm  = nn.LSTM(embed, hidden, num_layers=2,
                             batch_first=True, bidirectional=True,
                             dropout=0.3)
        self.fc    = nn.Linear(hidden * 2, n_classes)

    def forward(self, x):
        e = self.embed(x)
        out, (h, c) = self.lstm(e)
        # take last timestep output (both directions concatenated)
        return self.fc(out[:, -1, :])

# Replace LSTM with nn.GRU for GRU variant.

A. LSTM gates:

B. GRU gates:

• Update gate (z): Combines forget + input from LSTM.
• Reset gate (r): Controls how much past to forget.
• No separate cell state: Simpler, faster.

C. Variants: BiLSTM, BiGRU, stacked LSTM/GRU, peephole LSTM, ConvLSTM (for video), attention-augmented LSTM.

• Long-term memory: Cell state preserves info across many steps.
• Gating: Learns what to remember and forget.
• Stable gradients: Additive cell state updates combat vanishing.
• GRU: Fewer parameters, faster training, similar performance.
• Proven: Strong baseline in speech, NLP, and time series.
• Handles variable length and sequential data naturally.

• Slow to train: Sequential dependency limits parallelisation.
• LSTM is complex: More gates and parameters than GRU.
• Transformers now dominate long-sequence tasks.
• Memory hungry for very long sequences.
• Not ideal for very long contexts — attention handles them better.
• Hyperparameter sensitivity — layers, hidden size, dropout matter.

• Cell state: Highway for gradient flow (LSTM).
• Gating mechanism: Multiplicative interactions.
• Sigmoid + tanh: Sigmoid for gates, tanh for candidate values.
• Bidirectional: Context from both past and future.
• GRU simplified: Two gates; merged cell and hidden.
• Foundation for Seq2Seq: Pre-Transformer translation used LSTM + attention.

    LSTM cell (simplified):

                C_{t-1} ──────────────────────► C_t
                   │  │                    ▲
                   │  │  ┌────┐            │
                   │  └─▶│ f  │───┐        │
                   │     └────┘   │        │
                   │     ┌────┐   │  ┌───┐ │
                   │  ┌─▶│ i  │───┼─▶│ × │ │
                   │  │  └────┘   │  └───┘ │
                   │  │  ┌────┐   │        │
                   │  └─▶│ c̃ │───┘        │
                   │     └────┘            │
                   │                        │
                   │     ┌────┐   ┌───┐    │
                   │  ┌─▶│ o  │──▶│ × │────┘
                   │  │  └────┘   └───┘
                   │  │             │
                   │  │             ▼
                   │  │            h_t
                   │  │
                   x_t, h_{t-1} ────┘

    GRU cell (simplified):

      z = σ(Wz·[h_{t-1}, x_t])       update
      r = σ(Wr·[h_{t-1}, x_t])       reset
      h̃ = tanh(W·[r * h_{t-1}, x_t])
      h_t = (1 − z)*h_{t-1} + z*h̃

1. Write an LSTM for text classification on the IMDB dataset.
2. Replace LSTM with GRU and compare training time and accuracy.
3. Explain how the cell state helps gradients flow during BPTT.
4. Use a BiLSTM for named-entity recognition.
5. Compare LSTM, GRU, and Transformer on a small translation task.

import torch, math
import torch.nn as nn

class SelfAttention(nn.Module):
    def __init__(self, dim, heads=8):
        super().__init__()
        self.heads = heads
        self.to_qkv = nn.Linear(dim, dim * 3)
        self.out    = nn.Linear(dim, dim)

    def forward(self, x):
        B, T, C = x.shape
        qkv = self.to_qkv(x).chunk(3, dim=-1)
        q, k, v = [t.view(B, T, self.heads, C // self.heads)
                    .transpose(1, 2) for t in qkv]
        # Scaled dot-product attention
        scores = q @ k.transpose(-2, -1) / math.sqrt(q.size(-1))
        attn   = scores.softmax(dim=-1)
        out    = (attn @ v).transpose(1, 2).reshape(B, T, C)
        return self.out(out)

A. Attention variants:

B. Transformer block:

• Multi-Head Self-Attention
• Add & LayerNorm (residual)
• Feedforward (2 linear + GELU/ReLU)
• Add & LayerNorm
• Optional dropout

C. Transformer families: Encoder-only (BERT, RoBERTa), Decoder-only (GPT, LLaMA), Encoder-decoder (T5, BART), Vision (ViT, Swin), Multimodal (CLIP, GPT-4V).

• Parallel training: All positions processed together.
• Long-range dependencies: Direct connection between any two positions.
• Scalable: Larger models keep improving (scaling laws).
• Flexible: Same block works for text, images, audio, video.
• Pretrained ecosystem: Hugging Face, BERT, GPT, ViT.
• Interpretable: Attention weights can be visualised.

• Quadratic cost: Attention is O(T²) in sequence length.
• Data-hungry: Needs massive datasets (or pretraining).
• Compute-hungry: Training large Transformers is expensive.
• Position encoding needed (no built-in order).
• Memory for attention matrices on long sequences.
• Hallucination: Large language models can generate false content.

• Attention: Weighted sum over values using query-key similarity.
• Multi-head: Each head learns a different relation.
• Positional encoding: Sinusoidal or learned.
• LayerNorm: Pre-norm or post-norm variants.
• Residual connections: Enable deep stacks.
• Scaling laws: Performance improves with size, data, compute.

    Self-attention:

      Input X (T × d)
         │
         ├─▶ Q = X·Wq
         ├─▶ K = X·Wk
         └─▶ V = X·Wv

      Scores = Q·Kᵀ / √d      (T × T)
      Attn   = softmax(Scores)
      Output = Attn · V       (T × d)

    Transformer encoder block:

      x ──▶ MultiHeadSelfAttn ──▶ Add & LayerNorm
              │
              └── (residual)
                                       │
                                       ▼
                            FeedForward (2 layers)
                                       │
                                       ▼
                                   Add & LayerNorm
                                       │
                                       ▼
                                       y

    Decoder adds masked self-attention + cross-attention.

1. Implement scaled dot-product attention from scratch.
2. Explain why the √d scaling matters.
3. Compare RNN, LSTM and Transformer on a sequence task.
4. Fine-tune a pretrained BERT for text classification.
5. Compute the memory cost of attention for a 4,096-token sequence.

import torch.nn as nn, torch.nn.functional as F

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU(),
            nn.Linear(128, 32)      # bottleneck
        )
        self.decoder = nn.Sequential(
            nn.Linear(32, 128),
            nn.ReLU(),
            nn.Linear(128, 784),
            nn.Sigmoid()
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z)

# loss = F.mse_loss(output, x)

A. Autoencoder variants:

B. Generative models:

• Unsupervised: No labels needed.
• Learned representations: Useful for clustering, retrieval, transfer.
• Anomaly detection: High reconstruction error = anomaly.
• Denoising and compression.
• GANs generate realistic images, audio, text.
• Data augmentation and creative applications.
• Foundation of modern generative AI.

• Autoencoders can copy input (learn identity).
• GAN training unstable: Mode collapse, oscillation.
• Evaluation of generators is hard (FID, IS are proxies).
• Ethical risks: Deepfakes, misinformation.
• Compute: Training large GANs/diffusion is expensive.
• VAEs produce blurrier samples than GANs/diffusion.

• Bottleneck forces compression in autoencoders.
• Latent space is the learned code.
• Adversarial training in GANs is a two-player game.
• Discriminator is discarded at inference (in GANs).
• Diffusion models now dominate image synthesis.
• Latent interpolation: Smooth transitions in latent space.

    Autoencoder:

      Input x ──▶ [Encoder] ──▶ z (bottleneck) ──▶ [Decoder] ──▶ x̂
                                                                    │
                                                                    ▼
                                                              Loss = |x − x̂|

    GAN:

      Random noise z ──▶ [Generator] ──▶ fake x̂
                                            │
                                            ▼
                                ┌──────────────────────┐
      Real x ─────────────────▶ │  Discriminator D(x)  │ ──▶ real/fake
                                └──────────────────────┘
                                            │
                                            ▼
      Generator loss: fool D
      Discriminator: distinguish real vs fake

1. Train a denoising autoencoder on MNIST.
2. Use an autoencoder for anomaly detection on credit card fraud.
3. Implement a small DCGAN for generating 32×32 images.
4. Explain mode collapse and one mitigation technique.
5. Compare GAN, VAE and diffusion for image generation.

import tensorflow as tf
from tensorflow.keras import layers, models

# Load data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(-1, 784).astype('float32') / 255.0
x_test  = x_test.reshape(-1, 784).astype('float32') / 255.0

# Build model
model = models.Sequential([
    layers.Dense(128, activation='relu', input_shape=(784,)),
    layers.Dropout(0.3),
    layers.Dense(64,  activation='relu'),
    layers.Dense(10,  activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

history = model.fit(x_train, y_train,
                    epochs=5, batch_size=128,
                    validation_split=0.1)

model.evaluate(x_test, y_test)

A. TensorFlow APIs:

B. Execution models: Eager (default, Pythonic) and Graph (tf.function, faster, deployable).

C. Distribution strategies: MirroredStrategy (single node multi-GPU), MultiWorkerMirroredStrategy (multi-node), TPUStrategy.

• Complete ecosystem from research to deployment.
• Keras is beginner-friendly and production-ready.
• TensorBoard for rich training visualisation.
• TF Serving / TFLite for robust deployment.
• Graph mode (tf.function) speeds up training.
• Distributed training built-in.
• Google support and large community.

• API churn: Multiple versions over the years confuse beginners.
• Steeper low-level API than PyTorch.
• Graph mode debugging harder than eager.
• Larger framework — heavier install than PyTorch.
• Documentation spans versions and can be inconsistent.
• Research community has drifted toward PyTorch.

• Tensors: Multi-dimensional arrays with GPU support.
• Autograd: tf.GradientTape records operations.
• Graphs: tf.function compiles Python code to a graph.
• Keras: Sequential, Functional, and Subclassing APIs.
• Callbacks: EarlyStopping, ModelCheckpoint, TensorBoard.
• Saved models: SavedModel format for deployment.

    ┌────────────────────────────────────────────┐
    │             User Code (Python)             │
    ├────────────────────────────────────────────┤
    │       tf.keras  (Sequential/Functional)    │
    ├────────────────────────────────────────────┤
    │       tf.Module / tf.function              │
    ├────────────────────────────────────────────┤
    │       tf.GradientTape  (autograd)          │
    ├────────────────────────────────────────────┤
    │       TensorFlow Runtime (eager / graph)   │
    ├────────────────────────────────────────────┤
    │  CPU     GPU     TPU     Distributed       │
    └────────────────────────────────────────────┘
             │            │             │
             ▼            ▼             ▼
        TFLite       TF.js         TF Serving
        (mobile)     (browser)     (prod)

1. Train a CNN on CIFAR-10 with Keras.
2. Add EarlyStopping and ModelCheckpoint callbacks.
3. Write a custom training loop with tf.GradientTape.
4. Convert the trained model to TFLite and run it.
5. Log metrics with TensorBoard and view the dashboard.

import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Data
X = torch.randn(1000, 784)
y = torch.randint(0, 10, (1000,))
loader = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=True)

# Model
model = nn.Sequential(
    nn.Linear(784, 128), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(128, 64),  nn.ReLU(),
    nn.Linear(64, 10)
)

# Training
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(5):
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        out  = model(xb)
        loss = criterion(out, yb)
        loss.backward()
        optimizer.step()
    print(f"epoch {epoch} done")

A. Core components:

B. Model APIs: Sequential, Module (custom class), Functional.

C. Deployment: TorchScript, ONNX export, TorchServe, quantization.

• Pythonic: Debug with print, pdb, breakpoints.
• Dynamic graphs: Change control flow on the fly.
• Simple autograd: Just call loss.backward().
• Huge research adoption: Most papers release PyTorch code.
• Rich ecosystem: torchvision, Hugging Face, Lightning.
• Performance: torch.compile closes gap with static frameworks.
• Deployment path: TorchScript, ONNX, TorchServe.

• Deployment historically behind TensorFlow (improving).
• No single production platform comparable to TF Serving / TFX.
• Mobile / edge is less mature than TFLite.
• You write the training loop — more code than Keras.fit().
• Memory management for very large models is manual.
• Ecosystem fragmentation — many third-party tools.

• Eager by default: Code runs line-by-line.
• Autograd: Records operations, backward pass via chain rule.
• zero_grad, backward, step: The canonical training loop.
• Devices: tensor.to('cuda'), model.to(device).
• nn.Module: Base class for all layers and models.
• state_dict: Standard way to save/load weights.

    Canonical training loop:

      for epoch in epochs:
        for xb, yb in loader:
          optimizer.zero_grad()   # reset grads
          pred = model(xb)         # forward
          loss = criterion(pred, yb)
          loss.backward()          # backward
          optimizer.step()         # update

    Ecosystem:

      torch.nn        → layers, losses
      torch.optim     → SGD, Adam, AdamW
      torch.utils.data→ Dataset, DataLoader
      torchvision     → ResNet, transforms
      Hugging Face    → transformers, datasets
      Lightning       → high-level Trainer

1. Train a CNN on CIFAR-10 with a custom PyTorch training loop.
2. Save and load the model's state_dict.
3. Add a learning rate scheduler and observe loss.
4. Fine-tune a pretrained ResNet-18 on a small dataset.
5. Convert the model to TorchScript and run inference.

import torch, torch.nn as nn, torch.optim as optim
import torchvision.models as models

# Load pretrained ResNet-18
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# Replace the classifier for 5 classes
model.fc = nn.Linear(model.fc.in_features, 5)

# Freeze backbone (feature extraction)
for p in model.parameters():
    p.requires_grad = False
for p in model.fc.parameters():
    p.requires_grad = True

# Train only the head first
optimizer = optim.Adam(model.fc.parameters(), lr=1e-3)

# Later: unfreeze more layers and fine-tune with smaller LR
# for p in model.parameters():
#     p.requires_grad = True
# optimizer = optim.Adam(model.parameters(), lr=1e-5)

A. Strategies:

B. Popular pretrained models:

• Vision: ResNet, EfficientNet, ConvNeXt, ViT, CLIP, SAM.
• NLP: BERT, RoBERTa, GPT, T5, LLaMA, Mistral.
• Speech: wav2vec 2.0, Whisper.
• Multimodal: CLIP, BLIP, Flamingo, GPT-4V.

C. Sources: torchvision, Hugging Face Hub, TensorFlow Hub, TIMM.

• Less data: Works with hundreds instead of millions of examples.
• Faster training: Days instead of weeks.
• Better accuracy on small datasets.
• Accessible: Small teams can build strong models.
• Reusable: One backbone serves many tasks.
• Foundational for LLMs: Pretraining + fine-tuning is the standard paradigm.

• Domain mismatch: ImageNet features may not fit medical or satellite images.
• Catastrophic forgetting: Fine-tuning can destroy useful features.
• Licensing: Not all pretrained models allow commercial use.
• Compute for fine-tuning: Large models still need serious GPUs.
• Bias: Pretrained models carry the biases of their training data.
• Overfitting head: Small head can overfit if not regularized.

• Freeze then unfreeze: Start with frozen backbone, then fine-tune.
• Smaller LR for fine-tuning than for training from scratch.
• Discriminative LR: Lower LR for early layers, higher for later layers.
• BatchNorm: Keep in eval mode if data is small.
• Data augmentation helps even with pretrained models.
• LoRA / adapters: Efficient fine-tuning for large models.

    Pretrained backbone (e.g., ResNet-50 on ImageNet):

      [ Conv1 ] → [ Conv2 ] → ... → [ Conv5 ] → [ FC-1000 ]

    Feature extraction:

      [ Conv1 ... Conv5 ] (frozen) → [ New FC-n ]  ← train only this

    Partial fine-tuning:

      [ Conv1 ] (frozen) → [ Conv2 ... Conv5 ] → [ New FC-n ]
                            ↑ train with small LR

    Full fine-tuning:

      [ Conv1 ... Conv5 ] → [ New FC-n ]  ← all train with small LR

    LoRA / adapters (for large models):

      [ Backbone frozen ] + [ small adapters trained ]

1. Fine-tune ResNet-18 on a small custom image dataset.
2. Compare feature extraction vs full fine-tuning.
3. Fine-tune a pretrained BERT on a text classification task.
4. Apply LoRA to fine-tune a small language model.
5. Visualise first-layer filters of a pretrained model.

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

# Splits
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.15, stratify=y)
X_tr, X_val, y_tr, y_val = train_test_split(X_tr, y_tr, test_size=0.15, stratify=y_tr)

# Train on X_tr with early stopping on X_val
# ...

# Final evaluation on X_te (ONLY once)
preds = model.predict(X_te)
print(classification_report(y_te, preds))
print(confusion_matrix(y_te, preds))

A. Data splits:

B. Metrics:

C. Tuning methods: Grid search, random search, Bayesian optimization (Optuna, Hyperopt, Ax), population-based training, ASHA.

• Trustworthy estimates: Test set gives unbiased performance.
• Better models: Tuning finds configurations that generalize.
• Reproducibility: Fixed splits and seeds.
• Diagnosis: Confusion matrices reveal where models fail.
• Comparison: Fair benchmark against baselines.
• Resource efficiency: Bayesian search finds good configs faster than grid.

• Tuning is expensive: Each trial trains a model.
• Overfitting validation: Too many trials can leak into val set.
• Data leakage risk: Wrong splits ruin results.
• Metric misuse: Accuracy on imbalanced data is misleading.
• Reproducibility challenges: Nondeterminism across GPUs and frameworks.
• Time and compute cost for large models.

• Discipline: Never tune on the test set.
• Stratification: Preserve class balance.
• Early stopping: Stop when val loss increases.
• Seed control: Fix random seeds for reproducibility.
• Cross-validation: Small data → k-fold averages.
• Experiment tracking: MLflow, Weights & Biases, TensorBoard.

    Data splits:

      Total Data
      ├── Train (70%)           ── learn weights
      ├── Validation (15%)      ── tune hyperparams, early stop
      └── Test (15%)            ── final unbiased estimate

    Workflow:

      1. Split data
      2. Train on train, monitor on val
      3. Tune hyperparameters using val
      4. Retrain with best config (optional)
      5. Evaluate once on test
      6. Report + compare baseline

    Confusion matrix (binary):

                     Pred +    Pred -
       Actual +       TP        FN
       Actual -       FP        TN

    Precision = TP/(TP+FP)   Recall = TP/(TP+FN)
    F1 = 2·P·R/(P+R)         Accuracy = (TP+TN)/All

1. Split a dataset into train/val/test and compute all three accuracies.
2. Plot a confusion matrix for a 10-class classifier.
3. Run a small random search over learning rates and batch sizes.
4. Compare accuracy and macro F1 on an imbalanced dataset.
5. Track experiments with Weights & Biases or MLflow.

# FastAPI microservice
from fastapi import FastAPI
import torch
from torchvision import transforms

app   = FastAPI()
model = torch.jit.load("model.pt").eval()

@app.post("/predict")
def predict(payload: dict):
    x = preprocess(payload["image"])
    with torch.no_grad():
        y = model(x).softmax(-1)
    return {"class": int(y.argmax()), "confidence": float(y.max())}

# Run: uvicorn app:app --host 0.0.0.0 --port 8000

# Or TorchServe / TF Serving for scalable production serving.

# Optimisation: quantize to int8
quantized = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

A. Deployment patterns:

B. Optimisation techniques:

• Quantization: FP32 → INT8, 4× smaller, faster.
• Pruning: Remove small weights.
• Knowledge distillation: Small model learns from large model.
• ONNX: Portable format across runtimes.
• Batching and caching: Higher throughput.
• Compilation: torch.compile, TensorRT, XLA.

C. Ethical AI pillars: Fairness, Explainability, Privacy, Safety, Accountability, Transparency.

• Business value: Models create impact only when deployed.
• Scalability: Serve millions of requests.
• Optimisation: Quantization makes models cheap to run.
• MLOps: Reliable, monitored systems.
• Responsible AI: Fairness and privacy reduce harm.
• Edge deployment: Low latency, privacy by design.

• Complexity: Many moving parts (serving, monitoring, retraining).
• Latency vs accuracy: Smaller models are faster but often weaker.
• Data drift: Performance decays silently.
• Cost: GPUs are expensive; scaling multiplies costs.
• Security: Model extraction, adversarial inputs, prompt injection.
• Ethical risks: Bias, deepfakes, surveillance, job displacement.

• MLOps lifecycle: Data → train → evaluate → deploy → monitor → retrain.
• Monitoring: Latency, throughput, accuracy, drift, data quality.
• A/B testing: Compare models live.
• Shadow deployments: Run new model silently.
• Rollback: Ability to revert quickly.
• Explainability: SHAP, LIME, attention maps, feature importance.

    MLOps lifecycle:

      Data → Train → Evaluate → Deploy → Monitor → Retrain
         ▲                                            │
         └─────────── drift detected ─────────────────┘

    Serving options:

      Client ─▶ API Gateway ─▶ Inference Service ─▶ Model
               (REST/gRPC)        (TorchServe,
                                   TF Serving,
                                   Triton, vLLM)

    Ethical pillars:

      ┌────────────────────────────────┐
      │ Fairness  · Explainability     │
      │ Privacy   · Safety             │
      │ Accountability · Transparency  │
      └────────────────────────────────┘

    Real-world uses:

      Vision    : medical imaging, autonomous vehicles, retail
      Language  : chat, translation, code, search
      Speech    : assistants, transcription, voice cloning
      Science   : drug discovery, climate, protein folding

1. Serve a small PyTorch model with FastAPI and call it with curl.
2. Quantize a model to INT8 and measure latency and accuracy change.
3. Export a model to ONNX and run it with onnxruntime.
4. Write a short model card for a fictional classifier.
5. List three sources of bias in a face-recognition system.