Deep Learning Highlights Summary

Every highlight bullet from the 12 pages in this section, gathered on one page and grouped by the page it came from.

The Deep Learning Mental Model

• PyTorch rebuilds the computation graph on every forward pass, then walks it backward for gradients
• Zeroing gradients each batch matters because PyTorch accumulates them by default
• GPUs help because forward/backward passes are almost entirely parallelizable matrix math
• Mixed precision trades some numerical precision for speed and memory, hence the gradient scaler
• Freezing layers during fine-tuning removes them from the backward pass entirely, not just the forward pass

PyTorch Basics

• Tensors are n-dimensional arrays with GPU support
• device='cuda' moves computation to the GPU
• torch.nn provides layers and loss functions
• PyTorch 2.6+ includes torch.compile for speedups

Autograd

• Autograd builds a DAG of operations for reverse-mode differentiation
• requires_grad=True marks tensors that need gradients
• torch.no_grad() disables tracking for inference
• detach() breaks a tensor from the computation graph

Building Models with nn.Module

• Subclass nn.Module and implement forward()
• nn.Sequential chains layers for simple architectures
• named_parameters() exposes all learnable weights
• model.to(device) moves all parameters to GPU

Datasets & DataLoaders

• Dataset defines len and getitem
• DataLoader batches, shuffles, and parallelizes loading
• Transforms run per-sample before batching
• num_workers>0 speeds I/O on multi-core machines

Training Loops

• train/eval modes control dropout and batch norm behavior
• zero_grad before backward prevents gradient accumulation bugs
• Learning rate schedulers adjust LR across epochs
• Save checkpoints with model, optimizer, and epoch state

GPUs & Mixed Precision

• CUDA moves tensors and models to NVIDIA GPUs
• autocast runs ops in float16/bfloat16 for speed
• GradScaler prevents underflow in fp16 training
• torch.compile stacks with mixed precision in PyTorch 2.6+

Transfer Learning & Fine-Tuning

• Pretrained weights provide strong starting points
• Freeze backbone layers when data is limited
• Replace the classifier head for new class counts
• Use lower learning rates for pretrained layers

PyTorch Lightning

• LightningModule encapsulates model, training, and validation
• Trainer handles device placement, DDP, and checkpointing
• Callbacks add early stopping, LR monitoring, and logging
• Keeps research code in LightningModule, boilerplate in Trainer

Saving, Loading & Export

• state_dict saves learnable parameters only
• weights_only=True prevents unsafe pickle loading
• TorchScript traces models for C++ deployment
• ONNX exports to cross-framework inference runtimes

Distributed Training

• DDP is the standard multi-GPU training approach
• Each GPU processes a data shard and syncs gradients
• DistributedSampler ensures non-overlapping data per rank
• Launch with torchrun for process spawning

Deep Learning Best Practices

• Set seeds and document library versions
• Use mixed precision on modern GPUs
• Save state_dict, not full model pickles
• Monitor validation metrics, not just training loss