Deep Learning Best Practices

Q: What is the most common training bug?

Forgetting model.eval() during validation. Dropout randomizes predictions; batch norm uses batch statistics. Val metrics become meaningless.

Q: How do I detect a data loading bottleneck?

GPU utilization below 50% during training (check nvidia-smi). Increase num_workers and pin_memory. Pre-cache preprocessed data to disk.

Rules for reproducible, efficient PyTorch training and deployment. Apply during code review and before scaling to multi-GPU.

How to Use This List

Rules A-B cover setup and data pipelines.
Rules C-D govern training and evaluation.
Rules E-F cover export and production.
Revisit when changing hardware, framework version, or model architecture.

A - Reproducibility

Set random seeds. torch.manual_seed(42), np.random.seed(42), and DataLoader generator for reproducible runs.
Pin PyTorch and CUDA versions. Record torch.__version__ and CUDA driver in experiment logs.
Save hyperparameters with checkpoints. Include LR, batch size, architecture config, and dataset version.
Use deterministic flags when debugging. torch.use_deterministic_algorithms(True) for reproducibility (may reduce performance).

B - Data Pipeline

Separate train and val transforms. Random augmentation on train only; deterministic transforms on val/test.
Use pin_memory and non_blocking with CUDA. Overlap data transfer with GPU computation.
Set num_workers appropriately. 4-8 workers on multi-core machines; 0 in notebooks.
Validate a single batch before training. Print shapes, dtypes, and value ranges to catch pipeline bugs early.

C - Training

Call model.train() and model.eval() correctly. Dropout and batch norm depend on mode.
Use zero_grad(set_to_none=True) before backward. Prevents gradient accumulation bugs.
Clip gradients for RNNs and deep networks. clip_grad_norm_(model.parameters(), 1.0).
Use mixed precision on CUDA. torch.amp.autocast + GradScaler for 2x speedup.
Save checkpoints on validation improvement. Do not keep only the last epoch.
Monitor validation metrics every epoch. Training loss alone hides overfitting.

D - Model Design

Start from pretrained weights when data is limited. ImageNet for vision, HF hub for NLP.
Use differential learning rates for fine-tuning. Lower LR for pretrained layers, higher for new head.
Match loss function to task. CrossEntropyLoss for multi-class; BCEWithLogitsLoss for multi-label.
Profile before optimizing. Identify whether data loading or GPU compute is the bottleneck.

E - Checkpointing & Export

Save state_dict, not full model objects. Reconstruct model class on load.
Use weights_only=True when loading checkpoints. PyTorch 2.6+ security best practice.
Export in eval mode. model.eval() before TorchScript trace or ONNX export.
Verify exported model output matches PyTorch. torch.allclose against ONNX Runtime output.

F - Production

Version model artifacts in a registry. Tie each artifact to training data hash and metrics.
Include input schema in serving contract. Expected tensor shape, dtype, and normalization.
Test inference latency and memory. Profile batch sizes expected in production.
Plan for GPU cost. Use spot instances, mixed precision, and right-sized models.

FAQs

What is the most common training bug?

Forgetting model.eval() during validation.
Dropout randomizes predictions; batch norm uses batch statistics.
Val metrics become meaningless.

How do I make training reproducible?

torch.manual_seed(42)
torch.cuda.manual_seed_all(42)
generator = torch.Generator().manual_seed(42)
loader = DataLoader(ds, generator=generator, shuffle=True)

Full reproducibility on GPU may require deterministic algorithms flag.

When should I use PyTorch Lightning?

Multi-GPU, logging, and checkpointing boilerplate add up.
Lightning is worth it for team projects and production training.
Learn manual loops first for debugging skills.

How do I avoid CUDA OOM?

Reduce batch size, enable mixed precision, use gradient accumulation.
torch.cuda.empty_cache() between experiments.

Should I use torch.compile?

Yes for repeated training/inference on PyTorch 2.6+.
First iteration is slow (compilation); benchmark before committing.

How do I review a training PR?

Check train/eval mode switching, device placement, and checkpoint saving.
Verify validation metrics are logged.
Run fast_dev_run equivalent (few batches) to validate the loop.

What belongs in a training config?

Batch size, LR, epochs, model architecture, dataset path, seed, and optimizer.
Use YAML/JSON or Hydra for reproducible experiment configs.

How do I detect a data loading bottleneck?

GPU utilization below 50% during training (check nvidia-smi).
Increase num_workers and pin_memory.
Pre-cache preprocessed data to disk.

When is distributed training worth it?

Single epoch takes hours on one GPU.
Near-linear speedup with DDP on 2-8 GPUs.
Not worth the complexity for fast experiments.

How do I document a trained model?

Architecture, training data, metrics, limitations, and input/output spec.
Include PyTorch version and hardware used for training.

What tests should inference code have?

def test_model_output_shape():
    out = model(torch.randn(1, 3, 224, 224))
    assert out.shape == (1, num_classes)

Shape, dtype, and a golden-input golden-output test.

How do I choose batch size?

Largest that fits in GPU memory without OOM.
Scale LR with batch size (linear scaling rule) when increasing.
Use gradient accumulation for effective larger batches.

Training Loops - loop implementation
GPUs & Mixed Precision - AMP setup
Saving, Loading & Export - artifact format
Experiment Tracking - logging runs
Distributed Training - scaling out

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Deep Learning Best Practices

Rules for reproducible, efficient PyTorch training and deployment. Apply during code review and before scaling to multi-GPU.

How to Use This List

Rules A-B cover setup and data pipelines.
Rules C-D govern training and evaluation.
Rules E-F cover export and production.
Revisit when changing hardware, framework version, or model architecture.

A - Reproducibility

Set random seeds. torch.manual_seed(42), np.random.seed(42), and DataLoader generator for reproducible runs.
Pin PyTorch and CUDA versions. Record torch.__version__ and CUDA driver in experiment logs.
Save hyperparameters with checkpoints. Include LR, batch size, architecture config, and dataset version.
Use deterministic flags when debugging. torch.use_deterministic_algorithms(True) for reproducibility (may reduce performance).

B - Data Pipeline

Separate train and val transforms. Random augmentation on train only; deterministic transforms on val/test.
Use pin_memory and non_blocking with CUDA. Overlap data transfer with GPU computation.
Set num_workers appropriately. 4-8 workers on multi-core machines; 0 in notebooks.
Validate a single batch before training. Print shapes, dtypes, and value ranges to catch pipeline bugs early.

C - Training

Call model.train() and model.eval() correctly. Dropout and batch norm depend on mode.
Use zero_grad(set_to_none=True) before backward. Prevents gradient accumulation bugs.
Clip gradients for RNNs and deep networks. clip_grad_norm_(model.parameters(), 1.0).
Use mixed precision on CUDA. torch.amp.autocast + GradScaler for 2x speedup.
Save checkpoints on validation improvement. Do not keep only the last epoch.
Monitor validation metrics every epoch. Training loss alone hides overfitting.

D - Model Design

Start from pretrained weights when data is limited. ImageNet for vision, HF hub for NLP.
Use differential learning rates for fine-tuning. Lower LR for pretrained layers, higher for new head.
Match loss function to task. CrossEntropyLoss for multi-class; BCEWithLogitsLoss for multi-label.
Profile before optimizing. Identify whether data loading or GPU compute is the bottleneck.

E - Checkpointing & Export

Save state_dict, not full model objects. Reconstruct model class on load.
Use weights_only=True when loading checkpoints. PyTorch 2.6+ security best practice.
Export in eval mode. model.eval() before TorchScript trace or ONNX export.
Verify exported model output matches PyTorch. torch.allclose against ONNX Runtime output.

F - Production

Version model artifacts in a registry. Tie each artifact to training data hash and metrics.
Include input schema in serving contract. Expected tensor shape, dtype, and normalization.
Test inference latency and memory. Profile batch sizes expected in production.
Plan for GPU cost. Use spot instances, mixed precision, and right-sized models.

FAQs

What is the most common training bug?

Forgetting model.eval() during validation.
Dropout randomizes predictions; batch norm uses batch statistics.
Val metrics become meaningless.

How do I make training reproducible?

torch.manual_seed(42)
torch.cuda.manual_seed_all(42)
generator = torch.Generator().manual_seed(42)
loader = DataLoader(ds, generator=generator, shuffle=True)

Full reproducibility on GPU may require deterministic algorithms flag.

When should I use PyTorch Lightning?

Multi-GPU, logging, and checkpointing boilerplate add up.
Lightning is worth it for team projects and production training.
Learn manual loops first for debugging skills.

How do I avoid CUDA OOM?

Reduce batch size, enable mixed precision, use gradient accumulation.
torch.cuda.empty_cache() between experiments.

Should I use torch.compile?

Yes for repeated training/inference on PyTorch 2.6+.
First iteration is slow (compilation); benchmark before committing.

How do I review a training PR?

Check train/eval mode switching, device placement, and checkpoint saving.
Verify validation metrics are logged.
Run fast_dev_run equivalent (few batches) to validate the loop.

What belongs in a training config?

Batch size, LR, epochs, model architecture, dataset path, seed, and optimizer.
Use YAML/JSON or Hydra for reproducible experiment configs.

How do I detect a data loading bottleneck?

GPU utilization below 50% during training (check nvidia-smi).
Increase num_workers and pin_memory.
Pre-cache preprocessed data to disk.

When is distributed training worth it?

Single epoch takes hours on one GPU.
Near-linear speedup with DDP on 2-8 GPUs.
Not worth the complexity for fast experiments.

How do I document a trained model?

Architecture, training data, metrics, limitations, and input/output spec.
Include PyTorch version and hardware used for training.

What tests should inference code have?

def test_model_output_shape():
    out = model(torch.randn(1, 3, 224, 224))
    assert out.shape == (1, num_classes)

Shape, dtype, and a golden-input golden-output test.

How do I choose batch size?

Largest that fits in GPU memory without OOM.
Scale LR with batch size (linear scaling rule) when increasing.
Use gradient accumulation for effective larger batches.

Training Loops - loop implementation
GPUs & Mixed Precision - AMP setup
Saving, Loading & Export - artifact format
Experiment Tracking - logging runs
Distributed Training - scaling out

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Deep Learning Best Practices

Search Documentation

Deep Learning Best Practices