Hugging Face transformers
The transformers library loads and runs open-source models from the Hugging Face Hub. Use pipeline for quick inference or AutoModel for custom generation loops.
Recipe
Quick-reference recipe card - copy-paste ready.
from transformers import pipeline
generator = pipeline("text-generation", model="meta-llama/Llama-3.2-1B-Instruct")
output = generator("Explain Python decorators:", max_new_tokens=100)
print(output[0]["generated_text"])When to reach for this:
- Running open-source LLMs locally or on your infrastructure.
- Fine-tuning models with the Trainer API.
- Accessing thousands of pretrained models from the Hub.
- Building custom generation with control over sampling parameters.
Working Example
"""hugging_face_transformers.py - pipeline and manual generation."""
from __future__ import annotations
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_id = "meta-llama/Llama-3.2-1B-Instruct"
# Quick inference with pipeline
gen = pipeline("text-generation", model=model_id, torch_dtype=torch.float16, device_map="auto")
result = gen("What is pytest?", max_new_tokens=80, do_sample=True, temperature=0.7)
print("pipeline:", result[0]["generated_text"][-200:])
# Manual generation loop
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
messages = [{"role": "user", "content": "Write a one-line Python hello world."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(**inputs, max_new_tokens=60, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("manual:", response)What this demonstrates:
pipelinefor zero-config text generation.AutoTokenizerandAutoModelForCausalLMfor manual control.apply_chat_templatefor instruction-tuned models.device_map="auto"for automatic GPU placement.
Deep Dive
How It Works
- Hub hosts model weights, configs, and tokenizers.
- Auto classes resolve architecture from
config.json. - Tokenizer converts text to token IDs and back.
- generate() runs autoregressive decoding with sampling or greedy search.
- Trainer fine-tunes models on custom datasets.
Key APIs
| API | Use When | Complexity |
|---|---|---|
pipeline() | Quick prototyping | Low |
model.generate() | Custom generation params | Medium |
Trainer | Fine-tuning | Medium |
trl (SFT, DPO) | RLHF-style alignment | High |
Python Notes
# 4-bit quantization for limited GPU memory
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")Gotchas
- Not using chat template - instruction models produce poor output without proper formatting. Fix:
tokenizer.apply_chat_template. - OOM on large models - 70B models need multi-GPU or quantization. Fix: 4-bit loading,
device_map="auto", or smaller models. - Gated model access - Llama and some models require Hub login and license acceptance. Fix:
huggingface-cli loginand accept license on model page. - Not setting pad_token - generation fails without pad token. Fix:
tokenizer.pad_token = tokenizer.eos_token. - Downloading every run - re-downloads waste time. Fix: set
HF_HOMEcache directory; uselocal_files_only=Truein production. - float32 on GPU - doubles memory vs float16. Fix:
torch_dtype=torch.float16orbfloat16.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
pipeline | Quick inference | Custom decoding logic |
| vLLM | High-throughput serving | Simple local experiments |
| Ollama | Easy local model management | Fine-tuning |
| API providers | No GPU infrastructure | Data privacy requires local |
FAQs
How do I authenticate with the Hub?
pip install huggingface_hub
huggingface-cli login- Required for gated models.
What is device_map auto?
- Automatically spreads model layers across available GPUs.
- Essential for models larger than single GPU memory.
How do I fine-tune a model?
from transformers import TrainingArguments, Trainer
args = TrainingArguments(output_dir="./out", num_train_epochs=3, per_device_train_batch_size=4)
trainer = Trainer(model=model, args=args, train_dataset=dataset)
trainer.train()- See Transfer Learning.
What models run on a laptop?
- 1B-3B parameter models with 4-bit quantization.
- Llama 3.2 1B, Phi-3 mini, SmolLM.
How do I control generation length?
max_new_tokenslimits new tokens (preferred overmax_length).min_new_tokensensures minimum response length.
What is temperature in generate()?
- Same as API temperature: higher = more random.
do_sample=Falsefor greedy (deterministic) decoding.
How do I batch inference?
- Pad inputs to same length; pass batch tensor to
generate. - vLLM handles batching efficiently for serving.
Can I use transformers with PyTorch 2.6+?
- Yes - transformers supports recent PyTorch versions.
torch.compilecan speed up generation on supported models.
What is PEFT/LoRA?
- Low-Rank Adaptation: fine-tune small adapter weights instead of full model.
- Dramatically reduces memory for fine-tuning.
How do I cache models locally?
export HF_HOME=/data/hf_cache- Models download once and reuse from cache.
How do I run embeddings?
from transformers import AutoModel
embedder = pipeline("feature-extraction", model="sentence-transformers/all-MiniLM-L6-v2")- Or use
sentence-transformerslibrary directly.
What is the Hub model card?
- Documents model capabilities, limitations, and intended use.
- Read before deploying any open model.
Related
- Local & Open Models - Ollama and vLLM
- Embeddings & Similarity - embedding models
- Transfer Learning & Fine-Tuning - fine-tune patterns
- Streaming & Async LLM Calls - streaming generation
- Tokens, Cost & Rate Limits - local vs API cost
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.