Low-Rank Adaptation: fine-tune small adapter weights instead of full model. Dramatically reduces memory for fine-tuning.

Hugging Face transformers

Q: How do I control generation length?

max_new_tokens limits new tokens (preferred over max_length). min_new_tokens ensures minimum response length.

Q: What is temperature in generate()?

Same as API temperature: higher = more random. do_sample=False for greedy (deterministic) decoding.

Q: How do I batch inference?

Pad inputs to same length; pass batch tensor to generate. vLLM handles batching efficiently for serving.

Q: Can I use transformers with PyTorch 2.6+?

Yes - transformers supports recent PyTorch versions. torch.compile can speed up generation on supported models.

Q: How do I run embeddings?

Or use sentence-transformers library directly.

The transformers library loads and runs open-source models from the Hugging Face Hub. Use pipeline for quick inference or AutoModel for custom generation loops.

Recipe

Quick-reference recipe card - copy-paste ready.

from transformers import pipeline
 
generator = pipeline("text-generation", model="meta-llama/Llama-3.2-1B-Instruct")
output = generator("Explain Python decorators:", max_new_tokens=100)
print(output[0]["generated_text"])

When to reach for this:

Running open-source LLMs locally or on your infrastructure.
Fine-tuning models with the Trainer API.
Accessing thousands of pretrained models from the Hub.
Building custom generation with control over sampling parameters.

Working Example

"""hugging_face_transformers.py - pipeline and manual generation."""
from __future__ import annotations
 
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
model_id = "meta-llama/Llama-3.2-1B-Instruct"
 
# Quick inference with pipeline
gen = pipeline("text-generation", model=model_id, torch_dtype=torch.float16, device_map="auto")
result = gen("What is pytest?", max_new_tokens=80, do_sample=True, temperature=0.7)
print("pipeline:", result[0]["generated_text"][-200:])
 
# Manual generation loop
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
 
messages = [{"role": "user", "content": "Write a one-line Python hello world."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
 
with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=60, temperature=0.7, do_sample=True)
 
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("manual:", response)

What this demonstrates:

pipeline for zero-config text generation.
AutoTokenizer and AutoModelForCausalLM for manual control.
apply_chat_template for instruction-tuned models.
device_map="auto" for automatic GPU placement.

Deep Dive

How It Works

Hub hosts model weights, configs, and tokenizers.
Auto classes resolve architecture from config.json.
Tokenizer converts text to token IDs and back.
generate() runs autoregressive decoding with sampling or greedy search.
Trainer fine-tunes models on custom datasets.

Key APIs

API	Use When	Complexity
`pipeline()`	Quick prototyping	Low
`model.generate()`	Custom generation params	Medium
`Trainer`	Fine-tuning	Medium
`trl` (SFT, DPO)	RLHF-style alignment	High

Python Notes

# 4-bit quantization for limited GPU memory
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")

Gotchas

Not using chat template - instruction models produce poor output without proper formatting. Fix: tokenizer.apply_chat_template.
OOM on large models - 70B models need multi-GPU or quantization. Fix: 4-bit loading, device_map="auto", or smaller models.
Gated model access - Llama and some models require Hub login and license acceptance. Fix: huggingface-cli login and accept license on model page.
Not setting pad_token - generation fails without pad token. Fix: tokenizer.pad_token = tokenizer.eos_token.
Downloading every run - re-downloads waste time. Fix: set HF_HOME cache directory; use local_files_only=True in production.
float32 on GPU - doubles memory vs float16. Fix: torch_dtype=torch.float16 or bfloat16.

Alternatives

Alternative	Use When	Don't Use When
`pipeline`	Quick inference	Custom decoding logic
vLLM	High-throughput serving	Simple local experiments
Ollama	Easy local model management	Fine-tuning
API providers	No GPU infrastructure	Data privacy requires local

FAQs

How do I authenticate with the Hub?

pip install huggingface_hub
huggingface-cli login

Required for gated models.

What is device_map auto?

Automatically spreads model layers across available GPUs.
Essential for models larger than single GPU memory.

How do I fine-tune a model?

from transformers import TrainingArguments, Trainer
args = TrainingArguments(output_dir="./out", num_train_epochs=3, per_device_train_batch_size=4)
trainer = Trainer(model=model, args=args, train_dataset=dataset)
trainer.train()

See Transfer Learning.

What models run on a laptop?

1B-3B parameter models with 4-bit quantization.
Llama 3.2 1B, Phi-3 mini, SmolLM.

How do I control generation length?

max_new_tokens limits new tokens (preferred over max_length).
min_new_tokens ensures minimum response length.

What is temperature in generate()?

Same as API temperature: higher = more random.
do_sample=False for greedy (deterministic) decoding.

How do I batch inference?

Pad inputs to same length; pass batch tensor to generate.
vLLM handles batching efficiently for serving.

Can I use transformers with PyTorch 2.6+?

Yes - transformers supports recent PyTorch versions.
torch.compile can speed up generation on supported models.

What is PEFT/LoRA?

Low-Rank Adaptation: fine-tune small adapter weights instead of full model.
Dramatically reduces memory for fine-tuning.

How do I cache models locally?

export HF_HOME=/data/hf_cache

Models download once and reuse from cache.

How do I run embeddings?

from transformers import AutoModel
embedder = pipeline("feature-extraction", model="sentence-transformers/all-MiniLM-L6-v2")

Or use sentence-transformers library directly.

What is the Hub model card?

Documents model capabilities, limitations, and intended use.
Read before deploying any open model.

Local & Open Models - Ollama and vLLM
Embeddings & Similarity - embedding models
Transfer Learning & Fine-Tuning - fine-tune patterns
Streaming & Async LLM Calls - streaming generation
Tokens, Cost & Rate Limits - local vs API cost

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

Quick-reference recipe card - copy-paste ready.

from transformers import pipeline
 
generator = pipeline("text-generation", model="meta-llama/Llama-3.2-1B-Instruct")
output = generator("Explain Python decorators:", max_new_tokens=100)
print(output[0]["generated_text"])

When to reach for this:

Running open-source LLMs locally or on your infrastructure.
Fine-tuning models with the Trainer API.
Accessing thousands of pretrained models from the Hub.
Building custom generation with control over sampling parameters.

Working Example

"""hugging_face_transformers.py - pipeline and manual generation."""
from __future__ import annotations
 
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
model_id = "meta-llama/Llama-3.2-1B-Instruct"
 
# Quick inference with pipeline
gen = pipeline("text-generation", model=model_id, torch_dtype=torch.float16, device_map="auto")
result = gen("What is pytest?", max_new_tokens=80, do_sample=True, temperature=0.7)
print("pipeline:", result[0]["generated_text"][-200:])
 
# Manual generation loop
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
 
messages = [{"role": "user", "content": "Write a one-line Python hello world."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
 
with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=60, temperature=0.7, do_sample=True)
 
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("manual:", response)

What this demonstrates:

pipeline for zero-config text generation.
AutoTokenizer and AutoModelForCausalLM for manual control.
apply_chat_template for instruction-tuned models.
device_map="auto" for automatic GPU placement.

Deep Dive

How It Works

Hub hosts model weights, configs, and tokenizers.
Auto classes resolve architecture from config.json.
Tokenizer converts text to token IDs and back.
generate() runs autoregressive decoding with sampling or greedy search.
Trainer fine-tunes models on custom datasets.

Key APIs

API	Use When	Complexity
`pipeline()`	Quick prototyping	Low
`model.generate()`	Custom generation params	Medium
`Trainer`	Fine-tuning	Medium
`trl` (SFT, DPO)	RLHF-style alignment	High

Python Notes

# 4-bit quantization for limited GPU memory
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")

Gotchas

Not using chat template - instruction models produce poor output without proper formatting. Fix: tokenizer.apply_chat_template.
OOM on large models - 70B models need multi-GPU or quantization. Fix: 4-bit loading, device_map="auto", or smaller models.
Gated model access - Llama and some models require Hub login and license acceptance. Fix: huggingface-cli login and accept license on model page.
Not setting pad_token - generation fails without pad token. Fix: tokenizer.pad_token = tokenizer.eos_token.
Downloading every run - re-downloads waste time. Fix: set HF_HOME cache directory; use local_files_only=True in production.
float32 on GPU - doubles memory vs float16. Fix: torch_dtype=torch.float16 or bfloat16.

Alternatives

Alternative	Use When	Don't Use When
`pipeline`	Quick inference	Custom decoding logic
vLLM	High-throughput serving	Simple local experiments
Ollama	Easy local model management	Fine-tuning
API providers	No GPU infrastructure	Data privacy requires local

FAQs

How do I authenticate with the Hub?

pip install huggingface_hub
huggingface-cli login

Required for gated models.

What is device_map auto?

Automatically spreads model layers across available GPUs.
Essential for models larger than single GPU memory.

How do I fine-tune a model?

from transformers import TrainingArguments, Trainer
args = TrainingArguments(output_dir="./out", num_train_epochs=3, per_device_train_batch_size=4)
trainer = Trainer(model=model, args=args, train_dataset=dataset)
trainer.train()

See Transfer Learning.

What models run on a laptop?

1B-3B parameter models with 4-bit quantization.
Llama 3.2 1B, Phi-3 mini, SmolLM.

How do I control generation length?

max_new_tokens limits new tokens (preferred over max_length).
min_new_tokens ensures minimum response length.

What is temperature in generate()?

Same as API temperature: higher = more random.
do_sample=False for greedy (deterministic) decoding.

How do I batch inference?

Pad inputs to same length; pass batch tensor to generate.
vLLM handles batching efficiently for serving.

Can I use transformers with PyTorch 2.6+?

Yes - transformers supports recent PyTorch versions.
torch.compile can speed up generation on supported models.

What is PEFT/LoRA?

Low-Rank Adaptation: fine-tune small adapter weights instead of full model.
Dramatically reduces memory for fine-tuning.

How do I cache models locally?

export HF_HOME=/data/hf_cache

Models download once and reuse from cache.

How do I run embeddings?

from transformers import AutoModel
embedder = pipeline("feature-extraction", model="sentence-transformers/all-MiniLM-L6-v2")

Or use sentence-transformers library directly.

What is the Hub model card?

Documents model capabilities, limitations, and intended use.
Read before deploying any open model.

Local & Open Models - Ollama and vLLM
Embeddings & Similarity - embedding models
Transfer Learning & Fine-Tuning - fine-tune patterns
Streaming & Async LLM Calls - streaming generation
Tokens, Cost & Rate Limits - local vs API cost

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Hugging Face transformers

Search Documentation

Hugging Face transformers