LLMs & GenAI Highlights Summary

Every highlight bullet from the 12 pages in this section, gathered on one page and grouped by the page it came from.

The LLM Interface Model

• An LLM is a stateless function over a fixed-size window of tokens
• A model never remembers a conversation - your code resends the history every call
• Cost, latency, and 'the model forgot' are all token and context-window problems, not model mood
• Structured output and tool schemas exist because native LLM output is free text, not parseable data
• Tool calling has the model choose a function and arguments - your code executes it, not the model

LLMs Basics

• LLMs predict the next token given prior context
• System prompts set behavior; user prompts carry the task
• Token count drives cost and context window limits
• Temperature controls randomness in generation

Anthropic Claude SDK

• Messages API uses system + messages list
• Streaming yields text deltas as they generate
• Tool use lets Claude call your Python functions
• Content blocks support text, images, and tool results

OpenAI & Other SDKs

• OpenAI SDK v1+ uses client instances, not module-level calls
• Chat completions accept messages with roles
• LiteLLM abstracts multiple providers behind one API
• Always load API keys from environment variables

Hugging Face transformers

• transformers provides AutoModel and AutoTokenizer
• pipeline() is the fastest way to run inference
• device_map='auto' distributes large models across GPUs
• Quantization reduces memory for local inference

Prompt Engineering

• System prompts define role, constraints, and output format
• Few-shot examples teach format without fine-tuning
• Chain-of-thought improves reasoning on complex tasks
• Iterate on prompts before reaching for fine-tuning

Structured Output

• Pydantic models define and validate LLM output schemas
• OpenAI response_format enforces JSON object output
• Tool schemas guarantee structured function arguments
• Always validate parsed JSON before using in production

Embeddings & Similarity

• Embeddings map text to dense vectors capturing semantic meaning
• Cosine similarity ranks documents by relevance to a query
• Pin one embedding model per vector index
• Normalize vectors before cosine similarity for efficiency

Streaming & Async LLM Calls

• Streaming shows tokens as they generate for better UX
• AsyncOpenAI enables concurrent requests in web servers
• SSE (Server-Sent Events) delivers streams to browsers
• Semaphore limits concurrent API calls to avoid rate limits

Tokens, Cost & Rate Limits

• Token count drives API cost and context window usage
• Cheaper models handle simple tasks; reserve premium for hard ones
• Prompt caching reduces cost for repeated system prompts
• Rate limits require backoff and concurrency control

Local & Open Models

• Ollama runs open models locally with one command
• vLLM serves models with high-throughput OpenAI-compatible API
• Quantization fits large models on consumer GPUs
• OpenAI-compatible endpoints work with existing SDK code

LLMs Best Practices

• Never hardcode API keys
• Validate all structured output with Pydantic
• Log token usage and cost per feature
• Evaluate prompts on labeled datasets before deploying

3730363230327c3735366f6967637c6f692e73656469756765646f63